[YAML] A Streaming Inference Pipeline - Taxi Fare Estimation

[charlespnh]

Please add a meaningful description for your change here

Part of a larger effort #35069 and #35068 to add more examples involving Kafka and ML use cases.

Introduce a streaming inference pipeline for the regression problem of taxi fare estimation,
where the YAML pipeline reads from the public PubSub topic projects/pubsub-public-data/topics/taxirides-realtime
and writes to a Kafka topic, then reads from the same Kafka topic and does some transformations before
the RunInference transform performs remote inference with Vertex AI model handler and a custom trained
XGBoost model deployed to a Vertex AI endpoint.
A notebook for training and deploying this XGBoost model is also included.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Assigning reviewers:

R: @tvalentyn for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[charlespnh]

CC @chamikaramj and @damccorm

[derrickaw]

Suggested change

	how to create BigQuery datasets
	how to create BigQuery datasets.

[derrickaw]

Suggested change

	--yaml_pipeline_file transforms/ml/taxi-fare/streaming_sentiment_analysis.yaml \
	--yaml_pipeline_file transforms/ml/taxi-fare/streaming_taxifare_prediction.yaml \

[derrickaw]

Suggested change

"For a quick '0-to-1' model serving on Vertex AI, the model training process below is kept straighforward using the simple yet very effective [tree-based, gradient boosting](https://en.wikipedia.org/wiki/Gradient_boosting) algorithm. We start of with a simple feature engineering idea, before moving on to the actual training of the model using the [XGBoost](https://xgboost.readthedocs.io/en/stable/index.html) library.\n"

"For a quick '0-to-1' model serving on Vertex AI, the model training process below is kept straighforward using the simple yet very effective [tree-based, gradient boosting](https://en.wikipedia.org/wiki/Gradient_boosting) algorithm. We start off with a simple feature engineering idea, before moving on to the actual training of the model using the [XGBoost](https://xgboost.readthedocs.io/en/stable/index.html) library.\n"

[derrickaw]

Suggested change

"One of the columns in the dataset is the `pickup_datetime` column, which is of [datetimelike](https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.html) type. This makes it incredibly easy for performing data analysis on time-series data such as this. However, ML models don't accept feature columns with such a custom data type that is not a number. Some sort of conversion is needed, and here we'll choose to break this datetime column into multiple feature columns.\n"

"One of the columns in the dataset is the `pickup_datetime` column, which is of [datetime like](https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.html) type. This makes it incredibly easy for performing data analysis on time-series data such as this. However, ML models don't accept feature columns with such a custom data type that is not a number. Some sort of conversion is needed, and here we'll choose to break this datetime column into multiple feature columns.\n"

[derrickaw]

Suggested change

	"The input columns used for training (and subsequently for inference) will be the original feature columns (pick-up/drop-off longitude/latitude and the passenger count) from the dataset, along with the additional engineered features (pick-up year, month, day, etc...) that we generated above. The target/label column for training is the fare amount column.\n"
	"The input columns used for training (and subsequently for inference) will be the original feature columns (pick-up/drop-off, longitude/latitude, and the passenger count) from the dataset, along with the additional engineered features (pick-up year, month, day, etc...) that we generated above. The target/label column for training is the `fare_amount` column.\n"

[derrickaw]

Suggested change

	"Save the trained model to the Google Cloud Storage bucket as model artifact."
	"Save the trained model to the Google Cloud Storage bucket as a model artifact."

[derrickaw]

I would note somewhere in this readme that this is not tested like all of the other examples - just to make sure its clear to the reader. Thanks.

[charlespnh]

For future reference, pipeline has been tested with this Dataflow job
https://console.cloud.google.com/dataflow/jobs/us-central1/2025-07-18_07_16_21-10544402510837439480;bottomTab=WORKER_LOGS;expandBottomPanel=false;graphView=0;logsSeverity=INFO;mainTab=JOB_GRAPH;step=?pageState=(%22dfTime%22:(%22s%22:%222025-07-18T14:16:22.035Z%22,%22e%22:%222025-07-18T15:44:38.618Z%22))&project=apache-beam-testing