StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Preparing data for feature engineering, training and serving machine learning models (e.g., BigQueryML)
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
To make machine learning experiments reproducible, you need to track every piece of data and every step of the pipeline. Dataplex Universal Catalog acts as a central metadata service where you can search, tag, and see dataset lineage — that is, where data came from and how it changed over time. Vertex ML Metadata records details about your workflows, linking parameters and metrics to specific model versions in the Vertex AI Model Registry. These automated registries give you full traceability, which is important for auditing and debugging when something goes wrong.
Freezing training data is critical for getting repeatable results. You should create a BigQuery table snapshot as soon as raw data passes validation, so the training inputs never change. If you use a regularly updated table, tuning and evaluation results will shift each time you run. Snapshots prevent that problem and keep your data lineage consistent across the entire ML lifecycle.
Training-serving skew happens when the data your model sees in production differs from what it was trained on, causing performance to drop. To avoid this, Vertex AI Feature Store works with BigQuery feature tables that include timestamp columns. You can use functions like ML.FEATURES_AT_TIME to look up feature values exactly as they existed at the time of training. This prevents data leakage — accidentally including future information — and keeps the training data temporally aligned with production data.
BigQuery ML provides built-in monitoring to catch drift before it hurts predictions. The ML.VALIDATE_DATA_SKEW function checks whether the distribution of training features differs from what the model sees when serving. The ML.VALIDATE_DATA_DRIFT function watches for changes in feature data distributions over time. These checks let teams spot problems early and safely trigger retraining before model accuracy degrades.
Choosing the right storage format and partitioning strategy directly affects cost and performance for training and batch prediction. BigQuery and Cloud Storage are the main services used for these optimizations. Efficient columnar formats like Parquet or Avro provide compression to reduce storage costs and support schema evolution, allowing data structures to change without breaking pipelines. These formats are ideal for the large-scale, read-heavy operations common in ML. When storing intermediate data in Cloud Storage, using these formats speeds up processing jobs run by services like Dataflow or Dataproc.
Within BigQuery, partitioning is a primary optimization technique. Partitioning a table by a date, timestamp, or integer column lets BigQuery prune queries so it only scans the relevant partitions. This cuts the amount of data processed, lowering cost and speeding up training jobs that scan datasets repeatedly. Clustering tables within partitions on specific feature columns also improves the performance of joins and filtering during feature engineering.
To keep data reproducible, use immutable BigQuery table snapshots of prepared datasets before training or prediction runs. This ensures the data for a model never changes and provides a reliable point-in-time reference for debugging. Always reference these snapshots instead of live tables so that updates to source data do not affect your ML results.
Data pipelines should be built to take advantage of these optimizations. Managed services like Dataflow (for Apache Beam pipelines) and Dataproc (for Spark jobs) can efficiently read and write optimized formats in Cloud Storage and BigQuery. For batch ML workloads, using BigQuery’s FILE_LOADS method is cost-effective. For real-time serving, the Storage Write API provides exactly-once streaming ingestion directly into BigQuery.
Data transformation pipelines turn raw data into features that ML models can use. These pipelines handle tasks like normalization, encoding categorical variables, handling missing values, and creating derived features. The result is clean, consistent data ready for training and serving.
First, choose data sources based on your needs. Google Cloud offers databases like Cloud SQL, Spanner, AlloyDB, and BigQuery for structured data, and data lakes using Cloud Storage with BigLake for less structured data. For real-time applications, use streaming platforms such as Pub/Sub, Dataflow, or Apache Kafka. For ingestion, Dataflow handles both batch and stream processing with ML-component integration. Cloud Run functions work well for event-driven ingestion triggered by data changes. BigQuery is ideal for tabular data that you access frequently.
Once data is ingested, tools like Dataflow and Dataform automate feature scaling, encoding categorical variables, and creating new features. Vertex AI Feature Store centralizes feature management, so the same features can be reused across different models and projects, maintaining consistency. BigQuery ML provides automatic preprocessing during model training: it replaces missing numeric values with the mean, maps missing categorical values to an extra category, and applies one-hot encoding for categorical data or standardization for numeric columns. This built-in handling simplifies pipeline code.
ML pipelines with many steps and dependencies need orchestration. Vertex AI Pipelines organizes pipelines as directed acyclic graphs (DAGs) of containerized tasks. Cloud Composer provides managed Apache Airflow for more complex workflows with heavy ETL requirements. To keep data quality high, use Dataplex Universal Catalog to discover and manage data assets across your organization, standardizing definitions without moving datasets to one location.
A key principle is reusability: reusing datasets and features across multiple ML projects speeds up development and cuts costs. Data products can be published and discovered through Dataplex Universal Catalog and BigQuery sharing. For ongoing work, implement MLOps practices that automate the model-development lifecycle, including version control for code and models, managed pipeline orchestration, and continuous monitoring to ensure reliability in production.