Ingesting Data for ML Training Pipelines

Applying data validation, transformation, and feature engineering turns raw structured and unstructured inputs into trusted, model-ready features for training pipelines. TensorFlow Data Validation (TFDV) (opens in a new tab) checks schema consistency and detects anomalies, Dataflow (opens in a new tab) with Managed Service for Apache Spark and BigQuery (opens in a new tab) execute transformations at scale, and Agent Platform Feature Store (opens in a new tab) stores and serves the resulting features for reuse. Validation results flow to BigQuery for analysis and to Cloud Storage for archiving, while monitored distributions trigger retraining when drift occurs.

1.1Automate Schema Validation and Data Quality Checks

Automated data validation is the continuous enforcement of schema consistency, format correctness, and statistical quality before data reaches training. TFDV runs inside ingestion and preprocessing pipelines to validate schemas, verify data formats, ranges, and distributions, and monitor distribution shifts over time. For large-scale SQL-based checks, validation runs in BigQuery with automated data quality scans, while complex programmatic validation on batch or streaming data runs in Dataflow with Apache Beam. The automatic data quality capabilities in Knowledge Catalog provide governed quality rules, and anomaly detection in BigQuery with statistical or ML models flags patterns that may indicate data poisoning or pipeline breakage.

Validation extends beyond rows to configurations and files that bound each pipeline component. Pipeline configuration files in YAML or JSON are validated against schemas with libraries such as jsonschema before a run starts, command-line arguments and pipeline parameters are validated for type, value, and presence with argparse and the built-in input validation of Agent Platform Pipelines, and validated versioned configurations are stored in Cloud Storage or Artifact Registry for reproducibility and audit. Input and output files such as datasets, model artifacts, and evaluation reports are checked for format correctness for CSV, Parquet, and image types, and for integrity through file size and checksum validation with Cloud Storage data validation and change detection. File checks execute in Dataflow or in Cloud Run functions triggered by upload events, and all validation metrics are written to BigQuery for historical tracking.

1.2Transform Raw Structured and Unstructured Data for Model Consumption

Transformation is the conversion of heterogeneous source data into normalized, encoded, and de-identified features that models can consume directly. Dataflow handles batch and stream preprocessing with ML-component integration and can combine with Eventarc for event-driven ingestion, Managed Service for Apache Spark handles Apache Spark transformations, and BigQuery handles SQL-based transformation and materialization. Raw data is first extracted from operational sources such as Cloud SQL, Spanner, AlloyDB for PostgreSQL, Firestore, Pub/Sub, Apache Kafka, or external APIs into storage optimized for training, separate from the operational source. Structured and semi-structured data lands in BigQuery as materialized tables rather than views or subqueries for training speed and is read through the BigQuery Storage API, while image, video, audio, and other unstructured data lands in Cloud Storage as large sharded containers such as TFRecord for TensorFlow or Avro for other frameworks.

Transformation choices depend on data shape, volume, and privacy requirements:

Privacy-preserving and semantic transformations run as part of the same pipelines. Sensitive Data Protection applies pseudonymization, de-identification, and generalization such as bucketing to de-identify data while preserving utility, and an Agent Platform data labeling job adds tags that convert raw data into structured training data for classification, entity extraction, and sentiment analysis. For generative AI where real data is insufficient or imbalanced, BigQuery DataFrames and Agent Platform synthetic generation create or augment datasets to improve robustness, prompt diversity, and generalization, with labels that reflect quality, relevance, and safety for fine-tuning and reinforcement learning from human feedback. Unstructured throughput depends on shard design: many small media files are combined into files of at least 100 MB with between 100 and 10,000 shards, grouped by Cloud Storage buckets and directories for management.

1.3Construct Reusable Feature Sets from Structured and Unstructured Inputs

Feature engineering for reuse is the creation of centrally defined, versioned, and monitored features that multiple training and serving workloads can share. Agent Platform Feature Store provides that center for tabular features: an online store instance hosts feature views defined over a BigQuery table or view, BigQuery retains all feature data including historical values for offline training, and scheduled monitors track feature groups for drift. The training pipeline depends on the feature view for point-in-time correct values, and the serving path depends on the online store for low-latency lookup, so training-serving skew drops when both read from the same definitions. Unstructured features follow a parallel pattern where Cloud Storage holds the bytes, BigQuery object tables and signed URLs govern access to those objects, and Vector Search indexes manage vector embeddings with their metadata for retrieval.

Governed reuse depends on discovery, lineage, and controlled sharing. Knowledge Catalog centralizes discovery, data lineage including BigQuery column-level lineage, data profiles, feature and model artifact metadata, business glossaries, and aspects that enrich technical metadata with business context, and it unifies governance across BigLake and open formats such as Iceberg and Delta Lake. Curated datasets are published as data products discoverable through Knowledge Catalog and BigQuery sharing, where data exchanges expose listings and data clean rooms enable collaboration on sensitive data without uncontrolled copying. Lifecycle protection comes from Cloud Storage lifecycle controls, customer-managed encryption keys, VPC Service Controls perimeters with ingress and egress rules, and row-level and column-level security in BigQuery.

1.4Monitor Distributions and Protect Training Integrity Over Time

Continuous monitoring is the feedback loop that keeps validated features trustworthy after they enter production pipelines. TFDV distribution tracking integrated with Cloud Monitoring visualizes trends and detects data drift, Model Monitoring on Agent Platform detects input drift and prediction drift against thresholds, and drift signals automatically trigger retraining pipelines when patterns change significantly. The monitoring flow runs in order:

1. Validate at ingestion: Data flows from source systems through Dataflow, Spark, or BigQuery checks that verify types, formats, ranges, and distributions.
2. Track over time: Validation metrics flow to BigQuery and Cloud Monitoring dashboards that compare live distributions to training baselines.
3. Respond to change: A threshold breach triggers alerts through email, Slack, or PagerDuty and can launch a retraining or fine-tuning pipeline on updated data.
4. Preserve evidence: Validation results remain queryable in BigQuery and validation artifacts remain archived in Cloud Storage, with API and access activity captured in Cloud Audit Logs, Cloud Logging, and lineage in Agent Platform ML Metadata and Knowledge Catalog.

Robust training practices reduce the impact of corrupted or adversarial inputs that monitoring surfaces. Ensemble methods train multiple models on different data subsets with hyperparameter tuning to dilute poisoned points, data augmentation balances distributions and adds adversarial examples, and human review of evaluation metrics, biases, and unexpected behavior catches patterns that automated checks miss. Sensitive-data discovery scans Agent Platform datasets and inspects BigQuery results into Knowledge Catalog aspects, while Model Armor screening and strict input and output sanitization in preprocessing code prevent manipulation from propagating from ingestion through training to deployment.

Selecting ingestion services for machine learning training means matching each source to a landing service and a processing service based on volume, velocity, and format compatibility. Cloud Storage (opens in a new tab) is the landing zone for unstructured data such as text, images, video, and audio, while BigQuery (opens in a new tab) is the landing zone for structured and semi-structured tabular data. Pub/Sub (opens in a new tab), Dataflow (opens in a new tab), and Dataproc move and transform data between operational sources and those training-optimized stores. The training pipeline then reads from Cloud Storage or BigQuery through high-throughput APIs rather than from the live operational database or API.

2.1Identify Source Systems by Volume, Velocity, and Format

Source identification is the inventory of operational databases, lakes, streams, and external feeds by write latency, data volume, and data format. Cloud SQL, Spanner, AlloyDB for PostgreSQL, Firestore, and BigQuery cover database sources, where the choice depends on whether writes are static or dynamic, whether volume is high or low, and whether the format is structured, semi-structured, or unstructured. Cloud Storage with BigLake serves as the data lake for large unstructured and open-format holdings, while Pub/Sub, Dataflow, or Apache Kafka serve as streaming event platforms for continuous signals. External APIs and governed assets discovered through Knowledge Catalog, formerly Dataplex Universal Catalog, complete the source inventory for databases, lakes, streams, and third-party feeds.

Once sources are identified, each source needs an ingestion path that preserves format compatibility and handles its velocity. Batch extracts from databases and APIs flow through batch pipelines into BigQuery or Cloud Storage, while high-velocity events flow first through a messaging buffer before processing. Data that must remain governed and reusable is registered so lineage, quality checks, and discovery apply across data marts and feature stores. This separation lets the same training pipeline consume slow-changing tables, bulk object dumps, and real-time streams without coupling training reads to production systems.

2.2Buffer High-Velocity Events with Pub/Sub

Pub/Sub is the fully managed real-time messaging service that decouples event producers from training ingestion when velocity is high or bursty. Data flows from publishers such as application logs, sensors, or external APIs to a Pub/Sub topic, and then from the topic to subscribers such as a Dataflow pipeline or Cloud Storage sink. Because the topic buffers messages independently of the subscriber, spikes do not overwhelm transformation or training jobs and components can scale separately. Choose Pub/Sub when data arrives continuously, arrives unpredictably, or must fan out to multiple consumers including streaming training inputs and monitoring.

For event-driven architectures, streaming ingestion combines with Eventarc to trigger Dataflow processing when new events or object arrivals occur. In that pattern, traffic flows from the event source to Eventarc to a Dataflow job, and only after the trigger fires does Dataflow read, transform, and write the micro-batch for ML use. This ordering avoids polling and keeps preprocessing close to arrival time for fresh features. Use this combination when training or retraining must react to new data rather than run only on a fixed schedule.

2.3Transform Batch and Streaming Data with Dataflow, Dataproc, and BigQuery Engines

Transformation engines for diverse sources divide work by interface and processing model. Dataflow runs both batch and stream processing for Apache Beam pipelines with ML-component integration, and it supports GPU acceleration and right-fitting to improve MLOps and ML job efficiency. Managed Service for Apache Spark, serverless Dataproc, runs Apache Spark workloads including Spark SQL and custom code in Scala, Java, Python, or R. BigQuery performs SQL-based transformation directly on staged tables, which avoids moving large tabular data to a separate cluster.

The choice carries a tradeoff in movement versus expressiveness that matters at scale. Dataflow and Spark pull data from sources such as Cloud Storage, Pub/Sub, and databases, apply transforms in flight, and then write training-ready outputs to BigQuery or Cloud Storage. BigQuery extract, load, and transform (ELT) instead loads raw data into a staging table first and then runs SQL queries that write their output into a final production table. Choose in-flight processing when formats differ widely or streams must be windowed and joined, and choose BigQuery SQL transforms when tabular data is already centralized and the logic fits set-based SQL.

2.4Land Structured and Semi-Structured Data in BigQuery for Training

BigQuery landing is the recommended store for all structured and semi-structured training data, including intermediate processed data, following BigQuery project and table organization practices. Data is extracted from source systems and converted to a training-optimized copy that is separate from the operational source, so training scans do not contend with production latency or consistency needs. For maximum training speed, BigQuery stores materialized tables rather than views or subqueries as the direct training input, because materialization avoids recomputing logic on every read. The training job then depends on the BigQuery table, not on the live Cloud SQL, Spanner, AlloyDB, or Firestore instance.

Reads for training flow from BigQuery through the BigQuery Storage API to the framework, which provides high-throughput columnar access for large scans. TensorFlow for Keras uses the tf.data.dataset reader for BigQuery, TensorFlow Extended (TFX) uses the BigQuery client, Dataflow uses the BigQuery I/O Connector, and other frameworks such as PyTorch, XGBoost, or scikit-learn integrate through BigQuery connectors and export paths. For artifact tracking, a managed tabular dataset ties the exact table version to the experiment. Schemas are validated and anomalies are detected with TensorFlow Data Validation (TFDV) and automated data quality scans in BigQuery before the data enters training.

2.5Land Unstructured Data in Cloud Storage for Training

Cloud Storage landing is the landing zone for image, video, audio, and other unstructured data destined for training, organized with buckets and directories that group related shards for management. Data flows from capture systems or web scrapes into large container files in Cloud Storage, and then from those containers to training readers, rather than as millions of small individual files. Sharded TFRecord files are used when training with TensorFlow and Avro container files are used for other frameworks, since those formats pack many records into splittable, sequential reads. Combining many clips, images, or documents into large files improves read and write throughput to Cloud Storage and keeps distributed trainers fed.

Shard sizing and governance determine whether that throughput holds at expert scale. Cloud Storage objects for training should aim for files of at least 100 MB, with between 100 and 10,000 shards per dataset, so parallelism is high without creating metadata and open-file overhead. The bucket contents are governed for ML use with BigQuery object tables and signed URLs, which let SQL and access controls reference objects without copying bytes into tables. Vector embeddings derived from this unstructured corpus are then managed with metadata in Vector Search indexes, keeping the raw objects, tabular governance, and similarity search linked.

A scalable and reliable ingestion pipeline for training extracts data from operational systems, converts it into training-optimized storage, and feeds it to Vertex AI Training (opens in a new tab) on a repeatable schedule. The pipeline separates operational sources from training copies, stores structured data in BigQuery (opens in a new tab) and unstructured data in Cloud Storage (opens in a new tab), and orchestrates validation, transformation, and retraining with Vertex AI Pipelines (opens in a new tab), TensorFlow Extended (TFX), and Kubeflow components. Versioning, lineage, and monitoring are built into each stage so continuous training remains reproducible and fault-tolerant.

3.1Separate Operational Sources From Training Storage

Separation of operational and training storage is the practice of extracting data from any origin and converting it to a format and location optimized for machine learning. The training copy lives separate from the operational source so training reads do not contend with production traffic and can be reprocessed without affecting live systems. Origins include online service logs, on-device images, or scraped documents that differ in format and velocity. This separation also lets teams choose columnar tables for tabular data and sharded binary containers for media, which maximizes read throughput during distributed training.

3.2Store Structured Training Data in BigQuery with High-Throughput Connectors

BigQuery storage for training pipelines is the recommended store for all structured and semi-structured training data, including intermediate processed data. Project structure should follow BigQuery table recommendations, and training should read materialized tables rather than views or subqueries for maximum speed. For large datasets, partitioning improves both query performance and cost efficiency when scans feed preprocessing or training.

Data flows from BigQuery tables to training through the BigQuery Storage API, with a managed tabular dataset used when artifact tracking is needed. The choice of connector depends on the training framework:

The connector runs inside the training or preprocessing job, so BigQuery remains the system of record while the framework streams rows in parallel without exporting intermediate files.

3.3Store Unstructured Training Data in Cloud Storage with Sharded Containers

Cloud Storage for training pipelines is the recommended store for image, video, audio, and other unstructured data in large container formats. TensorFlow workflows use sharded TFRecord files, while other frameworks use Avro files, and buckets plus directories group the shards for management. Combining many small media files into large containers improves both read and write throughput to Cloud Storage.

Throughput depends on shard sizing, so files should be at least 100 MB with between 100 and 10,000 shards per dataset. Data flows from source systems to sharded containers in Cloud Storage, then from Cloud Storage directly to Vertex AI Training workers during training. This layout lets distributed trainers read in parallel while avoiding the overhead of opening millions of individual files.

3.4Process Tabular and Unstructured Data at Scale for Training Pipelines

Processing at scale for training pipelines offers three managed paths that all feed the same training stores, with the choice depending on data type, volume, and existing code. Dataflow, based on Apache Beam, handles large-volume batch and streaming conversion, including converting unstructured data into binary formats like TFRecord to improve ingestion performance during training. BigQuery handles SQL-expressible transformations for tabular data, including built-in preprocessing for BigQuery ML and AutoML, which is often the most cost-effective path for custom models. Managed Service for Apache Spark provides serverless Spark processing when an organization already has Spark code and skills, while one-off Python or pandas scripts fit smaller datasets that fit in memory.

For event-driven ingestion, Dataflow combines with Eventarc so new arrivals trigger processing for machine learning, and Dataflow GPU and right-fitting options improve MLOps and ML job efficiency. When a transform is not expressible in SQL or requires streaming, a combination of Dataflow and pandas bridges the gap between warehouse logic and pipeline code.

3.5Validate Data Quality and Consistency in Ingestion Pipelines

Validation in ingestion pipelines checks schemas, formats, ranges, and distributions before data reaches training. TensorFlow Data Validation (TFDV) validates schemas and detects anomalies in Beam or TFX pipelines, while automated data quality scans in BigQuery apply where SQL tables are the source. These checks run during collection, storage, and transformation so drift, missing values, or malformed records are caught before they poison a training run.

Governance extends validation with centralized monitoring through Knowledge Catalog, formerly Dataplex, which manages data lineage including BigQuery column-level lineage, data quality checks and profiles, discovery across data marts, and feature and model artifact metadata. Anomaly detection in BigQuery using statistical or machine learning methods flags training-data outliers that may indicate poisoning or pipeline faults. Together, TFDV for pipeline-level validation and Knowledge Catalog plus BigQuery scans for warehouse-level quality create overlapping defenses for consistency.

3.6Centralize Reusable Features With Vertex AI Feature Store

Vertex AI Feature Store is the central location for creating, maintaining, sharing, and serving machine learning features from structured data. Feature data lives in a BigQuery table or view, which preserves historical values for offline training while serving low-latency online requests from the same definitions. To use the store, first create an online store instance and then define feature views that map BigQuery sources to serving features.

Data flows from BigQuery source tables to feature views in the online store, then to both offline training jobs and online serving paths. Because BigQuery retains historical feature data, point-in-time correctness is preserved for retraining without rebuilding ad hoc joins. Reusing the same feature views across projects avoids redundant engineering, reduces infrastructure cost, and keeps training and serving consistent.

3.7Track Versioning and Lineage With Managed Datasets and Metadata

Versioning in training pipelines covers code, data, features, and models with a distinct store for each artifact type. Container images and pipeline components are versioned in Artifact Registry, datasets live in BigQuery or Cloud Storage, and model binaries and metadata are versioned in Model Registry on Vertex AI. Source code, pointers to data locations, and pipeline definitions are tracked in Git or a data catalog, while Vertex AI Feature Store maintains consistency and versioning of feature data itself.

Lineage links those versions into a reproducible chain from raw source to deployed model. Managed datasets in Vertex AI create a clear link between preprocessed data and custom-trained models and provide descriptive statistics plus automatic or manual train, test, and validation splits. Vertex ML Metadata logs metadata about data sources, transformations, model parameters, and experiment results, and Vertex AI Experiments builds linkage across parameters, metrics, and dataset and model artifacts. Knowledge Catalog tracks lineage of Vertex AI Pipelines artifacts for debugging and root-cause analysis, and audit logs record who accessed or modified data and models and when. Managed datasets are optional when teams need full control over splitting inside training code or when data-to-model lineage is not critical.

3.8Orchestrate Continuous Training With Vertex AI Pipelines

Orchestration with Vertex AI Pipelines is the managed automation of every stage from data preparation to training, evaluation, and deployment. A pipeline composes ingestion, validation, transformation, and training as Kubeflow-style component tasks that pass data through arguments, then submits the whole graph for execution on serverless infrastructure. Scheduled pipeline runs, execution metrics, and reusable pipeline templates standardize retraining so the same workflow runs on a timetable or in response to new data.

The training stage depends on dataset size and production needs. Training inside a Vertex AI Workbench instance is sufficient for small datasets or subsets, while the Vertex AI managed training service handles larger datasets and distributed training. Even small datasets should move to the managed training service when training runs on a schedule or on arrival of additional data, because only the managed service provides production-grade scheduling, hyperparameter tuning with Vertex AI Vizier, and integration with TensorBoard and Experiments for comparison. Data sources feeding the schedule include databases such as Cloud SQL, Spanner, AlloyDB, and Firestore, lakes such as Cloud Storage with BigLake, streaming platforms such as Pub/Sub, Dataflow, and Apache Kafka, Knowledge Catalog for governed discovery, and external APIs, with Dataflow selected as the ingestion tool for batch and stream sources that need ML-component integration.

Key Facts

• BigQuery is the landing zone for structured and semi-structured training data as materialized tables, and training reads flow from BigQuery through the BigQuery Storage API rather than from live operational databases.
• Cloud Storage is the landing zone for unstructured training data as large sharded TFRecord containers for TensorFlow or Avro containers for other frameworks, with files of at least 100 MB and between 100 and 10,000 shards.
• Pub/Sub buffers high-velocity events by flowing from publishers to a topic to subscribers such as Dataflow, while Eventarc triggers Dataflow processing when new events arrive.
• TensorFlow Data Validation (TFDV) validates schemas and distributions in pipelines, BigQuery automated scans validate large-scale SQL tables, and drift signals can trigger retraining pipelines.
• Vertex AI Feature Store serves features from feature views defined over BigQuery sources, and Vertex AI Pipelines orchestrates ingestion, validation, transformation, training, and deployment for continuous training.