BigQuery is the primary analytical data warehouse for storing and processing structured and semi-structured tabular datasets on Google Cloud. To maximize training speed, intermediate feature tables must be materialized directly into BigQuery tables rather than accessed through logical views or subqueries. Training applications read data out of these tables using the BigQuery Storage Read API, which bypasses the standard SQL query engine to stream row segments in parallel. Organizing BigQuery tables with partitioning splits the data by a specific column such as date, ensuring queries scan only relevant segments. Clustering further sorts the data within those partitions by up to four specified columns, reducing scanned bytes and accelerating read operations.
Google Cloud Storage (GCS) is the foundational object store for storing unstructured media such as images, audio recordings, video files, and raw text. Storing millions of individual small files creates metadata bottlenecks that degrade read and write throughput during training. Machine learning workflows resolve this by packaging individual media files into sharded container files of at least 100 MB each, distributing the overall dataset across 100 to 10,000 shards. Pipelines targeting TensorFlow store these shards in the TFRecord format, while pipelines using other frameworks package data into the Avro format. Organizing these shards within structured bucket directories ensures balanced load distribution and leverages the scalable bandwidth of object storage.
The choice between BigQuery, Cloud Storage, and parallel file systems depends on the data modality, file size, and access latency requirements of the workload. BigQuery provides SQL-driven analytics and integrates with feature management tools like Vertex AI Feature Store for tabular records. Cloud Storage, accessed through the Cloud Storage FUSE file system adapter, supports large files of 50 MB or more where applications can tolerate access latencies in the tens of milliseconds. In contrast, Google Cloud Managed Lustre serves as a specialized parallel file system for workloads that require sub-millisecond read latencies, small file handling under 50 MB, and full POSIX file system compliance.
Partitioning and clustering are native BigQuery performance optimizations that minimize the volume of data scanned by machine learning queries. Partitioning divides a table into distinct segments based on an ingestion timestamp, date column, or integer range, allowing queries to prune unrelated partitions entirely. Clustering sorts the underlying data blocks within each partition based on the contents of up to four specified columns, such as a user identifier or category code. When downstream extraction jobs filter on clustered fields, BigQuery skips irrelevant data blocks within the selected partitions. The BigQuery Storage Read API leverages these partitioned and clustered blocks to stream target datasets to compute engines with minimal input/output (I/O) overhead.
Batch extraction pipelines prepare and load data into optimized storage structures before model execution begins. Data flows downstream from raw source systems through extract, transform, and load (ETL) pipelines, where records are validated, transformed into columnar or binary container formats, and loaded into BigQuery or Cloud Storage. Structured training pipelines stream rows directly from BigQuery tables using framework readers like TensorFlow's tf.data integration with the BigQuery Storage Read API. Unstructured training pipelines stream pre-sharded TFRecord or Avro files directly from Cloud Storage buckets to accelerator nodes. Pre-processing datasets into immutable, sharded structures prior to training guarantees high I/O throughput and input reproducibility across training experiments.
Serializing and sharding unstructured data prevents storage metadata bottlenecks from leaving graphics processing units (GPUs) and Tensor Processing Units (TPUs) idle. Storing individual media files directly on object storage creates excessive metadata requests and high time-to-first-byte latency. Workloads eliminate these bottlenecks by packing media files into sequential container formats like TFRecord for TensorFlow or Avro for PyTorch. Datasets are split into shards ranging between 100 MB and several gigabytes across a total volume of 100 to 10,000 files. Enabling hierarchical namespaces on Cloud Storage buckets creates structured directory trees that optimize folder-level performance during parallel reads.
Bucket location settings determine the physical proximity between training data and compute accelerators, governing replication guarantees and network egress costs. Architects choose from three bucket deployment models based on accelerator placement and data resilience needs:
Specialized ingestion interfaces and caching layers enable accelerator clusters to read data from Cloud Storage at speeds exceeding 1 TB/s. The Cloud Storage FUSE adapter mounts object storage buckets as local file systems on Compute Engine virtual machines and Google Kubernetes Engine (GKE) clusters using a container storage interface (CSI) driver. For high-demand distributed workloads, distinct acceleration services scale read bandwidth:
Selecting the proper storage tier requires balancing POSIX compatibility, latency tolerances, and long-term retention costs across the machine learning lifecycle:
To manage long-term retention costs, administrators configure Cloud Storage Autoclass or Object Lifecycle Management rules. Lifecycle policies automatically transition aging training data from Standard storage to Nearline (monthly access), Coldline (quarterly access), and Archive (annual access) tiers, or delete obsolete checkpoints based on age and version policies.
Vertex AI Datasets provide a centralized management layer for organizing tabular, text, image, video, and speech data across machine learning pipelines. Tabular datasets store their underlying records in BigQuery to leverage SQL queries and high-throughput parallel extraction via the BigQuery Storage Read API. Text, vision, video, and speech datasets store raw media files and sharded containers within Cloud Storage buckets. Storing unstructured data in large shards of at least 100 MB preserves high ingestion bandwidth when Vertex AI provisions training workers. This dual-backend architecture separates metadata tracking from raw physical storage while preserving native framework access.
Manifest files define schemas and map physical data locations for ingestion into Vertex AI Datasets. Tabular data enters Vertex AI directly from BigQuery tables, which support nested and repeated schemas, or through CSV files stored in Cloud Storage for simpler tabular structures. Unstructured media and multimodal datasets rely on JSON Lines (JSONL) manifest files that pair labels and metadata with direct Cloud Storage object URIs. Vertex AI automatically infers data schemas from these source manifests during dataset creation. Using structured manifests standardizes how downstream custom containers and AutoML components parse multimodal inputs.
Vertex AI Datasets support three deterministic splitting strategies to allocate data across training, validation, and test subsets:
Vertex AI passes dataset paths and split configurations to training containers dynamically through standardized environment variables at execution time. The AIP_DATA_FORMAT variable defines whether the input payload is structured as JSONL, CSV, or BigQuery references. The training container retrieves its specific data partitions via the AIP_TRAINING_DATA_URI, AIP_VALIDATION_DATA_URI, and AIP_TEST_DATA_URI environment variables. This abstraction eliminates hardcoded file paths in model code and ensures the execution container reads the exact partitioned files assigned to that job.
Vertex AI Datasets link directly with Vertex ML Metadata to establish end-to-end data lineage and auditability across all pipeline stages. Each dataset version is registered as an immutable artifact, and subsequent training runs log those artifacts as execution dependencies. If a model must be retrained or audited for compliance, the metadata graph links the model artifact back to the exact dataset version, manifest file, and split strategy used. This lineage tracking guarantees experiment reproducibility and produces a verifiable audit trail without requiring manual data version logging.
StudyCerts
StudyCertsAIP_DATA_FORMAT, AIP_TRAINING_DATA_URI, AIP_VALIDATION_DATA_URI, and AIP_TEST_DATA_URI environment variables.Choose Google Cloud Managed Lustre when the training workload consists of small files under 50 MB, requires random I/O with sub-millisecond latency, or depends on full POSIX file system compliance up to 8 PB. Choose Cloud Storage with Cloud Storage FUSE when datasets contain files of 50 MB or larger, can tolerate latencies in the tens of milliseconds, and need to scale across multiple regions up to exabyte capacity.
A random split automatically distributes data into an 80% training, 10% validation, and 10% test distribution. A manual split requires a dedicated column in the source data that explicitly assigns each row to a partition. A chronological split uses a designated timestamp column to divide the data sequentially over time, which prevents future data from leaking into the training set of time-series models.
The BigQuery Storage Read API bypasses the standard SQL query processing engine and directly streams underlying columnar row segments in parallel across multiple worker streams. When combined with partitioned and clustered tables, it allows training pipelines to read filtered subsets of data with high throughput and low CPU overhead.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills