Prepare and test your skills
Prepare and test your skills
Vertex AI Datasets (opens in a new tab) are the standardization layer that turns raw files and tables into versioned, splittable inputs for training on Vertex AI. Data flows from operational source systems into training-optimized storage in BigQuery (opens in a new tab) or Cloud Storage, then into a managed dataset that records schema, statistics, and lineage to custom-trained models. This separation lets teams optimize ingestion once for performance and scalability, then reuse the same dataset across AutoML, custom training, and pipeline jobs with reproducible train-validation-test splits.
Training-data preparation is the process of extracting data from source systems and converting it into formats and storage separate from the operational source. Source data can come from online service logs, on-device images, or web-scraped documents, but training jobs must not read directly from those live systems. The converted copy is stored in BigQuery for structured data or in Cloud Storage container files for unstructured data, so throughput, partitioning, and sharding can be tuned for machine learning. This boundary protects operational performance while giving training jobs materialized, high-throughput inputs that can be tracked as artifacts.
BigQuery ingestion is the standard path for structured and semi-structured tabular data intended for machine learning. All raw and most intermediate processed data stays in BigQuery tables organized by project, and training uses materialized tables instead of views or subqueries for maximum read speed. For large datasets, partitioning improves both query performance and cost efficiency during preprocessing and extraction. Data flows out of BigQuery to training through the BigQuery Storage API, and wrapping the source as a managed tabular dataset adds artifact tracking that links the table to the trained model.
The reader that consumes the BigQuery Storage API depends on the training framework:
| Framework | Google Cloud tool to read BigQuery |
|---|---|
| TensorFlow with Keras | tf.data.dataset reader for BigQuery |
| TensorFlow Extended (TFX) | BigQuery client |
| Dataflow with Apache Beam | Google BigQuery I/O Connector |
| Other frameworks such as PyTorch, XGBoost, or scikit-learn | BigQuery import and connector paths |
Cloud Storage ingestion is the standard path for image, video, audio, and other unstructured data. Individual media files are combined into large container formats before training, which improves read and write throughput compared to thousands of small objects. TensorFlow workflows use sharded TFRecord files, while other frameworks use Avro files for the same purpose. Buckets and directories group the shards for data management, so a training job can list and stream a coherent prefix rather than discovering scattered objects.
Exam tip: Combine media into files of at least 100 MB with between 100 and 10,000 shards; many small files lower throughput, while too few large shards limit parallel reads.
Data processing converts stored raw data into preprocessed features ready to be registered as a managed dataset. BigQuery transformations handle structured and semi-structured preprocessing, including the built-in transformations in BigQuery ML and AutoML; for custom models, BigQuery transformations are often the most cost-effective option. Dataflow, which uses the Apache Beam programming model, handles large-volume and unstructured conversions such as converting raw media into binary TFRecord to improve ingestion performance during training. Managed Service for Apache Spark fits organizations with existing Spark code and skills for serverless Spark processing, while one-off Python scripts fit smaller datasets that fit into memory and Dataflow combined with pandas fits transformations not expressible in SQL or streaming cases.
A managed Vertex AI Dataset is a registered training input that creates a clear link between preprocessed data and custom-trained models. The dataset depends on the underlying BigQuery table or Cloud Storage objects, but adds descriptive statistics, centralized management in the console, SDK, or REST API, and integration with Vertex ML Metadata and Experiments to log parameters, metrics, and dataset-to-model artifacts. Creation and import can start in the Google Cloud console, including labeling workflows and text workflows for classification, entity extraction, and sentiment analysis, with marketplace options such as Labelbox and Snorkel Flow. Choose a managed dataset when lineage between data and model or shared reuse across jobs matters; skip it when training code needs full control over splitting or when lineage is not critical.
Standardized datasets are reused without duplication through an ordered lifecycle:
Dataset splitting is the assignment of every example to training, validation, or test inside the managed dataset definition. Automatic splitting lets Vertex AI (opens in a new tab) divide the data for AutoML and standard custom training, which preserves reproducibility because the same dataset version yields the same subsets for evaluation and comparison. Manual splitting keeps the assignment under user control for cases that need deterministic stratification, time-based cutoffs, or custom logic implemented in training code. Because aggregate metrics like accuracy, precision, and recall are computed on the held-out test split and slices of data, a stable split definition is what makes model versions and architectures comparable over time.
Pipeline reuse is how a standardized dataset feeds automated machine learning pipelines without reformatting. Dataset components (opens in a new tab) expose dataset operations as pipeline steps, so a step that creates or imports the dataset passes its artifact to a later TrainingPipeline step and data preparation, training, and evaluation share one lineage chain. The component set includes operations for image, tabular, text, and time-series datasets:
Multimodal datasets (opens in a new tab) extend the same managed-dataset pattern to generative AI workloads that mix text, image, audio, and video. A multimodal dataset is created once from BigQuery, DataFrames, or JSONL files in Cloud Storage and is stored in BigQuery, which is optimized for large datasets. The same dataset can then serve supervised fine-tuning and batch prediction, which prevents duplication and formatting drift between jobs. Multimodal datasets support assemble and assess methods to preview data, validate schema and structure, and estimate resources and cost before launching a task, and they are restricted to generative AI services while single-modality managed datasets serve predictive AutoML and custom models.
High-throughput Cloud Storage (opens in a new tab) layouts for unstructured data colocate large, sharded objects next to GPU and TPU compute and expose them as a parallel file interface. Cloud Storage is a globally scalable object store where data is stored as objects in containers called buckets. For distributed training, throughput depends on three linked choices: how media is serialized into shards, where buckets are located and what storage class they use, and how training nodes mount and cache the data.
Serialized container files are the unit of high-throughput training reads for images, video, audio, and unstructured text. Cloud Storage achieves its best read and write throughput when many small media clips are combined into large container files instead of stored as millions of individual objects. Combine individual images, videos, or audio clips into sharded TFRecord files when training with TensorFlow, or into sharded Avro files when training with PyTorch, XGBoost, scikit-learn, or other frameworks. Aim for each shard to be at least 100 MB, with a dataset split across 100 to 10,000 shards, so distributed workers can open many large streams in parallel without metadata or request-rate bottlenecks.
Large shards change how data flows from storage to accelerators. Data flows from Cloud Storage objects through parallel download streams to each GPU or TPU worker, and each worker reads a disjoint subset of shards. Small files reverse this advantage because each open, lookup, and close adds per-object latency and queries-per-second load. Cloud Storage FUSE (opens in a new tab) is not suitable for workloads dominated by files smaller than 50 MB or that require less than 1 millisecond latency for random I/O and metadata access.
Exam tip: Use sharded TFRecord for TensorFlow and sharded Avro for other frameworks, with shards of at least 100 MB and 100 to 10,000 shards per dataset.
Bucket and directory layout is how sharded datasets stay manageable and parallelizable at scale. Cloud Storage buckets and directories group related shards so ingestion jobs, preprocessing pipelines, and training jobs share one central repository as a single source of truth across all stages of the ML workflow. A typical hierarchy separates raw media, serialized training shards, validation shards, and checkpoints into distinct prefixes or buckets, and separates training buckets from serving buckets to prevent accidental exposure of sensitive training data during serving.
Directory structure also controls lookup performance when buckets are mounted as file systems. Hierarchical namespace (HNS) enables a true file system structure inside a bucket with atomic folder renames and faster file lookups when the bucket is mounted with Cloud Storage FUSE. Buckets with hierarchical namespace deliver 8 times higher queries per second (QPS) for object reads and writes than buckets without it. Enabling hierarchical namespace is highly recommended for high-throughput data loading and frequent model checkpointing, and hierarchical namespace is required when creating zonal buckets with Rapid Bucket.
Bucket location controls where bytes sit relative to accelerators and how they survive zone or region failures. Cloud Storage supports zonal, regional, dual-region, and multi-region bucket locations, and the choice trades network locality and peak throughput against cross-region availability.
| Bucket location | How data is stored | Choose when |
|---|---|---|
| Zonal bucket with Rapid Bucket | Objects are stored in the Rapid storage class colocated in the same zone as GPU or TPU clusters | Training needs maximum throughput up to 15 TB/s and lowest latency for I/O-intensive distributed training |
| Regional bucket | Data is replicated synchronously across multiple zones within one region | Training runs in one region and needs zonal resilience plus data residency within the specified region |
| Dual-region bucket | Data is replicated asynchronously across two regions | Accelerator capacity or inference is split across two regions and models must stay available across both |
| Multi-region bucket | Data is replicated asynchronously across multiple regions | Accelerator capacity is fragmented across regions and a central dataset must serve training or inference from many regions |
Data flows from the bucket location to compute over the Virtual Private Cloud (VPC) network, so distance directly affects training stall time. Products in the Cloud Storage Rapid family clear data bottlenecks by bringing data into the same zones as compute and scaling storage throughput for GPU or TPU clusters. Rapid Cache is an SSD-backed zonal read cache that accelerates reads to existing dual-region or multi-region buckets without code changes, serving reads at up to 2.5 TB/s with lower latency than the origin bucket and with reduced data transfer fees compared to reading directly from a multi-region bucket.
Storage class determines the price and performance profile of the objects that training loops read repeatedly. The Rapid storage class is a high-performance class optimized for I/O-intensive zonal workloads and is used by objects in zonal buckets with Rapid Bucket (opens in a new tab). Standard storage fits active high-throughput data lakes, streaming video, and media asset libraries, while Nearline, Coldline, and Archive fit backups and long-tail content that is not read in every training step. Place hot training shards in Rapid or Standard and move cold snapshots and exported artifacts to colder classes.
The access path determines whether that class performance reaches the GPUs. Cloud Storage FUSE is an open-source FUSE adapter that mounts buckets as local file systems so applications use standard file system semantics against object storage. With proper planning, architectures built around Cloud Storage FUSE can exceed 1 TB/s of aggregate throughput. Cloud Storage FUSE provides client-side caching and tuning parameters such as parallel downloads, which fetch parts of an object or model in parallel and use the cache directory as a prefetch buffer to shard and parallelize streams.
For training workloads that mount buckets with Cloud Storage FUSE, three properties make the central-bucket pattern practical:
Reliability and isolation depend on disciplined write and cache ownership. Failed requests from Cloud Storage FUSE to Cloud Storage are retried with exponential backoff, which gradually increases time between retries to ride out transient network issues. Concurrent writes follow a last write wins rule where the last writer overwrites earlier writes, so a single object must be modified by only one source at a time. Caches are specific to each Cloud Storage FUSE process and do not persist across unmount or restart, so the same cache directory must not be shared by different processes and the file cache directory must be deleted manually after unmount to avoid stale or exposed data.
Structured and semi-structured training data in Google Cloud belongs in BigQuery (opens in a new tab) as materialized, partitioned tables separate from operational sources, with batch extraction through the BigQuery Storage Read API. Cloud Storage complements BigQuery as the high-throughput object store for large container files, sharded exports, and unstructured payloads that feed the same training workflows. Together the two services optimize ingestion performance, I/O throughput, scalability, and reproducibility for machine learning workloads.
Training data preparation extracts data from source systems such as online service logs, devices, or web documents and converts it to a format and storage location optimized for ML training. The training copy lives separate from the operational source so training scans do not contend with production reads and writes. In most cases intermediate, processed data is also stored in BigQuery so the full lineage from raw to training-ready remains queryable and reproducible.
BigQuery is the recommended store for all structured and semi-structured training data, following BigQuery project structure recommendations for datasets and tables. BigQuery tables hold both raw ingested rows and intermediate processed features, which keeps preprocessing in SQL close to the data. For maximum training speed, BigQuery training inputs should be stored as materialized tables instead of views or subqueries, because materialization avoids recomputing logic on every read and lets the storage layer scan physical blocks efficiently.
The BigQuery Storage Read API (opens in a new tab) is the high-throughput path that streams rows from BigQuery tables directly to training and processing frameworks for batch extraction. Data flows from BigQuery storage to the training client, where a framework-specific connector manages parallel streams, column projection, and snapshot consistency. For artifact tracking and reproducibility, BigQuery reads for training should use a managed tabular dataset so the exact table version is recorded with the model run.
The connector choice depends on the training framework that consumes data from the BigQuery Storage Read API:
| Training framework | Google Cloud tool for BigQuery reads |
|---|---|
| TensorFlow for Keras | tf.data.dataset reader for BigQuery |
| TensorFlow Extended (TFX) | BigQuery client |
| Dataflow | Google BigQuery I/O Connector |
| Any other framework such as PyTorch, XGBoost, or scikit-learn | BigQuery ONNX import path |
A partitioned BigQuery table is a table divided into segments called partitions that make large tables easier to manage and query. Partition management is a key determinant of BigQuery performance and cost because a query over a specific date range scans only matching partitions instead of the full table. Teams typically split large tables so each partition holds a day's worth of data, which aligns storage layout with time-filtered training queries and incremental ingestion.
BigQuery supports three partitioning types:
TIMESTAMP or DATE column.A column-based, time-partitioned table removes the need to maintain partition awareness separate from filtering on the bound column. Data written to a column-based, time-partitioned table flows automatically to the appropriate partition based on the column value, so training queries that filter on that column naturally prune unrelated partitions.
BigQuery dataset and table configuration uses partitioning, clustering, and data locality to maintain large tables and reduce data load and response time. Partitioning first limits the partitions scanned, while clustering and locality further organize data inside the table so queries read less data per job. Together these configurations increase operational efficiency for repeated training extractions and exploratory queries over the same large datasets.
Cloud Storage is the object store for large container formats that carry training data outside BigQuery, including sharded exports and unstructured-adjacent structured dumps. The documented container choice depends on the consumer: sharded TFRecord files for TensorFlow and Avro files for any other framework. Data flows from many small source records into large container files in Cloud Storage buckets and directories, and from those objects to training workers during batch reads.
Throughput optimization in Cloud Storage depends on combining many small records into large files rather than storing millions of tiny objects. Combining individual records into large files improves both read and write throughput to Cloud Storage because each read streams a large sequential block. Aim for files of at least 100 MB with between 100 and 10,000 shards per dataset, and use Cloud Storage buckets and directories to group the shards for data management and parallel access.
Exam tip: Use sharded TFRecord files when training with TensorFlow and Avro files when training with other frameworks; both must use large files of at least 100 MB with 100 to 10,000 shards.
Agent Platform Feature Store (opens in a new tab) is the central location to create, maintain, share, and serve ML features built on structured data in BigQuery. Feature data lives in a BigQuery table or view, including historical feature data that supports offline training and point-in-time correctness. To use Agent Platform Feature Store, first create an online store instance and then define feature views, after which BigQuery serves as the offline store and the online store serves low-latency requests.
Use a managed Vertex AI Dataset when lineage between data and model or shared reuse across jobs matters because it registers the BigQuery table or Cloud Storage manifest as a versioned input with descriptive statistics and links to custom-trained models through Vertex ML Metadata and Experiments. Skip a managed dataset when training code needs full control over splitting or when lineage is not critical.
Store structured and semi-structured training data as materialized tables in BigQuery instead of views or subqueries and partition large tables so queries scan only matching partitions. Read the tables for training through the BigQuery Storage Read API with a framework-specific connector and wrap the source as a managed tabular dataset to add artifact tracking that links the table to the trained model.
Combine individual media files into large sharded TFRecord files when training with TensorFlow or into sharded Avro files when training with other frameworks such as PyTorch, XGBoost, or scikit-learn. Aim for shards of at least 100 MB with between 100 and 10,000 shards per dataset and group the shards with buckets and directories so a training job can list and stream a coherent prefix in parallel.
Use automatic splitting to let Vertex AI divide the data for AutoML and standard custom training, which preserves reproducibility because the same dataset version yields the same subsets for evaluation and comparison. Use manual splitting when you need deterministic stratification, time-based cutoffs, or custom logic implemented in training code.
Last updated