Prepare and test your skills
Prepare and test your skills
Vertex AI Feature Store (opens in a new tab) is a managed, cloud-native service for centralized feature management and secure feature serving in machine learning workflows. A feature is a measurable attribute of an entity used to train a model or make a prediction, and feature management is the process of creating, maintaining, sharing, and serving those features from one central location. Vertex AI Feature Store keeps offline feature data in BigQuery (opens in a new tab) and serves selected features online at low latency, while Identity and Access Management (IAM), Virtual Private Cloud (VPC) Service Controls, and encryption protect data integrity during integration.
Centralized feature management with Vertex AI Feature Store means using one metadata and serving layer over feature data stored in BigQuery. BigQuery serves as the offline store, so teams do not copy or import feature data into a separate offline store for training or serving. BigQuery retains all feature data, including historical data, which allows offline retraining while the same source supports online predictions. This central model reduces duplication because one BigQuery source can feed both training and online inference, and different teams can reuse the same registered features instead of rebuilding them. Vertex AI Feature Store also tracks feature metadata in Knowledge Catalog and supports embeddings for vector similarity search.
Centralized management supports sharing, reuse, and low-latency serving from the same source:
The feature registry, online store, and feature view are the three parts that divide feature management by role. The feature registry is the central interface that records which BigQuery tables or views are available for online inference. A feature group corresponds to one BigQuery source table or view, while a feature view is a logical collection of feature columns materialized from that source into an online store and refreshed periodically from BigQuery. A FeatureOnlineStore instance is the online serving cluster that holds one or more feature views, and a feature record is the collection of feature values for one entity at a point in time.
Data flows in order from BigQuery to online inference:
A four-step process flow showing BigQuery source data registered and synced through a FeatureOnlineStore and feature view to serve online predictions, offline training data, and vector search for RAG.
FeatureOnlineStore instance, choosing optimized online serving for ultra-low latency or Bigtable online serving for high data volumes.Because the feature view depends on both the BigQuery source and the online store, permissions and refresh must be correct on both sides or serving will fail or become stale.
Feature serving in Vertex AI Feature Store is the process of exporting or fetching feature values, with online and offline paths serving different consumers. Online serving retrieves the latest values for a small subset of entities for real-time inference, while offline or batch serving exports high volumes of current and historical data with timestamps for training and analysis. For FeatureOnlineStore instances created with optimized online serving, the service can also perform vector similarity search to return approximate nearest neighbors, which are semantically similar entities based on embedding distance. When Vertex AI Feature Store is used as a vector database for RAG Engine (opens in a new tab), embedding vectors stored in BigQuery flow into the online store and RAG Engine queries them with low latency.
| Serving type | Data retrieved | Used when |
|---|---|---|
| Online serving | Latest feature values for a subset of entities from the online store | The application needs low-latency features for live prediction |
| Offline serving | High volumes of current and historical feature data from BigQuery | The workflow needs full datasets for model training or retraining |
Exam tip: Choose optimized online serving for ultra-low-latency needs and Bigtable online serving for high data volumes; both use BigQuery as the offline source of truth.
Data security for Vertex AI Feature Store combines IAM access control, service-agent permissions, network boundaries, and encryption. Support for resource-level IAM policies on featurestore and entityType resources allows administrators to control who can access specific stores and entity types, rather than granting broad project access. Each feature view uses a service agent that must be granted access only to the specific BigQuery source data it needs, so the sync process can read from BigQuery without exposing unrelated datasets. The simplest access pattern stores raw data and Vertex AI resources in the same Google Cloud project, while cross-project designs use IAM to grant controlled access between the data project and the Vertex AI project. VPC Service Controls create a security perimeter around supported Vertex AI services to limit data exfiltration, and customer-managed encryption keys (CMEK) provide customer control over encryption for RAG Engine integrations.
Data sourcing and ingestion for Vertex AI (opens in a new tab) is the process of extracting data from operational source systems and converting it into formats and storage optimized for machine learning training. BigQuery is the primary source for structured and semi-structured data, while Cloud Storage is the primary source for image, video, audio, and other unstructured data. Data flows from the source systems to BigQuery or Cloud Storage, then into a Vertex AI managed dataset that links the data to custom-trained models.
Preparing training data from source systems means separating machine learning data from day-to-day operational systems. Source data can originate from online service logs, local devices, or documents scraped from the web, but it must be extracted and reformatted before training. The reformatted copy is stored in BigQuery (opens in a new tab) or Cloud Storage, not in the operational database, so training workloads do not interfere with production systems. This prepared copy then becomes the input for preprocessing pipelines and Vertex AI managed datasets.
Storing structured and semi-structured data in BigQuery means using BigQuery as the recommended storage for all tabular data, including intermediate processed data. BigQuery projects should follow standard table organization, and training data should be stored as materialized tables instead of views or subqueries for maximum speed. Data flows out of BigQuery to training frameworks through the BigQuery Storage API, and for artifact tracking the table can be referenced by a managed tabular dataset. Vertex AI Feature Store also depends on BigQuery, because it stores feature data in a BigQuery table or view and keeps historical data for offline work.
| Training framework | Google Cloud tool to read BigQuery |
|---|---|
| TensorFlow with Keras | tf.data.dataset reader for BigQuery |
| TensorFlow Extended (TFX) | BigQuery client |
| Dataflow with Apache Beam | BigQuery I/O Connector |
| Other frameworks such as PyTorch, XGBoost, or scikit-learn | BigQuery import for ONNX models |
Storing unstructured data in Cloud Storage means holding image, video, audio, and other unstructured data in large container formats. Large container formats combine many small files into fewer large files to improve read and write throughput. Use sharded TFRecord files when training with TensorFlow, and use Avro files for other frameworks. Aim for files of at least 100 MB, with between 100 and 10,000 shards, and use Cloud Storage buckets and directories to group the shards for data management.
Exam tip: Choose BigQuery for structured and semi-structured data, and choose Cloud Storage in large sharded files for image, video, audio, and unstructured data.
Processing and importing into managed datasets is the step that connects raw storage to a Vertex AI managed dataset that can be tracked and split for training. The recommended pipeline depends on data size, framework, and transformation needs. A Vertex AI managed dataset provides descriptive statistics and automatic or manual splitting into train, test, and validation sets. Managed datasets are optional when the team wants full control over splitting in training code or when lineage between data and model is not critical.
Data formatting and preprocessing requirements in Vertex AI define how raw source data must be converted into compatible storage, formats, and schemas before training. Data flows from operational source systems to machine learning-optimized storage that is separate from the source, and then through preprocessing pipelines to training. The correct format depends on the data type and the model objective, including whether the workflow uses BigQuery ML, AutoML, or custom training.
Preparing training data for Vertex AI (opens in a new tab) is the process of extracting data from source systems and converting it to the format and storage optimized for training. Source data can originate from online service logs, images from local devices, or documents scraped from the web. That data must not be trained directly from the operational source, and it must be moved to separate machine learning storage in a compatible schema. For AutoML, the data and problem must match one of the supported formats, data types, and model objectives.
Preprocessing tools in Vertex AI transform raw data into clean, validated features that match the training framework and model objective. BigQuery (opens in a new tab) handles SQL-based preprocessing for structured data, while Dataflow built on Apache Beam handles large-volume batch and streaming transformations. Managed Service for Apache Spark provides serverless Spark processing when a team already has Spark code and skills. The choice depends on scale and expressiveness, with BigQuery transformations as the most cost-effective method for custom models that can be expressed in SQL.
Data quality checks validate schemas and detect anomalies before training:
For generative AI, quality also means accuracy, relevance, diversity, and alignment with required output characteristics. When real-world data is insufficient or imbalanced, BigQuery DataFrames can generate synthetic data to improve robustness and prompt diversity. Labels for fine-tuning or reinforcement learning from human feedback must accurately reflect quality, relevance, and safety of generated outputs.
Formatting structured and semi-structured data in BigQuery means storing all raw and intermediate processed data as materialized BigQuery tables. Materialized tables provide maximum training speed compared to views or subqueries, and partitioning large datasets improves query performance and cost efficiency. Data flows from BigQuery to training through the BigQuery Storage API, and a managed tabular dataset can provide artifact tracking. If the model is built with BigQuery ML, use the transformations built into BigQuery ML for preprocessing, and if the model is built with AutoML, use the transformations built into AutoML.
| Training framework | Tool to read from BigQuery |
|---|---|
| TensorFlow for Keras | tf.data.dataset reader for BigQuery |
| TensorFlow Extended (TFX) | BigQuery client |
| Dataflow | BigQuery I/O Connector |
| Other frameworks such as PyTorch, XGBoost, or scikit-learn | BigQuery import path |
Formatting unstructured image, video, and audio data in Cloud Storage means combining many small files into large sharded container files. Large container formats improve read and write throughput to Cloud Storage during training. Files should be at least 100 MB, with between 100 and 10,000 shards, and shards should be grouped with Cloud Storage buckets and directories for data management. The container format depends on the training framework, because TensorFlow ingestion is optimized for TFRecord while other frameworks use Avro.
Exam tip: Use sharded TFRecord files for TensorFlow and Avro files for any other framework, with files of at least 100 MB and between 100 and 10,000 shards.
Managed datasets in Vertex AI are dataset objects hosted by Vertex AI that link preprocessed data to custom-trained models through Vertex ML Metadata. A managed dataset provides descriptive statistics and supports automatic or manual splitting into train, test, and validation sets, with the training set typically holding about 80% of the data. The managed dataset depends on the preprocessed source in BigQuery or Cloud Storage, and the trained model then depends on the managed dataset for lineage. Managed datasets are not required when a team wants full control over splitting in training code or when lineage between data and model is not critical.
Vertex AI Feature Store is a managed, cloud-native service for centralized feature management and secure feature serving that uses BigQuery as the offline store. Teams store offline feature data in a BigQuery table or view without copying it to a separate offline store, register BigQuery sources as feature groups and features, and serve features for online predictions at low latency through an online store instance and feature views.
In Vertex AI Feature Store, online serving retrieves the latest feature values for a small subset of entities for real-time inference, while offline serving exports high volumes of current and historical data for training and analysis. Online data comes from the online store and offline data comes from BigQuery with timestamps.
Use BigQuery for structured and semi-structured tabular data and use Cloud Storage for image, video, audio, and other unstructured data. Structured training data should be stored as materialized tables in BigQuery, while unstructured data should be held in large container formats that combine many small files into fewer large files.
Combine many small files into fewer large sharded files of at least 100 MB with between 100 and 10,000 shards grouped with buckets and directories. Use sharded TFRecord files when training with TensorFlow and use Avro files for other frameworks.
An enterprise retail organization is preparing historical transaction data to train a tabular demand forecasting model in Vertex AI to predict daily sales per product across multiple store locations. The data engineering team is using Apache Beam on Dataflow to build a scalable data preprocessing pipeline that extracts, cleans, and structures raw records before importing them into Vertex AI.
Which data formatting and schema design must the Dataflow pipeline implement to produce a valid dataset for the forecasting model objective?
Last updated