Creating and Consolidating ML Features in Feature Store

Optimizing feature storage and retrieval for model serving means keeping one consistent definition of each feature while using different storage for training and for inference. In this design, BigQuery (opens in a new tab) and Cloud Storage hold materialized feature data for offline training, and Vertex AI Feature Store (opens in a new tab) serves the same features online for low-latency inference. BigQuery remains the source of truth for all feature data, including historical rows, while the online store holds only the latest values for fast lookup.

1.1Materialize Structured and Semi-Structured Features in BigQuery

BigQuery is the recommended storage for structured and semi-structured feature data, including intermediate processed data. Feature pipelines extract data from operational source systems into storage that is separate from the operational source, then store the result as materialized tables instead of views or subqueries. Materialized tables give maximum read speed because training jobs do not recompute a view on every read, and partitioning large datasets further improves query performance and cost efficiency.

Data flows from BigQuery tables into training through the BigQuery Storage API, which provides high-throughput parallel reads. Teams can wrap a table in a managed tabular dataset backed by Vertex ML Metadata to track artifacts and manage train, test, and validation splits. Framework-specific connectors make the API easier to use from training code, including Dataflow support for preprocessing at scale.

1.2Materialize Unstructured Features in Cloud Storage

Cloud Storage is the recommended storage for image, video, audio, and other unstructured feature data used for model serving. Unstructured objects are stored in large container formats on Cloud Storage, using sharded TFRecord files for TensorFlow workflows and Avro files for other frameworks. Dataflow, using the Apache Beam programming model, can convert raw unstructured data into binary formats such as TFRecord to improve data-ingestion performance during training.

Throughput optimization for Cloud Storage depends on combining many small clips or images into large files rather than storing millions of individual objects. The documented target is files of at least 100 MB, with between 100 and 10,000 shards per dataset, with buckets and directories used to group shards for management. This sharding strategy lets training workers fetch data in parallel from Cloud Storage while keeping file counts manageable for listing and lifecycle operations.

1.3Serve Low-Latency Features from FeatureOnlineStore and Feature Views

Vertex AI Feature Store on Gemini Enterprise Agent Platform is the central repository for creating, maintaining, sharing, and serving machine learning features for online inference. To use Vertex AI Feature Store with structured data, an operator must first create a FeatureOnlineStore instance for optimized online serving and then define feature views that map BigQuery tables or views to servable features. At serving time, traffic flows from the inference application to the feature view in the online store instance, which returns the latest materialized values without running a BigQuery query in the request path.

This design creates two paths from one definition: BigQuery stores all feature data including historical rows for offline training and point-in-time correctness, while the online store instance serves current values through an efficient feature-serving API. For FeatureOnlineStore instances created with optimized online serving, the online store also supports vector similarity search to retrieve approximate nearest neighbors, which are semantically similar entities. That capability lets RAG Engine (opens in a new tab) use Vertex AI Feature Store as a vector database to handle large data volumes with low latency for retrieval-augmented generation.

1.4Maintain Consistency Between Training and Serving Environments

Feature consistency between training and serving means the inference path uses the same feature definitions, transformations, and versioned values that produced the training data. Vertex AI Feature Store maintains that consistency by centralizing feature definitions so multiple teams reuse the same features at scale, with versioning of feature data to preserve lineage between datasets and custom-trained models. Model Registry on Gemini Enterprise Agent Platform complements this by storing, organizing, and versioning model binaries and metadata, so a specific model version can be traced to the feature version it was trained on.

Skew between training and serving is detected in production with Model Monitoring on Gemini Enterprise Agent Platform, which monitors deployed models for training-serving skew and inference drift. When incoming inference data deviates too far from the training baseline, Model Monitoring sends alerts and exposes feature distributions that teams use to decide whether to retrain. This closed loop — materialize once in BigQuery or Cloud Storage, serve the same features online from Vertex AI Feature Store, and monitor for drift — keeps offline training and low-latency online inference aligned as data evolves.

Consolidating and managing feature lineage and versioning means keeping one traceable record for every feature from source data through transformation to the versions used for training and serving. In Gemini Enterprise Agent Platform, Gemini Enterprise Agent Platform Feature Store (Feature Store) (opens in a new tab) provides that consolidated record for feature data, while BigQuery stores the underlying historical data and Vertex ML Metadata links datasets, parameters, metrics, and model artifacts together. This combination preserves source provenance, transformation logic, and version history so teams can reproduce results, share features safely, and meet governance requirements.

2.1Feature Store as the Consolidated System of Record

Gemini Enterprise Agent Platform Feature Store (Feature Store) is the central repository for creating, maintaining, sharing, and serving machine learning features. To use Feature Store with structured data, a team must first create an online store instance and then define feature views, with feature data stored in a BigQuery table or view. The online store instance serves workloads that need low latency, while BigQuery stores all feature data including historical feature data to support offline training and backfills.

Reuse flows from the same Feature Store to both training and serving, so multiple teams consume the same versioned features instead of rebuilding divergent copies. Because the feature view defines the mapping once, offline reads for training and online reads for inference stay aligned. That shared definition is what makes Feature Store the system of record rather than a copy of data in each pipeline.

2.2Source Provenance and Transformation Lineage

Source provenance and transformation lineage in Feature Store tracks where feature data originated, how it was changed, and where the result is stored. Data flows from source systems such as online service logs, devices, or scraped documents into storage that is separate from the operational source and optimized for machine learning training. Transformations are applied with Dataflow for Apache Beam, Managed Service for Apache Spark for Apache Spark, or BigQuery for SQL-based processing, with schema validation and anomaly checks from TensorFlow Data Validation (TFDV) and automated data quality scans in BigQuery.

The lineage record that makes this reproducible centers on three peer elements:

• Source data provenance: the originating system plus the BigQuery dataset or Cloud Storage location holding the extracted, ML-optimized copy.
• Transformation logic: the Dataflow, Spark, or BigQuery processing and validation that converted source data into feature values.
• Version history and pointers: the Feature Store feature view version plus metadata and pointers to data locations held in version control or a separate data catalog.

When provenance, logic, and version pointers are stored together, a later user can trace a served feature value back to its source table, its transformation code, and its historical snapshot. For maximum training speed, materialized data in BigQuery is preferred over views or subqueries, and reads flow through the BigQuery Storage API into TensorFlow, TFX, Dataflow, PyTorch, XGBoost, or scikit-learn tooling.

2.3Versioning Strategy Across Code, Data, Features, and Models

The versioning strategy for lineage assigns each artifact type to the managed service that versions it, with Feature Store owning feature consistency and versioning. Code, data, features, and models each have a different home, so lineage depends on linking those homes through metadata rather than storing everything in one place.

Metadata and pointers to data locations flow from these systems into a version control system such as GitHub or GitLab or into a separate data catalog. Managed tabular datasets and Experiments on Gemini Enterprise Agent Platform (Experiments) (opens in a new tab) integrate with Vertex ML Metadata (opens in a new tab) to log and build linkage across parameters, metrics, and dataset and model artifacts. That linkage connects a specific model version in Model Registry back to the exact feature version and dataset that produced it.

2.4Reproducibility, Sharing, and Governance from Lineage

Reproducibility, sharing, and governance from lineage means automated pipelines and monitoring use the versioned record to repeat work and detect change. Gemini Enterprise Agent Platform Pipelines (Pipelines) automates every step from data preparation to training, evaluation, and deployment, with scheduled pipeline runs for recurring work, execution metrics for workflow monitoring, and reusable pipeline templates for standardized workflows. Experiments and Vertex AI TensorBoard visualize and compare runs by hyperparameters, metrics such as loss and accuracy, model graphs, and embeddings, while the Vertex ML Metadata linkage preserves which dataset and feature version produced each result.

After deployment, Gemini Enterprise Agent Platform Model Monitoring (Model Monitoring) detects data drift, training-serving skew, performance changes, and anomalies in production, including shifts in output quality and safety compliance for generative systems. When monitoring signals drift, teams use the lineage record to decide whether to retrain, which historical feature snapshot to use, and which model version to roll back to or promote. Lineage therefore closes the loop between building, sharing, and governing features over time.

Scalable feature pipelines in Gemini Enterprise (opens in a new tab) transform raw enterprise data into curated, reusable features that serve search, answers, and agentic workflows. Raw data flows from connected systems into dedicated stores, consolidated stores serve applications through a shared serving layer, and transformation logic runs in assistants, actions, and agents under governed observability. Storage, serving, and transformation stay separated so each layer can scale and change without rewriting the others.

3.1Ingesting Raw Data into Curated Data Stores

Gemini Enterprise data ingestion separates connection from storage to keep pipelines scalable and reproducible. Data sources (opens in a new tab) include Google and third-party systems such as Confluence, Jira, Microsoft SharePoint, and ServiceNow, while data stores hold the ingested entities from those systems. Each data source supports a set of entity types, and Gemini Enterprise creates a separate data store for each entity, so one Jira Cloud connection produces distinct stores for issues, attachments, comments, and worklogs.

Data moves through this ingestion path in order:

1. Connect source: The pipeline connects a Google or third-party data source to Gemini Enterprise.
2. Create entity stores: Ingestion creates a collection of data stores representing the ingested data entities.
3. Activate for serving: Each store changes from Creating to Active, and only after reaching Active can the store serve an app.

The serving layer depends on storage: an app must be connected to a data store in order to use the data from it to serve search results, answers, or actions. This dependency isolates upstream connector changes from downstream serving, because new entities add new stores without rewriting existing apps.

3.2Consolidating Features with Apps and Blended Search

A Gemini Enterprise app is the consolidation point where curated features from multiple stores become one serving interface for search results, actions, and agents. The term app can be used interchangeably with the term engine in the context of APIs. Apps have a many-to-many relationship with data stores, so one app can serve many stores and one store can serve many apps.

When multiple data stores are connected to a single app, this is referred to as blended search. In blended search, Gemini Enterprise sends the query to the connected stores, blends the results with those from other connected sources, and displays a comprehensive result. Choose a single-store app when the feature domain is narrow, and choose blended search when the use case requires joint retrieval across fragmented systems such as Drive, Jira, and Slack.

3.3Transforming and Serving Features with Assistants, Actions, and Agents

Transformation and serving in Gemini Enterprise are performed by the assistant, actions, and agents (opens in a new tab) operating on consolidated app data. The in-app chat box, known as the assistant, is grounded with the content in connected data stores and adheres to defined security, privacy, and compliance safeguards. The assistant performs multiple functions, including:

• Generating answers to queries and follow-up questions with citations grounded in enterprise data
• Grounding answers in enterprise data with optional grounding based on external public website data
• Analyzing and summarizing uploaded content, such as PDFs and images, and answering follow-up questions
• Generating images based on text prompts and creating reports with streamed research
• Creating multimodal outputs that synthesize information across connected stores

Actions extend the pipeline from read-only retrieval to write-back transformation, allowing Gemini Enterprise to create or update records in real time. For example, traffic flows from the assistant to Google Calendar to create events or to Jira Cloud to edit issues, while basic data stores without enabled actions focus only on search and retrieval. Agents automate multi-step processes over the same grounded data by connecting to enterprise data and tools to analyze information and generate new content, with prebuilt agents discovered in the Agent Gallery and custom agents built in Workflow Builder.

3.4Governing Pipelines with Analytics and Observability

Governance for scalable pipelines combines usage analytics, real-time telemetry, and centralized policy control across the agent lifecycle. Analytics gives insight into usage trends, search quality, and end-user engagement through an interactive dashboard powered by Looker. Observability provides real-time telemetry across enterprise search and agent workflows, where teams inspect end-to-end execution request traces, trace spans, and usage audit logs to diagnose latency, monitor agent step execution, and analyze system activity.

The control plane manages scale through the Agent Platform, Agent Registry, and Agent Gateway. The Agent Registry views, manages, versions, registers, and monitors agents, while the Agent Gateway configures routing for Gemini Enterprise instances with US, EU, or Global bindings. The Cloud Observability suite collects traces, logs, and metrics for latency and token usage from Agent Platform, Agent Gateway, and Model Armor using OpenTelemetry protocols, and policy enforcement applies IAM and Model Armor policies with audit logging and Security Command Center integration for threat detection.

Key Facts

• BigQuery is the source of truth for structured feature data for offline training, while a FeatureOnlineStore instance and feature views serve the latest values for low-latency online inference.
• Cloud Storage holds unstructured feature data in sharded TFRecord files for TensorFlow and Avro files for other frameworks, with a target of files of at least 100 MB and 100 to 10,000 shards per dataset.
• Vertex AI Feature Store centralizes feature definitions and versioning, while Model Registry versions model binaries and metadata and Artifact Registry versions code artifacts.
• Model Monitoring detects training-serving skew and inference drift and exposes feature distributions used to decide whether to retrain.
• A Gemini Enterprise app must connect to at least one Active data store before it can serve search results, answers, or actions, with blended search combining results from multiple stores.