Scalable feature pipelines in the Gemini Enterprise Agent Platform transform raw enterprise data from external platforms into structured data stores that feed downstream agentic workflows and analytical models. Raw information originates in source systems such as Salesforce, Jira, ServiceNow, and Google Workspace, flows across defined network boundaries through prebuilt connectors, and lands in dedicated entity-level data stores. The platform automates data ingestion by executing scheduled full extractions and incremental syncs to maintain up-to-date data representations.
Data synchronization pipelines manage the flow of enterprise information from external applications into curated data stores using scheduled full and incremental processing jobs. A data pipeline begins by establishing an authenticated connection to a data source, after which administrators configure the Full sync frequency and the Incremental data sync frequency. To optimize processing workloads, administrators can select custom options under the full sync settings to execute full entity syncs and full identity syncs on separate independent schedules. When ingestion initiates, the connector transitions through explicit lifecycle states, moving from Creating to Running as synchronization starts, and finally reaching Active once ingestion completes and the connector waits for its next scheduled cycle.
Entity-level data stores partition ingested source data by specific business entities to structure raw enterprise records into distinct data schemas. When a source like Jira Cloud or Salesforce is connected, the platform provisions a separate data store for each entity type, such as issues, attachments, comments, or custom entities formatted with the custom_object_name__c naming convention. During pipeline definition, administrators can apply custom URL field overrides to map entity search destinations directly to specific table columns, such as redirecting an account record to a custom website attribute instead of the standard object page. Data ingress routes through either a Public destination type requiring only a server login URL, or a Private destination type that routes traffic securely through a configured service attachment, base domain name, and private login endpoint.
Enterprise apps, also referred to as engines in API contexts, consolidate multiple individual data stores into unified query and feature serving layers. Apps maintain a many-to-many relationship with data stores, allowing a single app to ingest records from multiple disconnected sources in a configuration known as blended search. Developers query consolidated data and extract curated information programmatically through the Discovery Engine API, which supports complex multi-turn search queries and metadata field filtering across structured and unstructured data. Capacity and indexing scalability are governed by platform editions, which allocate 30 GiB of pooled storage per user per month in Gemini Enterprise Standard, 75 GiB in Gemini Enterprise Plus, and 2 GiB in Gemini Enterprise Frontline.
The Agent Platform Feature Store provides a centralized, managed service for feature lineage and versioning by acting as a metadata layer over a BigQuery data source. In this architecture, the raw or transformed feature data resides permanently in BigQuery tables, leveraging BigQuery's data management capabilities for historical storage and batch processing. The Vertex AI Feature Store service itself does not duplicate this offline data; instead, it maintains metadata registries, enables online serving via Bigtable, and orchestrates the synchronization between the offline store and the online serving layer. This separation means feature definitions, transformation logic, and version history are managed within Feature Store, while the actual data values are owned and versioned within BigQuery.
Knowledge Catalog is the fully managed metadata service integrated with Vertex AI Feature Store to establish comprehensive feature lineage. It tracks metadata such as the source data's provenance (for example, the specific BigQuery table and column), the sequence of transformation logic applied, and the creation history. This lineage is crucial for reproducibility; when a feature is used for model training or online inference, data scientists can trace back through Knowledge Catalog to understand exactly how that feature value was derived, which source data was used, and under what parameters. This system supports governance policies by providing an auditable trail of data operations.
Feature versioning within the Feature Store manages the lifecycle of a feature definition. A feature enters a draft state during development within a feature registry. Once published, it becomes available for use in training pipelines and online serving. When the underlying logic or schema changes, a new version of the feature is created, while previous versions remain accessible for existing models to ensure serving consistency and prevent training-serving skew. The service manages these versions, allowing consumers to specify which version of a feature they wish to retrieve, either for point-in-time correct batch training or for low-latency online prediction. This explicit versioning is key to managing breaking changes and facilitating A/B testing of new feature logic.
The consolidated lineage and versioning system enables operational best practices for team collaboration and compliance. Cross-functional teams can discover, share, and reuse features through the feature registry, reducing duplicate engineering efforts. IAM policies and resource-level permissions control which personasāsuch as data engineers, ML researchers, or DevOpsācan create, view, or serve specific feature versions. Furthermore, the lineage data feeds into monitoring for data drift and feature value anomalies; by comparing statistical distributions of feature values across versions or over time, the system can alert teams to potential model degradation or schema evolution, triggering model retraining workflows. This end-to-end traceability from raw data in BigQuery to a served prediction is foundational for scalable, reproducible, and governable ML pipelines in the Gemini Enterprise Agent Platform.
Vertex AI Feature Store is a managed service that provides centralized feature management for machine learning workflows, allowing teams to store, discover, share, and serve ML features from a unified repository. The platform supports two primary storage approaches: offline storage using BigQuery for structured and semi-structured feature data, and online storage using Bigtable for low-latency serving scenarios. When using BigQuery as the source, Vertex AI Feature Store acts as a metadata layer that provides online serving capabilities without requiring data to be copied to a separate offline store, taking advantage of BigQuery's built-in data management capabilities. Teams can also store image, video, audio, and unstructured data in Cloud Storage using large container formats like sharded TFRecord files for TensorFlow or Avro files for other frameworks, with recommended file sizes of at least 100MB per shard to optimize read and write throughput.
The feature store is optimized for ultra-low latency serving, enabling real-time feature retrieval for online predictions through Bigtable online serving. When you create a feature online store instance, you can configure autoscaling with parameters such as minimum and maximum node counts and CPU utilization targets, or allocate a fixed node count for predictable performance. The online serving layer maintains the latest timestamp values of features to efficiently handle serving requests, while the offline storage retains historical data for training purposes and supports point-in-time lookups to fetch feature values as they existed at specific historical moments. For RAG Engine integration, you can use Vertex AI Feature Store as a vector database to handle large data volumes with low latency, improving the performance and scalability of retrieval-augmented generation applications.
Training-serving skew occurs when the feature data distribution used in production differs from what was used during model training, leading to performance discrepancies between training and serving environments. Vertex AI Feature Store mitigates this by ensuring that a feature value is imported once into the featurestore and that same value is reused for both training and serving, eliminating the risk of different code paths generating different feature values. The platform provides point-in-time lookups that fetch only the feature values available before a prediction time, preventing data leakage by ensuring training data doesn't include future information. Additionally, the feature store helps detect drift by constantly tracking the distribution of feature values imported over time, alerting teams when significant changes require model retraining.
Blended search allows a single app to consolidate and query records from multiple disconnected data sources simultaneously, while a regular query typically accesses only one data store at a time. This enables unified search across entities from different platforms like Salesforce and Jira.
The platform supports point-in-time lookups that fetch only the feature values that were available before a specific historical prediction time. This prevents data leakage by ensuring training data doesn't include future information that wouldn't be available at prediction time.
Structured and semi-structured feature data typically use BigQuery for offline storage and Bigtable for online serving. Unstructured data such as images, video, audio, and text can be stored in Cloud Storage using large container formats like sharded TFRecord or Avro files, with recommended file sizes of at least 100MB per shard.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts