Governing feature data quality and access in the Agent Platform Feature Store involves validating data schemas, monitoring for problems like data drift, and controlling who can see or change feature data. These practices keep ML models reliable and secure when they use features for real-time predictions.
The Agent Platform Feature Store relies on BigQuery to define and validate the structure of feature data. When you create a FeatureView resource, you point it to a BigQuery table or view, and the schema of that source becomes the structure of your feature data. The service watches for data drift, which happens when the values in a feature change significantly over time, and it also detects staleness, which occurs when feature data becomes outdated. Monitoring for these problems helps catch when features are degrading in quality, which signals that models using those features may need retraining. This monitoring protects against training-serving skew, where the data used to train a model differs from the data the model sees in production.
Access to feature data in the Agent Platform Feature Store is controlled through Identity and Access Management (IAM). The service offers predefined roles for different jobs: the featurestoreDataViewer role lets ML researchers read feature values, the featurestoreDataWriter role lets data engineers write data, and the featurestoreAdmin role gives full administrative control. You can apply these roles at the project level or more precisely at the individual featurestore or entity type level. This flexibility means you can let DevOps teams manage infrastructure without seeing sensitive data while giving data scientists access only to the specific features they need.
The Agent Platform Feature Store uses a service agent (service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com) to read data from BigQuery during feature synchronization. By default, this agent can access data in the same project. If your feature data lives in a different project, you must grant the service agent the BigQuery Data Viewer role in that external project. For stronger security, you can configure a FeatureView to use its own dedicated service account instead of the project-wide default. This dedicated account can then receive precise permissions only to the specific BigQuery tables that particular feature view needs.
All interactions with the Agent Platform Feature Store are recorded in Cloud Audit Logs, which capture API calls, data access events, and configuration changes to featurestores, entity types, and features. You can send these logs to Cloud Logging for real-time analysis and alerting, or export them to BigQuery for long-term storage and security review. This audit trail supports forensic investigations, helps with debugging, and proves compliance with data governance policies.
Architecting feature storage and serving pipelines means designing how data flows into the Feature Store, how it gets served to models, and how versions are managed to keep training and production consistent.
The Agent Platform Feature Store is a managed service that stores, discovers, shares, and serves ML features for online model inference. It works as a metadata layer sitting on top of BigQuery, giving low-latency feature retrieval without copying data into separate offline stores. The service connects with Knowledge Catalog to track feature metadata and supports embeddings for vector similarity searches. It offers both online serving for real-time predictions and offline serving for batch training and historical lookups.
The Feature Store organizes data in a hierarchy of components. A featurestore is the top-level container holding entity types, features, and their values, letting permitted users share features without extra engineering work. Entity types represent the categories being tracked, such as customers, products, or users, and each entity type contains features that describe those entities. A feature view is a logical collection of features pulled from a BigQuery source into an online store, periodically refreshing the data.
Data pipelines fill the Feature Store from batch and streaming sources. For batch import, you can load feature values from Cloud Storage files (CSV or Avro), BigQuery tables, or BigQuery views using import jobs. The import job has a disableOnlineServing flag that lets you control whether data goes to the online store—disabling it during backfills prevents load on online serving nodes and speeds up import. For streaming, you publish messages to Pub/Sub and use the Apache Beam enrichment transform with VertexAIFeatureStoreEnrichmentHandler to add feature values to streaming data in real time.
Online serving gets the latest feature values for real-time predictions. Bigtable online serving handles high data volumes with low latency, while Optimized online serving gives ultra-low latency for latency-sensitive applications. When you create an online store, you pick the storage backend and define feature views pointing to BigQuery tables. The sync operation pulls data from BigQuery into the online store, and you should schedule sync jobs during off-peak times to reduce performance impact.
The Feature Store offers two scaling options: autoscaling and fixed node count. Autoscaling adjusts the number of online serving nodes based on traffic, scaling out when demand rises and scaling in when it falls, which saves money for variable workloads. Fixed node count keeps the same number of nodes regardless of traffic, giving predictable costs and performance. After adding nodes through either method, the online store needs up to 20 minutes to rebalance data before you see performance improvements. If you expect sudden traffic spikes, set minNodeCount high enough to handle them, since autoscaling cannot react quickly enough for short-lived bursts.
Managing feature versions keeps training and serving consistent, which prevents training-serving skew. The Feature Store supports point-in-time lookups that fetch historical data for training by retrieving only values available before a specific time, preventing data leakage where future information would leak into training data. Each feature record includes a timestamp showing when values were generated, and the system serves the latest non-null value based on that timestamp. When backfilling historical data, disable online serving in the import job because backfilling does not include the latest values needed for online serving—writing only to the offline store eliminates load on online nodes and improves performance.
Training-serving skew happens when the feature data distribution in production differs from training, causing model performance problems. The Feature Store solves this by importing a feature value once and using that same value for both training and serving, eliminating the risk of different code paths producing different values. Without a featurestore, separate code for training versus serving can create inconsistencies. The Feature Store also tracks feature value distribution over time to detect drift, which signals when models may need retraining.
Each feature view uses a service account to access BigQuery during sync operations. By default, a feature view uses the project's default service account, meaning anyone who can create a feature view can access the underlying BigQuery data. Alternatively, you can configure a feature view to use a dedicated service account that Vertex AI generates uniquely for that view, letting you restrict access to specific BigQuery datasets. The service account receives the BigQuery Data Viewer IAM role by default for read access during sync.
Optimizing feature retrieval and serving performance means configuring how features are stored, cached, and delivered to minimize the time it takes for models to get the data they need for predictions.
Online feature retrieval returns the latest feature values for a specific ENTITY_ID from the online store. Client systems can specify a FORMAT parameter to control how the response is structured. The system supports JSON key-value pairs and protocol buffer Struct format. The choice matters because proto Struct format does not support the bytes feature value type—when client applications need bytes values, they must request JSON format to receive the data correctly.
Feature views serve data online by keeping synchronized records against the underlying storage. Successful sync cycles ensure the online serving layer has the most current feature values for lookup queries. If synchronization fails, subsequent feature retrieval requests can encounter errors that prevent inference systems from getting the features they need.
Querying relational or non-relational databases directly causes high latency that gets worse as the number of entities grows. In-memory key-value caches like Memcache provide much faster lookups than full database scans for real-time applications. However, relying only on in-memory stores creates reliability risks because cache entries can be evicted anytime and service interruptions can break data consistency and availability.
Administrators use Cloud Monitoring to watch data throughput and find bottlenecks across serving and data transfer components. Bottlenecks can come from network bandwidth limits, WAN capacity constraints, or file system throughput saturation. When throughput levels off below target rates without hitting network limits, teams must check hardware bus limits, storage media types, and concurrent agent resource limits to restore performance.
StudyCerts
StudyCertsfeaturestoreDataViewer, featurestoreDataWriter, and featurestoreAdmin control access at project or resource levels, while service agents handle data access during sync operations.Bigtable online serving handles high data volumes with low latency, while Optimized online serving provides ultra-low latency for applications where every millisecond matters. Choose Bigtable when you have large feature sets; choose Optimized when latency is the primary concern.
The Feature Store imports each feature value once and serves that same value for both training and serving, eliminating the risk of different code paths generating different values. Point-in-time lookups also ensure training data only uses values available at the prediction time.
Disable online serving (disableOnlineServing flag) during backfill jobs that import historical data, because backfills do not include the latest values needed for online serving. This reduces load on online serving nodes and improves import performance.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills