Prepare and test your skills
Prepare and test your skills
Optimizing feature retrieval and serving performance is the practice of tuning how lookup and scoring requests flow from Spanner to Agent Platform to minimize latency and avoid overload. Requests that carry large payloads such as text fields are subject to a maximum token limit in a single request, so oversized batches fail instead of serving slowly. Performance tuning therefore balances batch size, concurrent request count, and continuous monitoring of the serving layer.
Request batching in Spanner controls how many rows are packed into each API request sent to Agent Platform. The default_batch_size setting defines this packing for a registered model, and smaller batches reduce the token and byte size of each request. For GoogleSQL databases, an oversized batch returns an INVALID_ARGUMENT error with a "Request is too large" message. Reducing default_batch_size resolves that error by splitting the same workload across more, smaller requests from Spanner to Agent Platform.
The @{remote_udf_max_outstanding_rpcs} query hint controls the number of concurrent API requests sent from Spanner to Agent Platform. Increasing the hint value raises parallelism and can lower end-to-end latency when the Spanner instance has headroom, while decreasing the value throttles load on both systems. For GoogleSQL databases, using this query hint overrides the default_batch_size configured for the model. Increasing the limit also increases CPU and memory usage of the Spanner instance, so concurrency tuning trades serving speed against Spanner resource consumption.
| Control | What it sets | When to adjust it |
|---|---|---|
default_batch_size | Number of rows packed into each request from Spanner to Agent Platform | Decrease it when GoogleSQL returns INVALID_ARGUMENT with "Request is too large" |
@{remote_udf_max_outstanding_rpcs} | Number of concurrent requests sent from Spanner to Agent Platform, overriding default_batch_size | Increase it to add parallelism, decrease it to reduce Spanner CPU and memory pressure |
Exam tip: Reduce default_batch_size to fix "Request is too large" failures, but tune @{remote_udf_max_outstanding_rpcs} to control concurrency — raising concurrency speeds serving but raises Spanner CPU and memory usage.
The system insights dashboard is the monitoring surface for requests sent from Spanner to Agent Platform. The dashboard exposes the signals needed to identify bottlenecks in the serving layer:
Monitoring these three signals together connects cause to effect. Rising network bytes with latency spikes points to batches that are too large, while high request counts with elevated Spanner CPU and memory points to concurrency that is too high. Adjust batch size first when payload size is the limiter, then adjust outstanding RPCs to match the capacity of the Spanner instance.
Agent Platform Feature Store (opens in a new tab) is a managed, cloud-native service that centralizes machine learning feature data in BigQuery for both offline training and low-latency online inference. Data flows from batch and streaming sources into BigQuery tables or views, then from BigQuery into a FeatureOnlineStore instance through FeatureView sync operations, and finally from the online store to real-time applications during prediction. This two-layer design keeps BigQuery as the durable source of truth including historical data, while the online store holds materialized, low-latency copies optimized for serving.
Feature storage architecture in BigQuery uses BigQuery tables or views as the authoritative feature data source behind every feature view. A typical setup follows an ordered sequence where each step depends on the previous one. First, source data is extracted from operational systems and stored in BigQuery separate from the operational source, with structured and semi-structured data kept as materialized tables for maximum read speed. Then feature groups and features are registered to catalog those BigQuery sources and preserve lineage. Then a FeatureOnlineStore instance is created to host online serving capacity. Then FeatureView resources are defined to select the subset of columns and rows needed for serving. Only after the feature view exists can data flow from BigQuery into the online store through a sync operation.
BigQuery stores all feature data, including extensive historical records that enable offline use, while the online store serves only the latest values needed for inference. For training speed, materialized data is preferred over views or subqueries, and reads use the BigQuery Storage API for high throughput. Unstructured inputs such as images, video, and audio remain in Cloud Storage in large sharded container formats, with at least 100 MB per file and between 100 and 10,000 shards grouped by buckets and directories. This separation lets tabular features benefit from BigQuery processing while large binary data benefits from Cloud Storage throughput.
Ingestion pipelines populate the Feature Store by landing batch and streaming data in BigQuery and then syncing that BigQuery source into the online store. Batch pipelines use tools such as Dataflow with the BigQuery I/O connector, TensorFlow I/O or TFX BigQuery clients, and scheduled transformations to append or overwrite feature tables on a cadence. Streaming updates land in BigQuery first, and the online store is then refreshed from BigQuery so training and serving observe the same source values. The feature view depends on the BigQuery source during every sync, and the serving application depends on the feature view, so freshness at inference is bounded by both the BigQuery landing latency and the feature view sync frequency.
Processing choices trade off latency, cost, and complexity for tabular data preparation. BigQuery is used to process tabular data with SQL, Dataflow is used for continuous beam-style transforms, and Managed Service for Apache Spark is used for serverless Spark processing, with managed datasets and Vertex ML Metadata tracking artifacts and lineage. Sync frequency must not exceed the rate at which the BigQuery source actually changes, because more frequent syncs consume BigQuery resources without improving freshness. Complex aggregations inside the feature view query increase sync duration and cost, so production views select only the columns needed for online serving and use a simple SELECT * pattern to estimate volume before adding logic.
A FeatureView is the materialized serving definition that selects feature data from BigQuery and controls how it is loaded into a FeatureOnlineStore for low-latency lookup. The FeatureOnlineStore provides the serving capacity and scaling boundary, while the FeatureView defines the source query, sync mode, and service identity used to read BigQuery during sync. During online serving, requests flow from the real-time application to the online store endpoint, which returns the latest feature values keyed by entity ID without querying BigQuery on the critical path. During offline serving, training and batch inference read historical feature rows directly from BigQuery, which preserves full history that the online store does not retain.
Online serving is configured as Bigtable online serving or Optimized online serving, and the choice determines latency, network path, and retrieval capability. Bigtable online serving hosts materialized features in Bigtable for high-throughput key lookups and is required for continuous sync and for serving the latest values including nulls when combined with dense feature groups and scheduled sync. Optimized online serving exposes a dedicated endpoint for feature lookups, either through a public endpoint domain name or through a Private Service Connect endpoint that keeps traffic inside private network boundaries. FeatureOnlineStore instances created with Optimized online serving can also support vector similarity search that returns approximate nearest neighbors, which lets retrieval-augmented generation applications fetch semantically similar chunks keyed by embeddings stored as repeated float fields alongside identifiers such as corpus, file, and chunk IDs.
Data sync modes determine when data moves from the BigQuery source into the online store and what guarantees hold for training-serving consistency. Scheduled sync runs on a timetable and re-materializes the feature view from BigQuery, while continuous sync refreshes the online store whenever new BigQuery data arrives. Consistency between training and serving depends on using the same BigQuery source for both paths: offline training reads point-in-time history from BigQuery, and online inference reads the synced latest values from the online store that were derived from that same source.
| Sync mode | Trigger | Key requirements | Key limitations |
|---|---|---|---|
| Scheduled data sync | Runs on a configured schedule per feature view | Works with Bigtable or Optimized online serving; supports BigQuery tables or views; supports dense feature groups for null-inclusive serving | Incurs BigQuery processing cost on each run; requires staggering and off-peak scheduling at scale |
| Continuous data sync | Refreshes when BigQuery source data is updated | Requires Bigtable online serving; requires feature view linked to feature groups and features; requires BigQuery source in eu, us, or us-central1; requires BigQuery table source, not a view | Feature view cannot be updated after selection; syncs only new records, not updates or deletes in BigQuery |
Exam tip: Continuous sync replicates only newly inserted BigQuery rows and never replicates updates or deletes, while serving the latest values including nulls requires a dense feature group with dense set to true plus Bigtable online serving plus scheduled sync.
Point-in-time correctness prevents leakage when time-sensitive features are used for training in BigQuery ML. Feature tables include a timestamp column that records the value as it existed at a given time, and queries use ML.FEATURES_AT_TIME or ML.ENTITY_FEATURES_AT_TIME to enforce a cutoff so training never sees future data. Those functions appear in the query_statement clause of CREATE MODEL for training and in table-valued functions such as ML.PREDICT for batch serving. When models are registered in Agent Platform, Vertex AI Feature Store extends this pattern by managing the same BigQuery feature tables for low-latency online serving and for offline historical retrieval.
Identity, cost, and operational controls for sync govern how each feature view reads BigQuery and how sync jobs are scheduled and observed. Each feature view uses a Google Cloud service account to read its BigQuery source during sync, which places an identity trust boundary between the Feature Store control plane and customer feature data. Vertex AI Feature Store grants the BigQuery Data Viewer role to that service account for read access to the source. By default the feature view uses the project-level service account, so any user with permission to create feature views in the project can indirectly read the underlying BigQuery data. Alternatively the feature view can be configured with FeatureView.service_agent_type to use a dedicated service account, which generates a unique service account email per feature view that can be retrieved with featureViews.list or featureViews.get and then granted least-privilege access or shared with additional users.
Sync operations move through an observable lifecycle that supports automation and cost control. A sync starts for a named feature view and returns a feature view sync ID, then transitions through Running to either Complete or Failed, with an error message on failure. Automated pipelines poll that status, for example every 30 seconds, until completion or until an operator-defined timeout expires, where timeout stops the waiter but does not stop the underlying sync. Cost and performance during scheduled sync are controlled through several coordinated practices:
max_node_count on the online store high enough to absorb load spikes during sync.Agent Platform Feature Store (opens in a new tab) governance ensures that features served for online inference are valid, fresh, traceable, and accessible only to authorized callers. Governance spans the full path from the BigQuery source table or view through feature groups and feature views to the FeatureOnlineStore instance that serves low-latency reads. Quality controls validate schema at ingestion, monitoring detects drift and staleness during serving, and access controls plus lineage preserve security and reproducibility.
Centralized feature management with BigQuery as source of truth creates, maintains, shares, and serves machine learning features from a central location. Feature data lives in a BigQuery table or view, including historical feature data for offline training and point-in-time correctness, while an online store instance plus defined feature views serve the latest values online. Data flows from the BigQuery source to registered feature groups and features, then to the online store through sync, then from the feature view to the real-time application at prediction time. To use the store for structured data, first create an online store instance and define feature views; BigQuery remains the system of record and the online store is the low-latency serving copy.
Schema validation and ingestion quality prevent malformed or unexpected feature values from reaching online serving. Validation starts in preprocessing with Dataflow for Apache Beam, Managed Service for Apache Spark for Spark, or BigQuery for SQL-based transforms, where pipelines check types, ranges, and required fields before writing to the BigQuery source. TensorFlow Data Validation (TFDV) validates schemas and detects anomalies in tabular features, while automated data quality scans in BigQuery provide continuous checks on the source tables. For maximum training and serving consistency, store materialized data in BigQuery instead of relying on views or subqueries for training data, then read through the BigQuery Storage API so offline training and online sync observe the same validated values.
Monitoring for feature serving detects when live feature values diverge from expectations and when the online copy falls behind its source. Model Monitoring on Gemini Enterprise Agent Platform monitors deployed models for training-serving skew and inference drift and sends alerts when incoming inference data skews too far from the training baseline. Training-serving skew compares serving features against the training baseline, inference drift tracks shifts in serving data over time, and staleness reflects an online store that is no longer consistent with its BigQuery source because sync is delayed or failed. Sync health is observable through feature view sync status as Complete, Running, or Failed, with error messages on failure, so a failed or long-running sync is treated as a freshness incident even if values still serve. Teams use alerts plus feature distributions to decide whether to fix the pipeline, resync the feature view, or retrain the model.
| Signal | What it compares | Governance action |
|---|---|---|
| Training-serving skew | Serving feature distributions vs. training baseline | Investigate pipeline mismatch and evaluate retraining |
| Inference drift | Current inference data vs. earlier inference data | Track performance, identify anomalies, and trigger review |
| Staleness | Online store feature view vs. BigQuery source | Check sync status, retry sync, and block promotion if Failed |
Exam tip: Skew measures serving versus training data, while drift measures serving data changing over time; staleness is neither and means the online store did not sync from BigQuery.
Access governance for feature data restricts who can define feature sources, trigger sync, and read online feature values. Identity and Access Management (IAM) applies at multiple trust boundaries: the Google Cloud project that owns the API and online store, the BigQuery dataset, table, or view that holds the source and history, and the Feature Store online store and feature view resources that expose serving. Only ingestion and sync identities can write or resync the online store, while only approved serving identities and applications can fetch feature values through Bigtable online serving or Optimized online serving endpoints. Audit logging records who created or changed feature groups, feature views, and online stores, who started sync operations, and who accessed serving endpoints, which supports compliance review and incident tracing when bad or stale features reach production.
Lineage and versioning for reproducibility connect each served feature value back to the data, code, and model that produced it. Experiments on Agent Platform integrates with Vertex ML Metadata to log and build linkage across parameters, metrics, and dataset and model artifacts, so a model version points to the feature snapshot and preprocessing logic used to train it. Model Registry on Gemini Enterprise Agent Platform stores, organizes, and versions model binaries and metadata for deployment, evaluation, batch inference, and endpoint serving. Feature Store maintains consistency and versioning of feature data itself, with BigQuery retaining history for offline replay and the online store sync preserving which source state was promoted to serving. Together, this chain lets teams reproduce training, audit which features influenced a prediction, and roll back to a known-good feature and model combination.
eu, us, or us-central1, and replicates only newly inserted rows, never updates or deletes.dense set to true plus Bigtable online serving plus scheduled sync.default_batch_size fixes GoogleSQL INVALID_ARGUMENT with "Request is too large" errors, while increasing @{remote_udf_max_outstanding_rpcs} raises parallelism but also raises Spanner CPU and memory usage.Scheduled data sync runs on a configured timetable per feature view and works with Bigtable or Optimized online serving, while continuous data sync refreshes the online store whenever new BigQuery source data arrives. Continuous data sync requires Bigtable online serving, a feature view linked to feature groups and features, and a BigQuery table source in eu, us, or us-central1, and it replicates only newly inserted rows and never replicates updates or deletes.
Reduce the default_batch_size setting, which defines how many rows are packed into each API request sent from Spanner to Agent Platform. For GoogleSQL databases, an oversized batch returns an INVALID_ARGUMENT error with a Request is too large message, and reducing default_batch_size resolves it by splitting the same workload across more, smaller requests.
Concurrency is controlled by the remote_udf_max_outstanding_rpcs query hint, which sets the number of concurrent API requests sent from Spanner to Agent Platform and overrides default_batch_size for GoogleSQL databases. Increase the value to add parallelism and lower end-to-end latency when the Spanner instance has headroom, and decrease it to throttle load because higher concurrency also increases Spanner CPU and memory usage.
Training-serving skew compares serving feature distributions against the training baseline, while inference drift tracks shifts in current serving data compared with earlier serving data. Staleness is different from both and means the online store feature view is no longer consistent with its BigQuery source because sync is delayed or failed.
Last updated