Scaling Online Inference for Throughput Growth

Agent Platform Feature Store (opens in a new tab) is a managed service that centralizes machine learning features in BigQuery for offline training and in a FeatureOnlineStore for low-latency online inference. Data flows from batch and streaming sources into BigQuery tables or views, then from BigQuery into the online store through FeatureView sync operations, then from the online store to real-time applications. BigQuery remains the durable source of truth with full history, while the online store holds materialized latest values optimized for serving.

1.1Feature Storage and Serving Pipeline Architecture

Feature storage architecture uses BigQuery tables or views as the authoritative source behind every feature view. Source data is first extracted into BigQuery separate from operational systems, with structured data stored as materialized tables for read speed and read through the BigQuery Storage API. Feature groups and features are then registered to catalog those sources, a FeatureOnlineStore instance is created for serving capacity, and FeatureView resources select the columns and rows needed for serving. Only after the feature view exists can data flow from BigQuery to the online store through sync.

Unstructured inputs such as images, video, and audio remain in Cloud Storage in large sharded containers with at least 100 MB per file and 100 to 10,000 shards. Batch ingestion uses Dataflow with the BigQuery I/O connector, TensorFlow I/O or TFX clients, and scheduled transforms, while streaming updates land in BigQuery first and then refresh the online store. Sync frequency must not exceed the rate at which the BigQuery source changes, and production views select only needed columns because complex aggregations increase sync duration and cost.

1.2Feature Views and Sync Modes for Training-Serving Consistency

A FeatureView is the materialized serving definition that selects data from BigQuery and controls how it loads into the FeatureOnlineStore. During online serving, requests flow from the application to the online store endpoint, which returns latest values by entity ID without querying BigQuery. During offline serving, training and batch inference read historical rows directly from BigQuery, preserving history the online store does not retain.

Online serving is configured as Bigtable online serving or Optimized online serving. Bigtable serving hosts features in Bigtable for high-throughput key lookups and is required for continuous sync. Optimized serving exposes a dedicated endpoint through a public domain name or a Private Service Connect endpoint, and can support vector similarity search that returns approximate nearest neighbors from embeddings stored as repeated float fields.

Point-in-time correctness uses a timestamp column with ML.FEATURES_AT_TIME or ML.ENTITY_FEATURES_AT_TIME in CREATE MODEL and ML.PREDICT so training never sees future data.

1.3Request Batching, Concurrency, and Monitoring for Spanner Lookups

Request batching and concurrency tuning controls how lookup requests flow from Spanner to Agent Platform to minimize latency and avoid overload. Requests with large payloads such as text fields are subject to a maximum token limit, so oversized batches fail instead of serving slowly. Tuning balances batch size, concurrent request count, and monitoring of the serving layer.

The default_batch_size setting packs rows into each API request, and smaller batches reduce token and byte size. For GoogleSQL, an oversized batch returns INVALID_ARGUMENT with "Request is too large", which is fixed by reducing default_batch_size to split work across more requests. The @{remote_udf_max_outstanding_rpcs} hint controls concurrent requests from Spanner to Agent Platform and overrides default_batch_size for GoogleSQL. Raising it increases parallelism but also increases Spanner CPU and memory usage.

The system insights dashboard exposes number of requests to Agent Platform, latency of those requests, and network bytes sent. Rising bytes with latency spikes points to batches that are too large, while high request counts with elevated Spanner CPU points to concurrency that is too high.

1.4Governance for Data Quality, Access, and Lineage

Governance ensures served features are valid, fresh, traceable, and accessible only to authorized callers across the path from BigQuery through feature views to the online store. Validation starts in preprocessing with Dataflow, Managed Service for Apache Spark, or BigQuery SQL checks, plus TensorFlow Data Validation (TFDV) for schema and anomaly detection and BigQuery data quality scans. Materialized tables are preferred over views for consistency so training and sync observe the same values.

Monitoring distinguishes three signals. Training-serving skew compares serving distributions to the training baseline, inference drift compares current serving data to earlier serving data, and staleness means the online store did not sync from BigQuery. Sync health moves through Running to Complete or Failed, and a Failed sync is treated as a freshness incident. Each feature view reads BigQuery with a service account granted the BigQuery Data Viewer role, with an option for a dedicated per-view account via FeatureView.service_agent_type for least privilege. Identity and Access Management (IAM) boundaries, audit logging, Vertex ML Metadata, and Model Registry preserve access control, lineage, and versioning for reproduction and rollback.

Vertex AI (opens in a new tab) online serving delivers predictions through a Vertex AI Endpoint that routes traffic to one or more DeployedModels. The serving path flows from model artifacts in Google Cloud Storage (GCS) to a Vertex AI Model in the Model Registry, then to the endpoint that owns routing and scaling. Each deployed model owns its container, machine type, accelerator, and traffic share.

2.1Public Endpoints for Scalable Online Serving

Public endpoints expose scalable prediction over HTTPS with a stable address that persists while models behind it change. Registration with aiplatform.Model.upload() binds artifacts under gs://BUCKET/DIRECTORY to a serving container such as us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest for TensorFlow CPU or a vLLM container such as pytorch-vllm-serve for large language models. The DeepSeek-V3 vLLM example sets --container-ports=7080, MODEL_ID=deepseek-ai/DeepSeek-V3, deployment timeout 7200 seconds, and launch args for vllm.entrypoints.api_server with --tensor-parallel-size=8, --pipeline-parallel-size=2, --gpu-memory-utilization=0.82, --max-model-len=163840, and --max-num-seqs=64.

Endpoint creation with aiplatform.Endpoint.create() binds the endpoint to a project and region, while dedicated endpoints for chat-completion containers require dedicatedEndpointEnabled: true and REST creation. Deployment attaches a Model ID to an endpoint ID with a machine type, accelerator, and traffic split, and can take more than five minutes. The two documented shapes are CPU TensorFlow on n1-standard-2 with no accelerator and distributed LLM on a3-highgpu-8g with nvidia-h100-80gb,count=8 and multi-host count 2. Traffic splitting with flags such as --traffic-split=0=100 enables canary, blue-green, and zero-downtime updates by first deploying alongside the old version, then shifting traffic in steps, then undeploying the old version.

2.2Private Endpoints with Private Service Connect and VPC Controls

Private endpoints keep prediction traffic inside the private Google Cloud network through Private Service Connect (PSC). Requests flow from a client in a consumer Virtual Private Cloud (VPC) to a consumer forwarding rule to the producer ServiceAttachment and then to the Vertex AI endpoint. PSC forwarding rules are unidirectional, so code inside the endpoint container cannot open private egress to other workloads.

Access is gated by an immutable projectAllowlist naming allowed consumer projects. Endpoints with identical allowlists might share one ServiceAttachment, while endpoints with different allowlists use different attachments. When endpoints share an attachment, reuse one consumer PSC endpoint rather than creating one per prediction endpoint. The consumer endpoint must be in the same region as the producer attachment. Internal DNS uses Cloud DNS private zones bound to the consumer VPC with an A record to 199.36.153.4, 199.36.153.5, 199.36.153.6, 199.36.153.7 with TTL 300 and a wildcard CNAME to that apex, repeated for aiplatform.googleapis.com and notebook and Spark domains. Consumer VPC firewall rules must allow egress to the PSC endpoint subnet, while VPC Service Controls with Access Context Manager, IAM, and Cloud Key Management Service (Cloud KMS) customer-managed keys enforce perimeter and identity boundaries.

Attachment stability depends on active deployments. When an endpoint has an active model the attachment remains stable, but with no deployments for more than 10 minutes Vertex AI might delete and recreate it, breaking the forwarding-rule binding. Preserve stability by deploying the new version alongside the old, shifting traffic, and only then undeploying. If PSC status is CLOSED, recreate the forwarding rule. Limits include up to 1,000 connections per attachment, up to 10 allowlist values per project, 7-day reuse block on deleted endpoint names, and no support for tuned Gemini models, private container egress, or Vertex Explainable AI.

2.3Serving Protocols, Latency Path, and High Availability Design

Unified serving design gives clients one stable API while isolating latency-sensitive work behind frontends, retrieval, and models. A private endpoint uses a regional internal Application Load Balancer in the RAG VPC reachable only through Network Connectivity Center, Cloud Interconnect, or Cloud VPN, while a public endpoint uses a regional external load balancer and Cloud Run (opens in a new tab) or GKE frontends use GKE Inference Gateway when on Google Kubernetes Engine (GKE). Private output is the default unless an external balancer is added.

For private retrieval-augmented generation, requests flow from the external network over Interconnect or VPN to the routing VPC, then over VPC spokes via the hub to the RAG VPC, then to the load balancer, frontend, serving subsystem, datastore embedding and vector search, and model generation. The Cloud SQL path flows from Cloud Load Balancing to the Cloud Run frontend to a retrieval service that uses Secret Manager for Vertex AI and Cloud SQL credentials, converts the request to an embedding, searches Cloud SQL PostgreSQL vectors, and sends results plus prompt to Vertex AI. Ingestion is decoupled through Cloud Storage to Pub/Sub to Cloud Run functions to JSONL to parsing and embedding, so the synchronous path pays only for query embedding, lookup, and generation. Resilience uses Shared VPC, Private Google Access, Cloud Router BGP failover, health-checked replicas, Google Cloud Armor and Model Armor, and multi-zone replicas.

Graphics processing units (GPUs) (opens in a new tab) and accelerators on Google Cloud are matched to workload shape and then to a consumption model that determines cost. Compute Engine accelerator-optimized families span A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, A3 Edge, A2, G4, G2, and N1 with T4, complemented by Cloud TPUs and CPU platforms. Throughput, latency, billing rules, and discount limits together determine total cost for training versus serving on GKE and Compute Engine.

3.1Accelerator Options for Training and Serving Workloads

Accelerator-optimized machine series map workload intensity to GPU capability, with larger A-series systems for foundation-model work and G-series for cost-efficient inference. Pre-training uses only high-end A-series systems, fine-tuning adds G4, and serving inference adds both G4 and A3 Edge. Graphics work uses smaller GPUs, while high performance computing can use any series based on offload.

Cloud TPUs (opens in a new tab) accelerate large matrix operations with on-chip high-bandwidth memory (HBM) for larger models and batches, and connect in groups called slices that scale with little to no code changes. Choose TPUs for dense matrix math and large batches, A4 and A3 variants for multi-GPU scale with high-bandwidth interconnects, and G4, G2, or N1+T4 when per-request latency is acceptable and budget dominates. GPU model guidance concentrates fine-tuning on NVIDIA A100 and H100 GPUs, spreads inference across A100, H100, RTX PRO 6000, and L4 for large to small models, and limits T4 to medium and small inference plus graphics.

3.2Billing Models and Discount Rules for Accelerators

Billing for accelerator-optimized machines charges for attached GPUs, predefined vCPU, memory, and bundled Local SSD where applicable. A resource is eligible for only one discount at a time, so on-demand, Spot, Flex-start, and reservation discounts do not stack. Flexible committed use discounts (CUDs) apply differently by family: for N1 with GPUs they cover only vCPUs, memory, and Local SSD, while for G2 and G4 they cover vCPUs, memory, GPUs, and Local SSD.

Consumption divides into on-demand with flexible CUDs for eligible N1, G2, and G4 spend, Spot VMs with 61% to 90% automatic discounts but preemption at any time, Flex-start through Dynamic Workload Scheduler (DWS) with automatic discounts where resources provision when capacity appears, and reservations with resource-based CUDs requiring attached reservations for GPUs and most Local SSD. Clustered capacity through AI Hypercomputer (opens in a new tab) favors dense allocation: future reservations and calendar mode up to 90 days and Flex-start up to seven days offer up to 53% discounts on supported types. Contact the account or sales team before requesting future reservations for supported GPUs or approval is likely to be declined.

3.3Inference Latency and Throughput Signals for Hardware Sizing

Inference signals separate user-perceived responsiveness from system efficiency, because requests per second alone misleads for large language models with variable context length. Latency indicators are Time to First Token (TTFT), Normalized Time Per Output Token (NTPOT) as request_latency / total_output_tokens, Time Per Output Token (TPOT) as (request_latency - time_to_first_token) / (total_output_tokens - 1), inter-token latency (ITL) as mean, median, and p90 between tokens, and end-to-end request latency. Throughput indicators are requests per second, output tokens per second, input tokens per second, and total tokens per second counting both input and output to compare prefill-heavy versus decode-heavy work.

Weight-loading throughput affects startup and scale-out. Cloud Storage FUSE fits limited node scaling, Google Cloud Managed Lustre attaches to 10,000 or more pods for high-throughput parallel access, and Hyperdisk ML attaches to up to 2,500 pods for massive-scale low-latency block access. CPU platforms on 4th generation and later Intel Xeon add Intel QuickAssist Technology (Intel QAT), Dynamic Load Balancer, In-Memory Analytics Accelerator, and Data Streaming Accelerator, and can create Confidential VMs.

3.4Scalable Serving Architectures with Specialized Hardware

Scalable serving combines heterogeneous compute, model-aware routing, and flexible capacity under AI Hypercomputer and GKE. General GPUs are managed as independent units with asynchronous maintenance and node auto-repair for serving fleets, while Clustered GPUs are managed as one tightly coupled system with synchronized maintenance and proactive replacement for multi-host inference. TPUs provide HBM-backed matrix acceleration in scalable slices.

Capacity choices trade stability against preemption. Stable 24x7 inference uses CUDs, fault-tolerant work uses Spot, Flex-start covers dense capacity up to seven days without a fixed start, and Calendar mode reserves H100, H200, and Trillium TPUs up to 90 days to three months. GKE Inference Gateway routes behind a load balancer with model-aware routing that prioritizes GPU and TPU instances, using vLLM or Triton metrics through Managed Service for Prometheus into Cloud Monitoring. Fallback patterns include custom compute classes that fall back from reserved to Spot, centralized balancers that route low-GPU work to Cloud Run and heavy work to GKE, Cloud Service Mesh for policy, and global external load balancers for low-latency distribution.

Scalable serving backends dynamically adjust containerized and managed inference capacity to handle fluctuating throughput without breaching latency objectives. On Google Cloud, Vertex AI Endpoints, Cloud Run, and GKE each scale horizontally but differ in GPU and TPU obtainability, cold starts, and custom metrics. Gemini Enterprise Agent Platform adds reserved throughput, quota-aware autoscaling, and multi-region routing for enterprise agents.

4.1Autoscaling Policies for Containerized Serving

Autoscaling signals combine the four golden signals — latency, traffic, errors, and saturation of CPU, memory, and GPU or TPU — with token-aware generative AI metrics. Model queries per second (QPS), token throughput, and first-token latencies flow from vLLM and NVIDIA Triton Inference Server through Managed Service for Prometheus into Cloud Monitoring, alongside DCGM GPU utilization, memory, and temperature. Objectives such as p95 latency below 300 ms and TTFT below 500 ms for 99% of requests define when to add replicas.

GKE Inference Gateway (opens in a new tab) provides model-aware routing that prioritizes GPU and TPU instances and selects healthy instances using detailed model health checks. GKE custom compute classes add fallback profiles that prefer reserved accelerators then fall back to Spot VMs, while centralized balancers split low-GPU work to Cloud Run and heavy work to GKE. Cold starts force new replicas to load multi-gigabyte weights before serving, so minimum replicas preserve warm capacity, maximum replicas cap cost and quota, and scale-to-zero saves idle cost but forces full initialization. Cloud Storage FUSE fits limited scaling, Managed Lustre scales to 10,000 or more pods, and Hyperdisk ML scales to 2,500 pods.

4.2Provisioned Throughput, Quotas, and Multi-Region Routing

Provisioned Throughput (PT) reserves deterministic throughput for production generative AI with consistent performance and weekly or monthly cost, while Standard pay-as-you-go uses shared regional capacity without manual quota requests. Under Standard, quota dashboards may show 0% even while requests succeed, so consumption is monitored in Model Garden monitoring. PT for partner models such as Anthropic Claude must be purchased by contacting Google.

Quotas define when the backend returns error 429 Resource Exhausted for queries-per-minute or token limits, such as 10 QPM for Gemini 2.5 Pro under Standard. Custom container requirements for inference (opens in a new tab) for DeployedModel scale replicas between dedicatedResources.minReplicaCount and maxReplicaCount toward a 60% utilization target on CPU, or the higher of CPU or GPU when accelerators are present. Use at least two replicas for high availability, and note that minReplicaCount must be at least 1 unless Scale To Zero preview is used. Quota is charged per CPU or GPU in real time, so an a2-highgpu-2g replica counts as 24 CPUs and 2 GPUs. Deployment succeeds only when nodes reach minReplicaCount before timeout, unless requiredReplicaCount allows partial success. Multi-region resilience distributes deployments across regions or uses the global endpoint, with Cloud Storage multi-regional buckets, Spanner, Cloud SQL replicas, multi-zonal vectors, and external state in Memory Bank or Memorystore so agents recover after failure.

4.3Throughput and Concurrency Tuning in High-Performance Runtimes

High-performance runtimes such as vLLM, Triton, TorchServe, TGI, Ray Serve, and MaxDiffusion execute inference in pods and expose concurrency controls. Throughput rises with concurrency only to the inflection point where throughput stops increasing and latency climbs sharply. Output tokens per second measures decode throughput, input tokens per second measures prefill load, and total tokens per second compares mixed workloads. Latency guards include TTFT, TPOT, NTPOT, ITL, and end-to-end latency.

Optimized profiles from Inference Quickstart report saturation metrics including TTFT, NTPOT, output tokens per second, and cost per million tokens, defaulting to us-east5 except A3 in us-central1. Selection runs gcloud container ai profiles models list, then profiles list filtered by model and target TTFT, then benchmarks list for request-rate curves. Dynamic batching, tensor parallelism, KV cache sizing, and async I/O are tuned so concurrency grows without exceeding accelerator memory. Observability combines model QPS and token metrics with CPU, GPU, TPU, network, and disk signals, plus Cloud Logging, Cloud Trace, Cloud Profiler, and DCGM, against objectives such as 99.9% successful responses.

Tuning for production balances predictive performance against inference latency, throughput, and infrastructure cost across distributed training and online serving. Agent Platform Vizier suggests parameter combinations as an independent black-box optimizer, training workers run trials and report measurements, and serving infrastructure is then sized, scaled, and compiled for live traffic. Prompt shape, caching, decoding, hardware, and routing all affect cost per request.

5.1Distributed Training and Hyperparameter Search with Vizier

Black-box optimization covers any evaluable system with a costly objective, from neural network depth, width, learning rate, and batch size to interface colors or thread counts. Vizier only suggests values and evaluates reported results, so it works with training jobs and multi-cloud systems. A study configuration defines the problem, a study implements it through trials, a trial is one set of input values, and a measurement is the observed outcome where each trial can report multiple measurements over time. Vizier suggests trials but does not run them; a study ends at a trial limit and a trial ends as finished or infeasible.

Hyperparameter tuning for custom training uses Vizier internally for training jobs, while Neural Architecture Search is separate and searches up to 10^20 architectures for accuracy, latency, or memory. Training follows cluster creation, initialization, execution for forward and backward propagation, and interruption for checkpointing or preemption. GPU hours equal trials times time per trial times GPUs per trial, so parallelism cuts wall-clock time but not total cost. Default Custom model serving TPU v5e cores per region quota is 4, enough for Llama 3.1 8B, while 70B needs 16 cores in us-west1; A100 80GB quota defaults to 0, needing at least 1 GPU for 8B and 4 for 70B.

5.2Online Serving Infrastructure, Autoscaling, and Resilience Tuning

Online serving infrastructure sizes Vertex AI Prediction endpoints so live traffic gets fast answers without overspending. The client sends instances to a cloud-hosted model on an endpoint and receives predictions per instance. Machine type sets baseline latency, throughput, and cost before autoscaling is tuned. Automatic scaling should set both minimum and maximum nodes, with a minimum of two nodes for high-availability SLA so one node failure does not remove capacity.

Feature inputs flow from a feature-serving solution to the endpoint with each request, using Google Cloud databases or Vertex AI Feature Store online serving with streaming ingestion available within seconds. Custom prediction routines (opens in a new tab) control request and response formats, while private endpoints keep traffic in the VPC when client and model share a local network. Multiple models on one endpoint support gradual replacement with traffic splitting, while multiple endpoints with co-hosted shared resources isolate test and production without full duplication. Vertex AI load-balances across zones in the selected region without cross-region routing; a zonal outage fails in-flight requests, public endpoints retry automatically while private endpoints require client retry with exponential backoff, and regional outages require multi-region deployments.

5.3Model Compression and Hardware-Aware Compilation for Inference

Hardware-aware inference selection chooses the tested combination of model, server, version, and accelerator that meets TTFT and tokens per second at saturation. Profiles list instance type, accelerator, cost per million input and output tokens in USD, output tokens per second, NTPOT, TTFT, server, and version, such as a3-highgpu-1g with H100 serving openai/gpt-oss-20b with vLLM at 297 ms TTFT. Frameworks divide by role: NeMo for pre-training, MaxText for training and serving baselines, SGLang for inference on A3 Mega and Ultra, TensorRT-LLM for compiled NVIDIA GPU execution, and vLLM for high-throughput serving with max_num_seqs and vllm:num_requests_waiting.

Runtime tuning adjusts max_num_seqs and max_num_batched_tokens for token shape, since 8,192-input customer support at 32:1 differs from 4,096-output deep research at 1:16. GKE Inference Gateway adds Cache Aware Routing that sends shared-context requests to the same replica to improve TTFT, dynamic LoRA multiplexing that packs adapters on one base model to save GPUs and TPUs, predicted latency routing with XGBoost for TTFT and TPOT, and Priority where only requests with Priority less than 0 drop first with 429. Prefix caching reuses prompt computations; on a3-high-gpu-4g serving Llama 3.1 70B, GPU KV cache is capped below 180GB after 140GB of weights while host memory cache reaches 936GB. Speculative decoding uses a fast drafter to guess tokens for parallel verification by the large model. Async buffering places Pub/Sub with 7 to 31 days retention before inference, pulls with max_messages as a multiple of max_num_seqs, scales subscribers by desiredReplicas = num_undelivered_messages / (target_latency_seconds x throughput_per_replica), and scales servers on engine backlog with dead-letter handling for failures.

Key Facts

• Agent Platform Feature Store keeps BigQuery as the durable source of truth and syncs latest values through FeatureView operations into a FeatureOnlineStore, where continuous sync requires Bigtable serving and replicates only new inserts.
• Private Service Connect prediction traffic flows from a consumer VPC forwarding rule to a producer ServiceAttachment gated by an immutable projectAllowlist, with private DNS using A records to 199.36.153.4 through 199.36.153.7.
• A resource is eligible for only one discount at a time, with Spot VMs offering 61% to 90% discounts with preemption and Flex-start and calendar reservations offering up to 53% with dense allocation.
• Provisioned Throughput reserves deterministic throughput with weekly or monthly cost for high-throughput workloads, while Vertex AI Endpoints scale DeployedModel replicas between min and max toward 60% CPU or GPU utilization and return error 429 when limits are exceeded.
• GRID_SEARCH requires all parameters to be INTEGER, CATEGORICAL, or DISCRETE, while GKE Inference Gateway Cache Aware Routing sends shared-context requests to the same replica to improve Time to First Token.