Prepare and test your skills
Prepare and test your skills
Public endpoints for scalable online serving on Vertex AI (opens in a new tab) deliver predictions over a public HTTPS endpoint that scales with request load. The serving path flows from model artifacts in Google Cloud Storage (GCS) to a Vertex AI Model in the Model Registry, then to a Vertex AI Endpoint with one or more DeployedModels that receive live traffic. The endpoint owns the routing and scaling boundary, while each deployed model owns its container, machine type, accelerator, and share of traffic.
Registering a containerized model creates a reusable Vertex AI Model resource that binds stored artifacts to a serving container. Artifacts staged under gs://BUCKET/DIRECTORY in GCS flow into aiplatform.Model.upload(), which records the display_name, artifact_uri, and serving_container_image_uri. The serving container defines the runtime contract, such as us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest for TensorFlow CPU serving or us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250312_0916_RC01 for vLLM-based large language models.
Custom container settings for vLLM serving control parallelism, memory, and request handling for every replica. The documented DeepSeek-V3 example sets container interface options --container-ports=7080, --container-env-vars=MODEL_ID=deepseek-ai/DeepSeek-V3, --container-deployment-timeout-seconds=7200, and --container-args that launch vllm.entrypoints.api_server on host 0.0.0.0 and port 7080. The launch args set --tensor-parallel-size=8 with --pipeline-parallel-size=2, --gpu-memory-utilization=0.82, --max-model-len=163840, --max-num-seqs=64, --enable-chunked-prefill, --kv-cache-dtype=auto, and --trust-remote-code. Choose a prebuilt TensorFlow CPU container when the model is a standard SavedModel, and choose a vLLM container with explicit launch args when the model needs chat-completion serving and distributed GPU inference.
A Vertex AI Endpoint is the stable public resource that clients call for online predictions, separate from the models deployed behind it. Creation with aiplatform.Endpoint.create() or REST POST /v1/projects/PROJECT_ID/locations/REGION/endpoints binds the endpoint to a project and region, such as {REGION}-aiplatform.googleapis.com. The endpoint persists independently, so models can be added, replaced, or re-weighted without changing the client-facing address.
Dedicated endpoints support Model Garden chat-completion containers that require them. Creation requires dedicatedEndpointEnabled: true and a displayName in the request body, and the response returns the endpoint ID used for later deployment. Dedicated endpoints are in preview and do not support creation through the Google Cloud CLI, so the creation request must flow from curl with a bearer token to the regional aiplatform.googleapis.com API. Use a standard endpoint for general online prediction, and use a dedicated endpoint only when the container requires it.
Deployment attaches a registered model to an endpoint as a DeployedModel with dedicated compute. The deployment depends on both sides existing first: the Model ID from gcloud ai models list and the endpoint ID from gcloud ai endpoints list flow into gcloud alpha ai endpoints deploy-model or model.deploy(). Deployment is a long-running operation that can take more than five minutes, and large-model deployment can exceed the CLI timeout while continuing in the background until gcloud ai operations describe returns "done": true.
Compute shape is set by machine type plus optional accelerators and multi-host count. The machine type controls host CPU, memory, and GPU attachment, the accelerator controls GPU type and count per replica, and the multi-host count controls how many nodes cooperate on one replica. The following patterns show the two documented shapes:
| Deployment pattern | Machine type | Accelerator and scale | Choose when |
|---|---|---|---|
| CPU TensorFlow serving | n1-standard-2 | No accelerator | The model is a TensorFlow SavedModel served by a CPU prebuilt container |
| Distributed LLM serving | a3-highgpu-8g | nvidia-h100-80gb,count=8 with --multihost-gpu-node-count=2 | The model is a large language model such as DeepSeek-V3 that needs vLLM, high GPU memory, and tensor plus pipeline parallelism |
Traffic splitting controls how prediction requests flow from one endpoint to multiple deployed models behind it. The flag --traffic-split=0=100 routes 100% of traffic to deployed-model ID 0, and re-issuing deployment with adjusted splits shifts load without deleting the endpoint. Because the endpoint boundary stays fixed while the deployed models change, clients see a single address during canary testing, blue-green releases, and zero-downtime updates. First deploy the new version alongside the current version, then move a small share of traffic to validate, then shift to 100% only after the new version is healthy.
A full update preserves availability when steps run in order:
Exam tip: If deploy-model times out on a large model, the deployment still continues in the background — poll the returned operation ID with gcloud ai operations describe until it shows "done": true.
Private endpoints for online prediction on Vertex AI keep inference traffic inside the private Google Cloud network instead of traversing the public internet. The architecture uses Private Service Connect (PSC) (opens in a new tab) to expose a Vertex AI endpoint as a producer ServiceAttachment that consumer Virtual Private Cloud (VPC) networks reach through a consumer PSC endpoint and forwarding rule. VPC Service Controls, Identity and Access Management (IAM), internal DNS, and firewall rules layer identity, perimeter, name-resolution, and network boundaries around that private path to isolate prediction traffic with low latency.
Private Service Connect endpoint architecture is the private connectivity model where prediction requests flow from a client in a consumer VPC to a forwarding rule to the producer ServiceAttachment and then to the Vertex AI endpoint. The producer side lives in the Vertex AI project and is identified by a ServiceAttachment resource, while the consumer side lives in the allowed client project as a PSC endpoint backed by a forwarding rule. Traffic flows only from consumer to producer because PSC forwarding rules are unidirectional, which means code running inside the endpoint container cannot open private egress to other private Google Cloud workloads.
Access to the ServiceAttachment is gated by a projectAllowlist that names which consumer projects may connect. When multiple prediction endpoints use identical project allowlists, they might share the same ServiceAttachment so allowlisted projects can reach those endpoints through one attachment, but this reuse is not guaranteed. When endpoints use different project allowlists, they use different ServiceAttachment resources to ensure connections are accepted only from projects on their respective allowlists. When multiple endpoints share a ServiceAttachment, create and reuse one PSC endpoint for that ServiceAttachment rather than creating a separate PSC endpoint for each prediction endpoint. The consumer PSC endpoint must be in the same region as the producer ServiceAttachment, so regional alignment is a hard placement dependency.
Internal DNS for private prediction traffic ensures that a private client resolves Vertex AI API names to private addresses instead of public endpoints. The design uses Cloud DNS private managed zones with private visibility bound to the consumer VPC network for each Google API domain. Each zone contains an A record that reroutes the apex domain to Google restricted virtual IP addresses and a CNAME record that redirects all wildcard names to that apex record.
Build the private name resolution in order:
--visibility=private bound to the consumer VPC network, starting with aiplatform.googleapis.com.DNS_NAME. with addresses 199.36.153.4, 199.36.153.5, 199.36.153.6, 199.36.153.7 and TTL 300 to route traffic to Google restricted addresses.*.DNS_NAME. pointing to DNS_NAME. with TTL 300 so every service hostname under the domain follows the private A record.*.aiplatform.googleapis.com, *.aiplatform-notebook.googleusercontent.com, and *.aiplatform-notebook.cloud.google.com, plus other needed domains such as dataproc.byoid.googleusercontent.com and dataproc.googleusercontent.com for private Spark access.Create these DNS entries even when VPC Service Controls is not in use, because private clients still need private name resolution to avoid public egress. Prediction clients then send predict calls to the internal IP address or to the configured DNS name, using endpoint_override=DNS_NAME in the SDK or an HTTPS POST to https://INTERNAL_IP_ADDRESS/v1/projects/VERTEX_AI_PROJECT_ID/locations/REGION/endpoints/ENDPOINT_ID:predict with a bearer access token.
Firewall rules for private endpoints define the network boundary that permits private prediction traffic to leave the client network toward the PSC endpoint subnet. The VPC network that connects on-premises and multicloud environments to Google Cloud must allow egress traffic to the Private Service Connect endpoint subnet. Without that egress allowance, requests from hybrid clients never reach the consumer forwarding rule even when IAM and DNS are correct. This control sits on the consumer side of the boundary, while the producer ServiceAttachment sits on the Vertex AI service side and accepts only allowlisted consumer connections.
VPC Service Controls is a managed networking perimeter that blocks data access from outside the perimeter even when IAM permissions are overly permissive. Storage and AI services such as BigQuery, Cloud Storage, and the Vertex AI API (aiplatform.googleapis.com) are placed inside the perimeter, and VPC Service Controls denies access to anyone not explicitly permitted. Designated IAM principals are granted contextual entry through an Access Context Manager access level that can evaluate identity type, identity, device data, and network origin such as IP address or VPC network.
Cross-perimeter communication is denied by default and requires explicit ingress and egress rules. A project can belong to only one service perimeter, so sharing logs or features across perimeters requires configured secure data exchange with ingress rules for context-aware inbound access and egress rules for controlled outbound access. For prediction workloads, lock sensitive training and inference data in its own perimeter, enforce IAM at project, folder, or dataset levels, require multifactor authentication for human access to sensitive data, and encrypt stored data with customer-managed encryption keys (CMEK) from Cloud Key Management Service (Cloud KMS).
Exam tip: ServiceAttachment reuse is network resource reuse for PSC connectivity and is different from model co-hosting with DeploymentResourcePool, which is compute resource reuse for deployments.
The ServiceAttachment lifecycle depends on whether the endpoint has an active deployed model. When an endpoint has an active deployed model, the ServiceAttachment does not change and remains stable for inference. When an endpoint has no active deployments, Vertex AI might delete the ServiceAttachment and recreate it on the next deployment, which breaks the existing consumer forwarding-rule binding. Always check the ServiceAttachment and PSC endpoint status before making an inference call, and if the Private Service Connect connection status is CLOSED, recreate the forwarding rule to restore the connected state.
To preserve attachment stability during model updates, create a traffic split and gradually migrate traffic to the new model version before undeploying the earlier version. This ordered sequence — first deploy the new version alongside the old, then shift traffic, and only after migration undeploy the old version — avoids a zero-deployment window that could trigger deletion. If all models remain undeployed for more than 10 minutes, assume the ServiceAttachment might be deleted and verify status before the next prediction. A single ServiceAttachment supports up to 1,000 connections, and forwarding rules have their own Cloud Load Balancing quota, so large fan-in designs must track both limits.
Operational limits for private endpoints are fixed product constraints that shape resilient serving designs. The endpoint projectAllowlist value cannot be changed after configuration, and a project can have up to 10 different projectAllowlist values across its Private Service Connect configurations. Before deleting an endpoint, you must undeploy its models, and after deletion the endpoint name cannot be reused for up to 7 days.
The following limits block specific features on private endpoints:
These constraints mean private serving is chosen when isolation and exfiltration control outweigh explainability and in-container private callbacks. For workloads that need those unsupported features, retain a separate public or differently scoped endpoint outside the private perimeter rather than weakening the private endpoint boundary.
Optimizing serving protocols, latency, and high availability means designing prediction endpoints that move inference traffic over the shortest secure path and survive zone, region, or connection failures. In Google Cloud, Application Load Balancer (opens in a new tab) frontends, Cloud Run frontend and backend services, and Vertex AI models form the serving path, while Virtual Private Cloud (VPC), Network Connectivity Center, Cloud Interconnect, and Cloud VPN provide the private, high-availability network under it. Private endpoints keep input, inter-service traffic, and output on private IP addresses without traversing the internet, and public endpoints are added only when internet clients must reach the service.
Private and public prediction endpoints differ by who can reach the serving IP address and which load balancer exposes it. A private endpoint uses a regional internal Application Load Balancer whose IP address lives in the RAG VPC network, and only users in the designated external network connected through Cloud Interconnect or Cloud VPN and the Network Connectivity Center (NCC) hub can reach it. A public endpoint uses a regional external Application Load Balancer that is reachable by clients on the internet, and a GKE Inference Gateway is used instead of an Application Load Balancer when the frontend subsystem runs on Google Kubernetes Engine (GKE).
Traffic to a private endpoint flows from the external network over Cloud Interconnect or Cloud VPN to the routing VPC network, then over the VPC spoke connection to the RAG VPC network, then to the Application Load Balancer. The RAG network, the routing network, and the hybrid connections are all spokes of the same NCC hub, and NCC exports all spoke subranges to the hub and re-exports them to the other spoke networks. This hub-and-spoke routing is what lets authorized on-premises users reach the load-balanced service without exposing inference results to the internet.
| Endpoint option | Reachability | Choose when |
|---|---|---|
| Regional internal Application Load Balancer | Private IP in RAG VPC, reachable only via NCC hub spokes and hybrid connectivity | Input, output, and inter-service traffic must stay on private addresses for security and compliance |
| Regional external Application Load Balancer | Public IP reachable from the internet | Application users on the internet must send queries to the service |
| GKE Inference Gateway | Frontend on GKE, replaces Application Load Balancer | Frontend subsystem runs on GKE and needs gateway-based inference routing |
Exam tip: Private output is the default — inference results are not reachable over the internet unless you explicitly add a regional external Application Load Balancer.
The inference request flow is the ordered path a prediction request follows from the client to the frontend, retrieval, and model services and back. Each hop adds latency, so the design minimizes hops by using a centralized backend as a single unified endpoint and by keeping datastore and model calls on Google's internal infrastructure, Private Service Connect endpoints, or Private Google Access. The frontend handles client experience, the backend handles preprocessing and fan-out, and the serving subsystem handles retrieval augmentation and model generation.
For the private RAG architecture, customer requests move through a fixed sequence:
The Cloud SQL-based serving path uses the same principle with explicit Google Cloud services. Cloud Load Balancing receives external requests and distributes traffic to the Cloud Load Balancing backend, which distributes traffic to the Cloud Run frontend service. The frontend service then communicates with a retrieval service for a generative AI call, the retrieval service uses Secret Manager to access credentials for Vertex AI and Cloud SQL, converts the request to an embedding, searches for similar vectors in the Cloud SQL PostgreSQL database, and sends the search results plus the original prompt to Vertex AI to create a response.
A fault-tolerant network architecture spreads risk across connections, routes, zones, and service replicas so strict service-level objectives survive failures. Cloud Interconnect provides high-availability, low-latency hybrid connectivity, Cloud VPN provides secure IPsec extension of the peer network, and Cloud Router provides distributed Border Gateway Protocol (BGP) speaker and responder capabilities that create dynamic routes for Cloud Interconnect, Cloud VPN, and router appliances. When a hybrid link or route fails, BGP withdraws and re-advertises routes so traffic from the external network can still reach the routing VPC network and the NCC hub.
Resilience inside Google Cloud depends on private connectivity controls that keep dependencies explicit and isolated:
Health checking and failover operate at two layers. Cloud Load Balancing distributes traffic only to healthy Cloud Run frontend replicas and backend services, so an unhealthy replica stops receiving requests while healthy replicas absorb load. At the network layer, the NCC hub with exported spoke subranges, Cloud Router BGP routes, and redundant Cloud Interconnect or Cloud VPN tunnels provide alternate paths from the external network to the routing VPC network and onward to the RAG VPC network.
A unified serving design gives all clients one stable API surface while isolating latency-sensitive work behind it. The design uses one or more Cloud Run frontend services for different experiences such as chatbot UI, search page, or mobile application, and one centralized Cloud Run backend service that provides a single unified endpoint for all frontends. The backend service performs preprocessing such as constructing search filters, which keeps logic transparent to frontends and prevents each client from making separate slow calls to the datastore and model.
Data ingestion is decoupled from serving so indexing work does not block predictions. Data engineers upload raw data to a Cloud Storage bucket, Cloud Storage publishes a message to a Pub/Sub topic, the topic triggers processing in Cloud Run functions, functions write JSON Lines (JSONL) metadata to a separate bucket, and a second Pub/Sub message triggers parsing, chunking, and embedding into the managed datastore or Cloud SQL vectors. Vertex AI creates embeddings of text fields and stores them as vectors, and options such as Vertex AI Vector Search, AlloyDB, and Cloud SQL with pgvector support low-latency similarity search at serving time. Because ingestion flows through Pub/Sub and Cloud Run functions asynchronously, the synchronous inference path only pays for embedding the query, vector lookup, and model generation.
--traffic-split=0=100.a3-highgpu-8g with nvidia-h100-80gb,count=8 and multi-host count 2, while CPU TensorFlow serving uses n1-standard-2 with no accelerator.projectAllowlist.199.36.153.4 through 199.36.153.7 and wildcard CNAME records to keep Vertex AI traffic on private addresses.Use a regional internal Application Load Balancer when input, output, and inter-service traffic must stay on private addresses for security and compliance, because its private IP lives in the RAG Virtual Private Cloud (VPC) network and is reachable only via Network Connectivity Center hub spokes and hybrid connectivity. Use a regional external Application Load Balancer when application users on the internet must send queries to the service, because it exposes a public IP reachable from the internet. Inference results are not reachable over the internet unless you explicitly add the external load balancer.
Traffic splitting controls how prediction requests flow from one stable endpoint to multiple deployed models behind it, so clients see a single address while the models change. A split such as 0=100 routes 100 percent of traffic to deployed-model ID 0, and re-issuing deployment with adjusted splits shifts load without deleting the endpoint. For safe updates, first deploy the new version alongside the current version, then move a small share of traffic to validate, then shift to 100 percent only after the new version is healthy.
CPU TensorFlow serving uses machine type n1-standard-2 with no accelerator when the model is a TensorFlow SavedModel served by a CPU prebuilt container. Distributed large language model serving uses machine type a3-highgpu-8g with accelerator nvidia-h100-80gb count 8 and multi-host count 2 when the model is a large language model such as DeepSeek-V3 that needs vLLM, high GPU memory, and tensor plus pipeline parallelism.
The ServiceAttachment remains stable while the endpoint has an active deployed model, but Vertex AI might delete it when the endpoint has no active deployments, and if all models remain undeployed for more than 10 minutes you should assume it might be deleted and verify status before the next prediction. To preserve stability, create a traffic split and gradually migrate traffic to the new model version before undeploying the earlier version, which avoids a zero-deployment window that could trigger deletion. Always check the ServiceAttachment and PSC endpoint status before inference, and if the connection status is CLOSED, recreate the forwarding rule to restore the connected state.
Last updated