Prepare and test your skills
Prepare and test your skills
Manage Model Versioning, Monitoring, and Lifecycle is the practice of tracking every trained model version, watching prediction quality and serving health in production, and promoting or rolling back releases through automated pipelines. On Google Cloud, Vertex AI Model Registry (opens in a new tab) stores versions and lineage, Vertex AI Model Monitoring (opens in a new tab) watches drift and attribution, and Cloud Monitoring (opens in a new tab) watches endpoint and infrastructure signals. Together they connect training outputs to serving endpoints to alerts to retraining and rollback actions.
Model versioning on Vertex AI centers on Vertex AI Model Registry, which stores trained artifacts from Cloud Storage after each training run and records lineage for reverting to the most recent stable version. Each version links to its parameters, code version, dataset version, and evaluation results tracked in Vertex AI Experiments and Vertex AI ML Metadata, so comparisons across iterations are reproducible. Model signing and verification protect this chain, with Cloud Build signing artifacts, Binary Authorization verifying containerized models, and custom checks verifying models deployed directly to Vertex AI endpoints.
Deployment strategy for lifecycle control uses staged releases and automated rollback tied to monitoring signals. Cloud Deploy integrated with Cloud Monitoring tracks the status of canary releases and rollbacks, while endpoints support deploying a new model version alongside a stable version for comparison and traffic shifting. A robust rollback plan uses Model Registry lineage to restore the prior version when Vertex AI Model Monitoring detects significant drift, when generative AI evaluation plus Cloud Logging and Cloud Monitoring flag output-quality shifts or harmful content, or when Model Armor or custom rules flag malicious inputs and unsafe outputs.
Prediction-quality monitoring objectives define what distribution shift the system compares against a baseline such as training data or past production traffic. Vertex AI Model Monitoring v2 monitors streaming endpoints on managed Vertex AI and self-hosted services such as Google Kubernetes Engine (GKE), and it can monitor at model level rather than only endpoint level for granular tracking. Selection depends on how sensitive the use case is to input change, output change, or shifts in reasoning.
| Objective | What is compared | Detects when |
|---|---|---|
| Input feature drift | Live input distribution versus baseline such as training data or golden dataset | Serving inputs diverge from training-serving expectation, indicating training-serving skew |
| Output drift | Live prediction distribution versus baseline prediction distribution | Prediction mix or confidence shifts, including spikes in low-confidence predictions |
| Feature attribution drift | Live attribution values versus baseline attributions | Same inputs produce predictions for different reasons, signaling behavior change even if accuracy looks stable |
Exam tip: Use input drift to catch data change, output drift to catch prediction change, and feature attribution drift to catch reasoning change when inputs and outputs alone look normal.
Continuous evaluation for traditional machine learning models focuses on computation-based metrics such as accuracy, precision, recall, F1-score, AUC-ROC, and log loss with custom alert thresholds in Model Monitoring. Training-serving skew detection starts by building a golden dataset that represents ideal production distribution and using TensorFlow Data Validation (TFDV) to analyze training data and establish a baseline schema, then configuring Model Monitoring to compare live inputs against that schema. Training loss such as cross-entropy for fraud detection is not the decision metric; business-aligned inference metrics such as recall for fraud coverage and area under the precision-recall curve (AuPRC) from the Vertex AI evaluation service determine promotion.
Continuous evaluation for generative models uses the Gen AI evaluation service (opens in a new tab) for response quality, safety, instruction adherence, grounding, writing style, and verbosity, plus human-in-the-loop review for quality, relevance, and guideline adherence. Serving stability for large language model (LLM) endpoints adds time to first token (TTFT), token throughput, queries per second (QPS), latency, and error rates viewed in Metrics Explorer and the model observability dashboard. Batch prediction follows a different path where Model Monitoring supports batch inferences, standard evaluation jobs run in the Vertex AI evaluation service, and Batch resource consumption is tracked with Batch metrics in Metrics Explorer.
System performance monitoring on Vertex AI uses Cloud Monitoring to track endpoint request rate, error rate, latency, and resource utilization, and to track Compute Engine, GKE, Cloud Run, TPU, and GPU health including CPU, memory, network, disk I/O, and autoscaling behavior. Application health follows the four golden signals, which link infrastructure behavior to user experience and business key performance indicators (KPIs) through service-level objectives (SLOs):
Collection uses Cloud Monitoring dashboards including metrics that Vertex AI exports, Managed Service for Prometheus for vLLM and NVIDIA Triton Inference Server metrics, Cloud Logging for application and infrastructure logs, Cloud Trace for request flows across microservices, Cloud Profiler for code-level CPU and memory bottlenecks, and Active Assist for optimization recommendations. Reliability goals tie these signals to SLOs such as 99.9% successful API responses, 95th percentile inference latency below 300 ms, TTFT below 500 ms for 99% of requests, and harmful-output rate below 0.1%.
The automated lifecycle standardizes evaluation so results are reproducible across retraining, promotion, and rollback decisions. Standardization follows an ordered workflow where each step feeds the next decision gate:
When Model Monitoring triggers an alert, the feedback loop flows from the alert to automatic retraining in Vertex AI Pipelines (opens in a new tab), to re-evaluation, to staged deployment, and back to monitoring, with rollback to the Registry version if post-deploy metrics breach SLOs. Custom business alerting supports this loop with a shared observability stack where Cloud Logging collects logs, Cloud Monitoring builds dashboards and sends alerts to PagerDuty or Slack via Pub/Sub, Error Reporting groups exceptions, Cloud Trace isolates latency, and Cloud Profiler finds hotspots. Incident response then moves from preparation with inventories and playbooks, to identification with Logging and audit logs, to containment by disabling endpoints or pausing pipelines, to eradication of the root cause before re-promotion.
Design for Scalability and Cost Efficiency is the architecture practice of keeping machine learning deployments responsive under variable load while paying only for needed capacity. On Google Cloud, the practice combines capacity planning and quotas, horizontal autoscaling, machine type and accelerator choice, discounted consumption options, and controlled traffic distribution. Cost, latency, availability, and obtainability are traded against each other at each layer.
Capacity planning for ML systems starts by forecasting load and then translating that forecast into reserved resources and quota headroom. Forecasting considers number of users, request rate, historical trends, seasonal variation, spikes for special events, and growth from business changes such as new geographies. Estimation then sizes the deployment architecture, such as the size of Compute Engine managed instance groups (MIGs), virtual machine (VM) machine type, and number, type, and size of persistent disks, with cost checked in the Google Cloud Pricing Calculator.
A scalable plan adds redundancy, benchmarking, and quota management before peak events arrive:
Autoscaling lets ML serving and stateless application tiers add capacity when traffic rises and release it when demand falls. Stateless MIGs support autoscaling for graceful handling of traffic increases and lower cost when resource need is low, while stateful MIGs cannot be autoscaled. On GKE, autoscaling works with node pools and custom compute classes so pods can scale across preferred hardware and fall back to alternate capacity when preferred accelerators are unavailable.
Effective autoscaling depends on observability and rightsizing. Cloud Monitoring tracks endpoint request rate, error rate, latency, and resource utilization, plus saturation signals such as CPU, GPU, tensor processing unit (TPU), memory, network throughput, and disk I/O. If autoscaling is enabled, teams must monitor scaling behavior to confirm it responds appropriately to traffic patterns, and use Active Assist recommendations and Compute Engine machine-type recommendations to rightsize instances to workload requirements.
Machine-type selection balances latency, throughput, memory, and hourly price for training versus inference. Compute Engine (opens in a new tab) provides predefined and custom machine types, so a team can provision CPU cores independently of memory and match the VM shape to predictable requirements instead of overprovisioning. Graphics processing units (GPUs) accelerate ML and data processing, with GKE offering NVIDIA H100, A100, and L4 options; in practice, high-end GPUs such as A100, H100, H200, and B200 suit training, while L4 GPUs suit cost-sensitive inference.
For inference with fluctuating accelerator needs, vLLM can dynamically use TPUs and GPUs to address capacity and performance shifts, an approach described as GPU/TPU fungibility. For advanced multi-GPU topology needs, Cluster Director (opens in a new tab) deploys and manages colocated accelerator groups for A3 Ultra H200 and A4 B200 across GKE and Slurm clusters, while Ray on Gemini Enterprise Agent Platform abstracts distributed VMs and containers so applications request resources for training and serving without direct infrastructure management.
Consumption options trade stability and obtainability against discount depth and preemption risk. Stable 24x7 inference or predictable training justifies committed use discounts (CUDs) for VMs and accelerators, while fault-tolerant, batch, evaluation, and short-lived work can use preemptible or short-term options. Flex-start, powered by Dynamic Workload Scheduler (DWS), persists the GPU request and automatically provisions Flex-start VMs when capacity becomes available, which improves obtainability for high-demand accelerators without complex quota management.
The following options cover the main cost-availability tradeoffs supported for GKE nodes, Compute Engine VMs, and clustered GPUs:
| Option | Best for | Price and duration |
|---|---|---|
| Committed use discounts (CUDs) | Stable 24x7 inference or training with fixed, predictable capacity | Discounted rate for a commitment term on VMs and accelerators |
| Flex-start with DWS (opens in a new tab) | Short-lived dense clusters or batch training needing high-end GPUs without a fixed start date | Up to 53% discount on vCPUs, GPUs, TPUs, and attached Local SSD disks; pay as you go; runs up to 7 days |
| Calendar mode reservations | Longer stable runs such as pre-training or fine-tuning that need specific GPUs or TPUs | Up to 53% discount with dense allocation; runs up to 90 days or 3 months for H100, H200, and Trillium TPUs |
| Spot VMs | Fault-tolerant, batch, evaluation, or experimentation work that tolerates disruption | Deepest discount between 61% and 90%; capacity can be preempted at any time and MIG scale-out may wait until capacity returns |
Exam tip: Choose Spot VMs for the lowest price when preemption is tolerable, and choose Flex-start when the request can wait in a queue for up to seven days for better obtainability with up to a 53% discount.
Traffic distribution for rollout safety spreads inference load across instances while limiting the blast radius of a new model version. GKE Inference Gateway places models behind a load balancer with model-aware routing that prioritizes instances with GPU and TPU accelerators for compute-intensive generative AI and LLM inference. Detailed health checks assess model status, while serving frameworks such as vLLM or NVIDIA Triton Inference Server export LLM metrics into Cloud Monitoring through Google Cloud Managed Service for Prometheus.
Traffic architecture also controls cost and rollout risk across platforms and regions. A centralized load balancer can route low-GPU requests to Cloud Run and GPU-intensive requests to GKE, with Cloud Service Mesh handling inter-service communication and policy, and a global external Application Load Balancer distributing global user traffic for low latency. For safe releases, teams split traffic between model versions for canary releases and track deployment, canary, and rollback status by integrating Cloud Deploy (opens in a new tab) with Cloud Monitoring. Model Registry preserves model lineage so operations can revert to the most recent stable version, while Model Monitoring detects drift to trigger retraining or rollback.
Evaluate Deployment Patterns for Model Serving is the selection of batch, online, or streaming inference based on data velocity, latency limits, and cost. On Google Cloud, Google Kubernetes Engine (GKE) (opens in a new tab) is the primary platform for custom, accelerator-intensive online inference, with GKE Inference Gateway, autoscaling, and GPU and TPU obtainability controls managing scale, availability, and cost. Successful deployments balance request latency and token throughput against startup time, accelerator availability, and price.
Online inference serving on GKE hosts models in containers on GPU or TPU node pools to answer interactive requests with millisecond-scale latency. A production deployment uses the GKE inference reference architecture, which provisions a standardized, secure, and scalable environment with Infrastructure as Code (IaC) using Terraform wrapped in command-line scripts, and deploys single-host GPU nodes running a serving framework such as vLLM, NVIDIA Triton Inference Server (Triton), Text Generation Inference (TGI), TorchServe, Ray Serve, or MaxDiffusion. Traffic flows from the client to a load balancer to GKE Inference Gateway to a model server Pod equipped with accelerators such as NVIDIA H100, A100, or L4 GPUs, and the response flows back along the same path.
The gateway adds model-aware routing with affinity-based endpoint picking for fine-tuned models that share low-rank adaptation (LoRA) files, dynamic access to GPU and TPU capacity across regions for high availability, detailed health checks for model status, and Model Armor add-on policies for security. For production onboarding, Inference Quickstart (opens in a new tab) provides pre-configured deployment paths for open models including Gemma, Llama 4 Scout, Qwen3, and gpt-oss.
Inference latency and throughput metrics quantify user-perceived responsiveness and system capacity for generative models on GKE. Request latency is end-to-end time to complete a request, while token-level metrics separate the prefill phase from the decode phase so teams can diagnose whether slowness comes from processing the prompt or generating output. Throughput metrics measure how many tokens or requests the server completes per second, which determines cost-efficiency at scale.
The benchmark indicators for GKE inference are:
(request_latency - time_to_first_token) / (total_output_tokens - 1)request_latency / total_output_tokenstotal_output_tokens_generated_by_server / elapsed_time_in_secondstotal_input_tokens_generated_by_server / elapsed_time_in_secondstotal_tokens_generated_by_server / elapsed_time_in_seconds, counting both input and output tokens to compare workloads with high prefill versus high decode timesServing frameworks expose these metrics for observability, and vLLM or Triton metrics can be integrated into Cloud Monitoring (opens in a new tab) by using Google Cloud Managed Service for Prometheus, with model queries per second, token throughput, and first-token latencies visible in the model observability dashboard.
Model weight loading controls inference startup time and scale-out consistency when many Pods must load multi-gigabyte weights simultaneously. The model server Pod depends on a storage layer that delivers weights from Cloud Storage or block storage to accelerators, and the choice determines whether cold starts and autoscaling stay fast as the deployment grows from tens to thousands of Pods. Minimizing weight-loading latency is therefore a first-order decision for autoscaled LLM serving on GKE.
| Storage option | Scale and access model | Choose when |
|---|---|---|
| Cloud Storage FUSE | Mounts Cloud Storage buckets with caching and parallel downloads | Deployments with limited node scaling need simple, low-operational access to weights |
| Google Cloud Managed Lustre | Fully managed parallel file system that can attach to 10,000 or more Pods | Inference workloads demand consistent scale-out performance with high-throughput, low-latency file access from multiple Pods simultaneously |
| Google Cloud Hyperdisk ML | Network-attached block disk that can attach to up to 2,500 Pods | Massive-scale scenarios demand consistent, low-latency access to large model weights through a dedicated block solution |
Exam tip: Choose Cloud Storage FUSE for limited scaling, Google Cloud Managed Lustre for consistent scale-out to 10,000 or more Pods, and Google Cloud Hyperdisk ML for dedicated low-latency block access up to 2,500 Pods.
Distributing traffic across instances preserves low latency and high availability when demand fluctuates and accelerator capacity is constrained. The routing layer sits between clients and model server Pods and selects destinations based on health, locality, cost, and model-specific load such as key-value cache pressure or LoRA affinity. Effective patterns combine Kubernetes-native routing with Google Cloud load balancing and observability through Cloud Logging and Cloud Monitoring.
Traffic distribution patterns for AI inference are:
Accelerator availability and cost optimization matches GPU and TPU consumption models to workload stability, duration, and tolerance for preemption. GKE inference depends on node capacity with accelerators such as H100, H200, B200, A100, and L4 GPUs and Trillium TPUs, and different purchase models trade guaranteed capacity against discount depth and interruption risk. Planning for inference must therefore weigh cost-efficiency, performance, flexibility, and resource obtainability together.
Obtainability strategies for GKE inference are:
Custom compute classes tie these choices to autoscaling by defining a hierarchy of node configurations with fallback policies, so the cluster first provisions preferred reserved capacity and only then falls back to Spot VMs during shortages.
Input feature drift compares live input distribution versus a baseline such as training data to detect training-serving skew, output drift compares live prediction distribution versus baseline predictions to detect shifts in prediction mix or confidence, and feature attribution drift compares live attribution values versus baseline attributions to detect behavior change when the same inputs produce predictions for different reasons.
Spot Virtual Machines (VMs) are best for fault-tolerant, batch, evaluation, or experimentation work that tolerates disruption because they offer the deepest discount between 61% and 90% but can be preempted at any time. Flex-start with Dynamic Workload Scheduler (DWS) is best for short-lived dense clusters or batch training needing high-end Graphics Processing Units (GPUs) without a fixed start date with up to 53% discount for runs up to 7 days, while calendar-mode reservations are best for longer stable runs such as pre-training or fine-tuning with up to 53% discount for runs up to 90 days.
Cloud Storage FUSE mounts Cloud Storage buckets with caching and parallel downloads and suits deployments with limited node scaling that need simple, low-operational access to weights. Google Cloud Managed Lustre is a fully managed parallel file system for inference workloads that demand consistent scale-out performance to 10,000 or more Pods, while Google Cloud Hyperdisk ML is a network-attached block disk for massive-scale scenarios that demand consistent, low-latency access through a dedicated block solution for up to 2,500 Pods.
Last updated