Prepare and test your skills
Prepare and test your skills
Defining application success metrics and automating rollbacks means measuring deployment health in real time and returning to the last stable version when health violates its targets. Cloud Monitoring (opens in a new tab) evaluates service-level indicators against objectives, Cloud Logging (opens in a new tab) provides event context for diagnosis, and Cloud Deploy (opens in a new tab) tracks release status so a failed canary or rolling update can revert. For machine learning workloads, the same pattern extends to Model Monitoring on Vertex AI (opens in a new tab) and Model Registry for drift detection and versioned rollback.
Service-level indicators (SLIs), service-level objectives (SLOs), and error budgets turn user-perceived reliability into measurable deployment gates. A service-level indicator (SLI) is the measured ratio or distribution of good events, such as success rate or latency, while a service-level objective (SLO) is the target value for that indicator over a time window. Aligning SLOs to business key performance indicators (KPIs) keeps reliability work focused on behavior that affects users and revenue, such as conversational-assistant satisfaction mapped to request success rate, time to first token (TTFT), harmful-response rate, and task-completion rate.
Quantified SLO targets make canary, blue/green, and traffic-splitting decisions objective. Examples include 99.9% of API calls must return a successful response, 95th percentile inference latency must be below 300 ms, TTFT must be below 500 ms for 99% of requests, and harmful-output rate must be below 0.1%. When the measured SLI burns through its error budget faster than allowed, the deployment is unhealthy even if infrastructure looks normal. That SLI-based verdict is what automated promotion or rollback should consume.
Cloud Monitoring is the service that collects, stores, charts, and alerts on deployment health signals in real time. Cloud Monitoring automatically collects system metrics from Google Cloud services, collects system and application metrics from Compute Engine virtual machines (VMs) through the Ops Agent, ingests Prometheus metrics through Google Cloud Managed Service for Prometheus, stores user-defined metrics created through the Cloud Monitoring API or OpenTelemetry, and derives log-based metrics from logs written to Cloud Logging. When an alerting policy or chart is created, a query describes the data to watch, written with menu selections or with a query language such as Monitoring Query Language (MQL) or Prometheus Query Language (PromQL).
Cloud Logging is the service that collects, stores, and analyzes logs from every component for troubleshooting and performance analysis. Logs add the context that metrics alone cannot provide, linking an SLI drop to specific errors, deployments, or request flows. Together, Cloud Monitoring and Cloud Logging support the four golden signals for applications, including AI serving endpoints:
Custom dashboards and Metrics Explorer turn those signals into deployment views. Google Cloud-managed dashboards show curated health for resources and App Hub applications, while custom dashboards can display metric charts and tables, log data and error groups, alert charts, alert status, text, and operational events such as a reboot or crash. Tracking request rate, error rate, latency, and resource utilization for serving endpoints, plus underlying Compute Engine, Google Kubernetes Engine (GKE), TPU, and GPU health and autoscaling behavior, lets a team compare the canary revision against the stable revision during progressive delivery.
An automated rollback is a closed loop in which an alerting policy detects an SLO violation and its notification triggers a return to the previous stable revision. An alerting policy in Cloud Monitoring defines the metric query and threshold criteria that indicate failure and the list of people, groups, or systems to notify through notification channels such as email, Cloud Mobile App, PagerDuty, Slack, or a Cloud Pub/Sub (opens in a new tab) topic for programmatic response. Each firing creates an incident record that stores status, links to logs, a chart of recorded metric data, labels, and duration, so the rollback event remains auditable.
A progressive infrastructure and application change follows an ordered guardrail that makes rollback safe:
Tracking canary releases and rollbacks by integrating Cloud Deploy with Cloud Monitoring keeps deployment state and health signals in one view. Because global resources such as Virtual Private Cloud (VPC) networks and global load balancers can become single points of failure, changes to them need high-risk review, since a misconfigured forwarding rule can stop the frontend from receiving user requests even when the backend is intact.
Exam tip: Trigger rollback from an SLI-based alert such as success rate or p95 latency breaching its SLO, not only from a raw infrastructure signal such as VM CPU utilization exceeding 80%.
Model success metrics extend application SLIs with training-serving skew, prediction drift, and output-quality signals. Model Monitoring on Vertex AI proactively tracks performance, compares live input distribution against a golden dataset that represents ideal data, uses TensorFlow Data Validation (TFDV) to establish a baseline schema, and raises alerts that can trigger retraining with Vertex AI Pipelines or a rollback. Model lineage in Model Registry identifies the most recent stable model version to restore when drift or degradation is confirmed.
Traditional ML and generative AI use different metric families but share the same alert-to-rollback pattern:
| Workload | Primary success signals | Rollback trigger example |
|---|---|---|
| Traditional ML | Accuracy, precision, recall, F1-score, AUC-ROC, and log loss, plus data-drift and skew distance | Accuracy falls below minimum or error rate exceeds maximum defined in Model Monitoring |
| Generative AI | Response quality, safety, instruction adherence, grounding, writing style, and verbosity from the Gen AI evaluation service, plus harmful-content rate | Shift in output quality or emergence of harmful content detected by model evaluation with Cloud Logging and Cloud Monitoring |
Custom alerting for business-specific thresholds completes the telemetry framework. Cloud Logging collects and analyzes logs, Cloud Monitoring builds custom dashboards and custom metrics and sends notifications to incident tools, Error Reporting captures exceptions, Cloud Trace isolates latency between pipeline components, and Cloud Profiler finds production code hotspots. Configuring generative AI-specific alerts to invoke the same rollback procedure used for application canaries ensures that a model that passes functional deployment but fails safety or relevance in production is replaced with its last stable registry version.
Progressive deployment on Google Cloud is the practice of releasing a new version to a subset of traffic and expanding exposure only as telemetry proves success. Google Kubernetes Engine (GKE) (opens in a new tab) , Cloud Run (opens in a new tab), and App Engine (opens in a new tab) each host multiple versions side by side, while Anthos Service Mesh (opens in a new tab) (Cloud Service Mesh), Traffic Director, and Cloud Load Balancing (CLB) (opens in a new tab) control how requests split between versions. Feature flags add a separate application-layer switch so code can deploy without becoming visible to all users.
Google Kubernetes Engine (GKE) rolling deployments replace Pods incrementally so the application stays available during the update. The Deployment controller creates new Pods with the new container image while traffic flows from the Service to a mix of old and new Pods, then terminates old Pods only after new Pods pass readiness probes. Because the Service depends on Pod labels and readiness, unready Pods never receive production traffic and failed rollouts can halt before all replicas change. Choose rolling updates when the goal is continuous availability with minimal extra capacity, accepting that two versions briefly serve traffic at the same time.
Blue/green deployment keeps two full environments where the blue environment serves all production traffic and the green environment runs the new version for validation. On GKE, the blue and green versions run as separate Deployments behind one Service; on Cloud Run, they run as separate revisions behind one service; on App Engine, they run as separate versions behind one service. Cutover happens when the routing rule is changed so traffic flows from the load balancer or Service to the green version instead of the blue version, and rollback flows traffic back to the blue version without redeploying. Choose blue/green when the release needs full pre-production testing on identical capacity and an instant, complete switch with fast reversal.
Canary deployment exposes the new version to a small fraction of live traffic, analyzes telemetry, and then expands the fraction in stages. Traffic splitting makes this possible because the routing layer sends a defined percentage of requests to the canary while the remainder continues to the stable version, so blast radius stays limited to the canary share.
A canary rollout follows an ordered progression:
On Cloud Run and App Engine, traffic splitting is native because traffic flows from the service URL to multiple revisions or versions by percentage and can migrate gradually. On GKE, precise splitting depends on Anthos Service Mesh, Traffic Director, and CLB, where Envoy sidecars or proxyless xDS clients receive routing policy from the control plane and split traffic flows from the ingress gateway to stable and canary Services by weight, header, or cookie. Choose header or cookie-based splitting when internal testers or specific cohorts must reach the canary deterministically, and choose weight-based splitting when the goal is random sampling of production load.
Feature flags isolate deployment of code from exposure of behavior by wrapping new logic in a conditional that reads a flag value at request time. The new code deploys to all instances through the normal rolling, blue/green, or canary pipeline, but requests flow from the flag evaluation to either the old code path or the new code path depending on user, cohort, percentage, or environment. Because the flag service or configuration depends on application startup only for its default, operators can disable a failing feature without redeploying or rerouting infrastructure traffic. Choose feature flags when the release needs per-user targeting, long-lived experimentation, or instant kill-switch control that traffic percentages alone cannot provide.
Success metrics for progressive releases define when a progressive release may advance, and the metrics flow from application and machine learning (ML) pipelines to monitoring, alerting, and promotion decisions. A telemetry framework compares the canary against the stable baseline over the same window so infrastructure noise does not mask regressions. Core signals to compare include:
Promotion depends on all guardrails passing together, so a canary with good latency but rising errors or degraded prediction quality does not advance. When any metric breaches its threshold, traffic flows back to the stable version and the canary revision is held for analysis, which keeps automated progression safe across application and ML workloads.
| Strategy | Traffic pattern | Choose when |
|---|---|---|
| Rolling update | Traffic flows from the Service to old and new Pods simultaneously during replacement | Availability matters most and brief version mixing is acceptable |
| Blue/green | Traffic flows 100% to blue, then 100% to green after validation | Full isolation and instant cutover with instant rollback are required |
| Canary with traffic splitting | Traffic splits by percentage, header, or cohort between stable and canary | Blast radius must stay small while live telemetry validates each stage |
Exam tip: Use revision or version traffic splitting to control what fraction reaches new infrastructure, and use feature flags to control which users see new behavior inside the same version.
Monitoring ML pipeline telemetry and model performance is the practice of collecting serving, pipeline, and model-quality signals to decide when a deployed model is healthy or needs redeployment. On Google Cloud, Vertex AI endpoints, Dataflow pipelines, and GKE load balancing each emit telemetry that covers a different layer: endpoint resource pressure, pipeline data flow and inference results, and serving distribution and latency. Together these signals let teams track execution health, compare model versions, and trigger updates when performance degrades.
Vertex AI online prediction autoscaling telemetry is the set of per-replica metrics that control how many endpoint replicas serve a deployed model. The autoscaler watches the selected metric from each replica and adds or removes replicas between a configured minimum and maximum so the metric stays near its target threshold. Configuration happens at deploy time with gcloud ai endpoints deploy-model, which takes the endpoint ID, model ID, machine type such as n1-standard-4, optional accelerator type such as NVIDIA_L4 and accelerator count, minimum and maximum replica counts, metric name, and target threshold.
The supported autoscaling metrics cover different bottlenecks:
aiplatform.googleapis.com/prediction/online/cpu/utilization): percent per replica, target 0-100, default 60%.aiplatform.googleapis.com/prediction/online/accelerator/duty_cycle): percent per replica, target 0-100, default 60%.prometheus.googleapis.com/vertex_dcgm_fi_dev_gpu_util): percent per replica, disabled by default.prometheus.googleapis.com/vertex_vllm_gpu_cache_usage_perc): cache percent per replica, disabled by default.prometheus.googleapis.com/vertex_vllm_num_requests_waiting): waiting requests per replica as an integer, disabled by default.aiplatform.googleapis.com/prediction/online/request_count): requests per minute per replica as an integer, disabled by default.pubsub.googleapis.com/subscription/num_undelivered_messages): undelivered messages per replica as an integer, disabled by default with no Google Cloud CLI support.The Google Cloud CLI uses shorter metric keywords including cpu-usage, gpu-duty-cycle, dcgm-fi-dev-gpu-util, vllm-gpu-cache-usage-perc, vllm-num-requests-waiting, and request-counts-per-minute. Because each metric tracks a different bottleneck, the deployment depends on choosing the metric that matches the model server: a GPU-heavy large language model depends on GPU or KV cache signals, while a request-bound service depends on request count or queue size.
Exam tip: A deployment scaled only on CPU utilization will not scale up when GPU usage is high, so GPU workloads must explicitly configure a GPU, DCGM, or vLLM metric.
Dataflow ML (opens in a new tab) is the framework for deploying and managing complete machine learning pipelines for batch and streaming data on Dataflow. Data flows from a source such as Pub/Sub or Cloud Storage through Apache Beam transforms to inference and then to downstream processing, so pipeline health, throughput, and prediction output can be observed in one execution graph. The RunInference transform applies a model handler to each element for local inference with PyTorch, scikit-learn, TensorFlow, spaCy, Gemma, or Hugging Face models, or for remote inference by sending requests to a Vertex AI endpoint (opens in a new tab).
The MLTransform class wraps multiple preprocessing transforms in one step to prepare data for training and to keep training and serving inputs consistent. Common preprocessing and enrichment patterns in the pipeline include:
Because preprocessing, inference, and post-processing share the same pipeline, a change in input distribution, embedding quality, or remote endpoint latency appears as pipeline telemetry alongside prediction results. The pipeline depends on the model handler for inference logic and on the Vertex AI endpoint for remote serving capacity, so endpoint autoscaling metrics and pipeline throughput must be read together.
Model evaluation and continuous update in pipelines is the use of in-pipeline analysis and hot-swapping to compare versions and replace a degraded model without stopping serving. TensorFlow Model Analysis in Beam investigates and visualizes performance by creating and comparing two models inside the same pipeline, which supports side-by-side checks before promotion. A keyed model handler extends the same idea by running inference with multiple differently trained models on a per-key basis, so traffic segments can be scored by different versions.
Automatic model refresh uses Beam side inputs to deliver a new model to a running pipeline, so the pipeline pulls the updated artifact and applies it to new elements without a restart. This pattern lets a telemetry-driven decision — falling accuracy, rising error rate, or detected drift in embeddings and predictions — flow directly into redeployment of the pipeline's model reference. The running pipeline therefore depends on the model artifact source for freshness and on evaluation output for the decision to switch.
Serving distribution and observability on GKE is handled by two load balancing options that differ in which signals they route on and expose. GKE Inference Gateway extends the GKE Gateway API with InferencePool and InferenceModel resources for AI-aware routing, while GKE Gateway with Custom Metrics uses Application Load Balancers and Open Request Cost Aggregation (ORCA) headers for general-purpose routing. The gateway distributes client requests to the most suitable model server replica, and the replicas report load signals back to the gateway and to autoscaling.
| Option | Routing signals | Routing behavior | Observability |
|---|---|---|---|
| GKE Inference Gateway | Native GPU or TPU utilization, KV cache utilization and hits, request queue length, LoRA adapter availability, per-request cost | Model-aware splitting, mirroring, priority and criticality levels, LoRA affinity, cost-aware distribution for non-uniform LLM requests | Built-in AI metrics including GPU or TPU utilization, KV cache hits, queue length, and model latency |
| GKE Gateway with Custom Metrics | Application-reported ORCA metrics including CPU, memory, and custom-named constrained resources | WEIGHTED_ROUND_ROBIN endpoint weighting within a locality, best for uniform-cost requests | Custom metrics the application reports, viewed in Cloud Monitoring |
Choose GKE Inference Gateway when serving depends on LLM-specific state such as KV cache pressure, queue buildup, variable prompt cost, LoRA placement, request prioritization, or Model Armor filtering at the gateway. Choose GKE Gateway with Custom Metrics when load can be described by one or two application metrics and consistency with existing Application Load Balancer deployments matters. In both cases, serving telemetry flows from replicas to the load balancer to the Horizontal Pod Autoscaler, so routing decisions and scaling decisions share the same load signals.
Progressive delivery, SLI-based health gates, and ML telemetry form one closed loop: traffic splitting limits blast radius, Cloud Monitoring and Cloud Logging compare canary against stable on latency, errors, saturation, and model quality, and alert-driven rollback or model-registry restore returns traffic to the stable version. Vertex AI autoscaling, Dataflow pipeline telemetry, and GKE AI-aware routing extend the same loop to prediction pressure and data drift, so both application revisions and model versions advance only when telemetry proves success.
A rolling update on Google Kubernetes Engine (GKE) replaces Pods incrementally while the Service sends traffic to a mix of old and new Pods, a blue/green deployment keeps two full environments and switches routing from 100 percent on blue to 100 percent on green for instant cutover, and a canary deployment with traffic splitting sends only a small defined percentage of requests to the canary while the remainder stays on the stable version and then expands only after telemetry passes.
Use feature flags to control which users see new behavior inside the same version, and use revision or version traffic splitting to control what fraction of traffic reaches new infrastructure. Choose feature flags when the release needs per-user targeting, long-lived experimentation, or instant kill-switch control that traffic percentages alone cannot provide.
An automated rollback is a closed loop in which an alerting policy in Cloud Monitoring (Cloud Monitoring) detects a service-level indicator breach of its service-level objective and its notification through channels such as email, Slack, PagerDuty, or a Cloud Pub/Sub (Cloud Pub/Sub) topic triggers a return to the previous stable revision. Cloud Deploy (Cloud Deploy) then halts promotion and restores the previous stable state, while the incident record preserves status, logs, metric charts, and duration for audit.
GPU-heavy large language models must explicitly configure a GPU, Data Center GPU Manager (DCGM), or vLLM metric such as GPU duty cycle, DCGM GPU utilization, vLLM GPU KV cache usage, or vLLM requests waiting. Vertex AI defaults CPU utilization and GPU duty cycle to 60 percent, and a deployment scaled only on CPU utilization will not scale up when GPU usage is high.
Your team manages a high-traffic microservices application running on Google Kubernetes Engine (GKE) and is introducing automated canary deployments through your CI/CD pipeline.
You have established request-based availability and latency Service-Level Objectives (SLOs) in Cloud Monitoring. During canary rollouts, you need to evaluate the health of the canary version in real time and automatically trigger a deployment rollback if the canary release severely degrades application performance and rapidly depletes your error budget.
Which architectural approach should you implement to satisfy these requirements?
Configure a scheduled BigQuery query to run every 12 hours against exported Cloud Logging tables, and invoke an App Engine task to revert the deployment when error counts increase.
Build a custom dashboard with Metrics Explorer charts, and configure a Cloud Monitoring snooze schedule during the canary deployment window.
Configure an SLO burn-rate alerting policy with a short lookback period to detect fast-burn conditions on the canary service, and attach a Cloud Pub/Sub notification channel that triggers an automated rollback service.
Create a metric-absence alerting policy on the HTTP request count metric with an incident duration of 60 minutes, and send an email notification to the on-call DevOps engineer.
Last updated