Prepare and test your skills
Prepare and test your skills
Canary deployment with progressive rollout is a controlled release strategy that exposes a new model version to a small share of live traffic first and then increases that share in stages. The stable model version continues to serve most traffic while the canary version serves a bounded remainder, so prediction quality, latency, and errors can be compared under real production load. Traffic advances only after health checks and success criteria pass, and traffic returns to the stable version immediately when rollback triggers fire.
Traffic splitting for incremental exposure is the mechanism that divides live requests between the stable model version and the canary model version. App Engine supports traffic splitting between versions for Standard and Flex applications, Cloud Run applies the same principle between revisions, and Google Kubernetes Engine (GKE) supports Kubernetes-based deployments where separate Deployments or services receive weighted traffic. Request flow moves from the front-end router to both model versions according to configured weights, and progressive rollouts (opens in a new tab) shift those weights over time from the stable destination to the canary destination.
A progressive canary follows an ordered sequence:
This sequence limits blast radius because most users remain on the stable version during early validation, and each increase depends on successful completion of the prior stage.
Controlled ramp-up rates protect model serving and downstream dependencies from sudden traffic spikes during a release. Cloud Tasks (opens in a new tab) gradually ramps up dispatch rate toward the configured maxDispatchesPerSecond when a queue is new or has been idle, rather than dispatching immediately at full rate. If task creation is spiky with bursts separated by more than a week of inactivity, this ramp-up behavior is expected and preserves queue health. Newly created queues and groups of high-throughput queues, such as queue0000 through queue0199, are especially vulnerable during initial rollout and require the same gradual treatment as single queues.
The same protection applies when a model release increases traffic to a queue or queue group, when traffic moves between queues, when traffic rebalances across more or fewer queues, and when batch jobs inject large numbers of tasks. Gradual rollout smooths the increase so downstream model servers, queues, and task handlers scale safely instead of receiving the full peak at once.
Exam tip: Limit the initial launch to 500 total operations to new queues and increase by no more than 50% every 5 minutes when a release significantly increases queue traffic.
Ordered rollouts across clusters and fleets sequence progressive exposure from low-risk environments to critical production environments. The mesh-modernization-order label controls sequential order with three waves: early for the first wave, default for the second wave, and late for the third and final wave. When a fleet modernization is triggered, each cluster group modernizes sequentially and waits for automated steps on the current group to complete before beginning the next group. This ordering behavior is part of Cloud Service Mesh (opens in a new tab) managed control plane modernization.
Cluster rollout ordering applies to both self-service and automated pathways and is set on each cluster with Google Cloud CLI. Fleet rollout ordering applies to automated organization-wide scheduling and is set as a project-level label on the fleet host project, where Google completes each tier before starting the next in the order early, then default and unlabeled, then late. Fleets marked as deferred are excluded from automated ordering, and ordering cannot be controlled across fleets when Google Cloud organizations are not used.
An organization with multiple fleets does not need one approach for all fleets:
--modernization-strategy deferred while dependencies are remediated or a later self-service execution is planned.This combination lets a team validate a new model serving stack in non-production fleets marked early, leave others at default, and hold an especially critical fleet at late or deferred until confidence is high.
Health checks, success criteria, and rollback triggers are the control plane that decides whether a canary advances to the next segment or reverts to the stable version. Health checks observe liveness and serving errors, success criteria define acceptable latency, error rate, and model quality thresholds for each stage, and rollback triggers define the degradation or error conditions that return traffic to the stable version. The rollout depends on both standardized platform monitoring and the team's existing application monitoring, because platform signals catch infrastructure failures while application signals catch model-specific regressions.
Control can be self-service fleet-by-fleet or automated across the organization:
| Rollout control | Scope | Prerequisite | Rollback behavior |
|---|---|---|---|
| Customer-triggered modernization | Individual fleet identified by FLEET_PROJECT_ID | Target fleet must be modernization-compatible | Team decides when to initiate and when to roll back at the fleet level, with immediate self-service rollback using Google Cloud CLI driven by application monitoring alerts |
| Google-driven modernization | Entire Google Cloud organization covering all non-deferred fleets | All non-deferred fleets in the organization must be modernization-compatible | Platform rolls back when standardized monitoring shows an error, while application-specific errors detected by team monitoring require contacting Cloud Customer Care |
Self-service control fits model canaries because the team defines stage-specific success criteria, watches prediction and serving metrics at each traffic weight, and executes rollback immediately when triggers fire rather than waiting for a central schedule.
A/B testing for model comparison is the practice of deploying two versions of a model to the same serving endpoint and splitting live prediction traffic between them. In Gemini Enterprise Agent Platform, the Endpoint (opens in a new tab) layer is decoupled from the models, so one stable endpoint URL can route requests to multiple DeployedModel resources with configurable percentages. This design lets teams compare accuracy and serving behavior on real production traffic and then gradually shift traffic to the winner without changing the application.
Experiment parameters for an A/B test define which model variants compete, what percentage of traffic each variant receives, and which key performance indicators decide the winner. A valid comparison requires production signals for both quality and serving performance, including prediction accuracy improvements from new training data, inference latency and throughput, and ongoing quality drift. Because compute resources are associated with the DeployedModel rather than the Endpoint, each variant can carry its own machine type, accelerator, and replica counts, which keeps performance measurements isolated to that variant.
To collect a statistically meaningful sample, the test must run long enough and receive enough request volume at its assigned traffic percentage to distinguish a real difference from noise. The standard pattern is to start the challenger on a small share, such as 20%, while the champion serves 80%, and then gradually increase the challenger share toward 100% only after metrics favor it. All traffic percentages assigned to an Endpoint must add up to 100%, so every change to one variant requires an explicit update to the others to preserve data integrity.
A shared Endpoint is the traffic splitting (opens in a new tab) mechanism that routes inference requests from one service URL to two DeployedModel resources. Traffic flows from the client application to the Endpoint URL, and the Endpoint then forwards each request to one of the deployed variants according to the configured split. Deploying both variants behind the same URL avoids breaking changes, avoids updating the application to point to a new endpoint, and avoids a sudden cutover in application behavior.
Best practice is to deploy models of the same type to one Endpoint, for example two AutoML tabular models or two custom-trained models, because mixed types are harder to manage and compare. Deploying the same model to more than one Endpoint serves a different goal: supporting separate testing and production environments, different service-level objectives, or different cost and performance tradeoffs with higher-performance versus lower-performance machine resources. Choose a shared Endpoint when the goal is direct comparison and gradual replacement; choose separate Endpoints when the goal is environment or SLO isolation.
| Deployment pattern | Routing behavior | Choose when |
|---|---|---|
| Multiple models on one Endpoint | One Endpoint URL splits traffic, such as 20% to new and 80% to old | Comparing versions and gradually replacing the champion without a URL change |
| One model on multiple Endpoints | Each Endpoint URL serves independently with its own resources | Isolating testing from production or supporting different SLOs and costs |
Configuration of an A/B test creates the Endpoint once and then deploys each Model variant to it with an explicit traffic allocation. Deployment settings for location, container defined in ModelContainerSpec, and compute resources are fixed at deployment time and cannot be changed in place; to change them the model must be redeployed. The Google Cloud console (opens in a new tab) workflow and API workflow follow an explicit order:
https://.../v1/projects/{project}/locations/{location}/endpoints/{endpoint}:predict URL used for inference.--traffic-split mapping that sums to 100 across all variants.The traffic-split syntax maps each DeployedModel to a percentage of Endpoint traffic. A first deployment with no competitor uses --traffic-split=0=100, where 0 is the temporary ID for the new DeployedModel receiving 100% of traffic. An A/B deployment uses --traffic-split=0=20,OLD_DEPLOYED_MODEL_ID=80, which sends 20% of requests to the new variant and 80% to the existing variant identified by OLD_DEPLOYED_MODEL_ID. In the console, the same rule applies: a new deployment defaults to 100, and when other models already serve on the Endpoint, every percentage must be edited so the total remains 100.
Exam tip: Every traffic split on one Endpoint must total exactly 100 — a new 20% variant requires reducing existing variants to a combined 80%.
Isolation in an A/B test comes from per-variant compute and autoscaling, while data integrity comes from consistent logging and monitoring across variants. Each DeployedModel specifies its own machine type, where the default is n1-standard-2, plus a minimum replica count that must be greater than or equal to 1 and a maximum replica count that bounds scale-out. Vertex AI Inference autoscaling then scales the number of inference nodes for each variant based on concurrent requests, so changing request load adjusts capacity while minimum nodes remain available and billable even without traffic.
Monitoring flows from each variant to prediction logs, inference metrics, and Model Monitoring, which can monitor several versions of a model on the same Endpoint to support rapid iteration. Operators observe logs and per-variant inference metrics to compare latency, throughput, error behavior, and prediction quality in production. When monitoring objectives are configured with training-data location and target column, the system can track quality drift per variant, which provides the evidence needed to promote the challenger, roll back to the champion, or continue the test at a larger traffic share.
Analyze Results and Make Data-Driven Deployment Decisions is the process of turning a controlled rollout into a promotion, iteration, or rollback choice based on evidence. Gemini Enterprise Agent Platform decouples the endpoint layer from the models behind it, so two versions can serve the same endpoint URL while traffic is split between them. Results are judged by comparing business and technical metrics per version, checking whether differences are real and sustained against predefined success thresholds, and weighing gains against operational cost.
Traffic-split comparison on a shared Endpoint (opens in a new tab) lets a new model prove itself without changing the application URL or forcing a sudden cutover. A client sends an online inference request to the Endpoint URL, and the endpoint routes each request to one of its DeployedModel resources according to the configured percentages. All traffic percentages must add up to 100, for example --traffic-split=0=20,OLD_DEPLOYED_MODEL_ID=80 sends 20% to the new deployment and 80% to the existing deployment, while --traffic-split=0=100 sends all traffic to the new deployment.
Because compute resources are associated with the DeployedModel rather than the endpoint, each version can run with its own machine type, accelerator, and replica settings while sharing one entry point. The recommended pattern is to start the challenger on a small share, observe per-version behavior, and then gradually increase the challenger share until it serves 100% of traffic. This gradual migration limits blast radius while producing live comparison data from the same endpoint under the same request mix.
Evaluation per model version means separating results by DeployedModel and comparing the same signals for control and challenger over the same period. Model Monitoring on Agent Platform v2 can monitor several versions of a model at once, which supports rapid iteration during A/B testing and canary deployments. Online prediction logs and endpoint inference metrics provide the raw evidence for serving performance and model behavior in production.
The comparison covers both outcome and serving signals:
A version is a candidate for promotion only when its improvement persists across enough traffic and time to be treated as significant, not a brief spike in one slice. Differences must clear the predefined success thresholds for both business and technical metrics before traffic is increased further.
Cost and scaling assessment determines whether a metric win is worth running at scale. Vertex AI Inference autoscaling scales the number of inference nodes based on concurrent requests, bounded by the deployment's minimum and maximum replica counts. The minimum must be at least 1 and defaults to 1 when omitted, the default machine type is n1-standard-2, and some compute settings can be changed with a mutate operation without creating a new deployment.
Several design choices shift the latency, throughput, and cost balance for each version. Dedicated private endpoints and optimized online serving with Private Service Connect may reduce latency, while larger machine types and accelerators can improve throughput at higher cost. By default each model uses a separate virtual machine (VM), but model co-hosting can deploy several models on a single VM to improve utilization. Reservations, Dynamic Workload Scheduler, and Spot VMs provide additional levers to assure availability or lower cost for a version that will be promoted. These tradeoffs are central to performance optimization (opens in a new tab) for AI and ML serving.
Exam tip: Use traffic-split A/B deployment with multiple DeployedModel resources on one endpoint for gradual comparison, and use rolling deployment when the goal is to replace a deployed model while reusing its compute resources.
The deployment decision synthesizes metric comparison, significance, and cost into one of three endpoint actions. Promotion increases the winner toward 100% and removes the loser, iteration keeps the split while the model or configuration is fixed, and rollback returns traffic to the prior stable version. The endpoint URL stays constant in all three cases; only the traffic split and the set of DeployedModel resources change.
| Decision | Condition | Endpoint action |
|---|---|---|
| Promote | Challenger clears quality, latency, and cost thresholds with sustained significance | Gradually raise challenger share to 100%, then undeploy the old DeployedModel |
| Iterate | Results are mixed, inconclusive, or cost is too high for the observed gain | Hold a small challenger share, retrain or resize compute, then re-evaluate |
| Roll back | Challenger misses thresholds, regresses quality, or degrades serving health | Shift traffic back to the prior DeployedModel and undeploy the challenger |
First, confirm that per-version metrics were collected under the same endpoint and comparable load. Then, compare the challenger against thresholds and the control version, including scaling and cost impact. Only after the evidence supports promotion should traffic be shifted fully, since deployment settings such as location, container, and core compute choice require redeployment to change.
n1-standard-2 and a minimum replica count of at least 1.A canary deployment with progressive rollout exposes a new model version to a small share of live traffic first and then increases that share in bounded steps while the stable model version serves the remainder. Traffic advances only after health checks and success criteria for error rate, latency, and prediction quality pass, and traffic returns to the stable version immediately when rollback triggers fire.
A/B testing deploys two model versions as separate DeployedModel resources behind one stable endpoint URL that routes each inference request to one variant according to the configured percentages. Because compute resources are associated with the DeployedModel rather than the endpoint, each variant can use its own machine type, accelerator, and replica counts while serving the same request mix without changing the application.
All traffic percentages assigned to one shared endpoint must add up to exactly 100, for example 20 percent to the challenger DeployedModel and 80 percent to the champion DeployedModel. The standard pattern is to start the challenger on a small share and gradually increase it toward 100 percent only after metrics favor it, with every change requiring explicit updates to preserve the 100 percent total.
Promotion requires sustained improvement that clears predefined quality, latency, and cost thresholds, after which the challenger share is gradually raised to 100 percent and the old DeployedModel is undeployed. Iteration holds a small challenger share while the model or configuration is fixed when results are mixed or cost is too high, and rollback shifts traffic back to the prior stable version when the challenger misses thresholds or degrades quality or serving health.
Last updated