Prepare and test your skills
Prepare and test your skills
Rollback execution and health-based traffic steering is the practice of shifting request proportions back to a known-good revision and using live health signals to decide when and how much to shift. In Google Kubernetes Engine (GKE), HTTPRoute resources define explicit weights between backends, while Cloud Monitoring metrics and load-balancer health checks provide the signals for manual or automated decisions. Traffic flows from clients to the Gateway (opens in a new tab), and the Gateway splits traffic to backends according to the configured rule, which lets operators reverse a bad rollout in seconds and then re-ramp safely.
Rapid rollback with weight-based splitting is a routing change that sends a fixed proportion of HTTP requests to each backend based on relative weights. Traffic flows from clients to the Gateway, and the Gateway splits traffic to backends according to the configured rule, for example 90% to store-v1 and 10% to store-v2. A rollback keeps the previous Deployment and Service running and then reverses the weights, such as returning from 10% on store-v2 to 100% on store-v1, so recovery does not require redeploying code.
Two splitting scopes support different rollback needs. Traffic splitting between Services handles version rollouts inside one cluster, where store-v1 and store-v2 are separate Deployments with their own Services and weights shift gradually until the new version is fully rolled out or rolled back. Traffic splitting between ServiceImports handles cluster-level moves, where ServiceImports represent multi-cluster Services and weights shift traffic to or from specific clusters for maintenance, migration, or emergencies.
Weight versus capacity is the distinction between explicit traffic proportions and implicit protection based on real load. Weight is an explicit control of traffic that defines exact proportions independent of incoming traffic, backend utilization, or backend health. If store-v2 is over capacity or all of its replicas have failed, the configured 10% is still allocated to store-v2 and can be dropped, because weight alone does not react to health. Capacity is an implicit control that shapes traffic indirectly based on incoming traffic, backend utilization, and client location, and it is typically updated much less frequently than weights.
The following table compares the two controls across how they are set and when to use them:
| Control | How proportion is set | Reacts to health and utilization | Best use |
|---|---|---|---|
| Weight | Explicit ratio in HTTPRoute | No; allocation stays fixed even if backends fail | Shifting between versions, manual onboarding, shifting away for emergency or maintenance |
| Capacity | Inherent Service property such as max rate per endpoint | Yes; excess overflows or triggers autoscaling | Preventing over-utilization during spikes, controlling autoscaling rate |
Weights and capacities should be used together for different purposes: weights decide the intended split for a rollout or rollback, while capacity protects backends when that intent meets real load. Weighted splitting also takes precedence over session affinity, so client requests can move to different Pods based on weights even when affinity by cookie, header, or client IP is enabled.
Exam tip: Weight still sends its fixed share even when the destination is unhealthy or over capacity; only capacity, health checking, and overflow react to utilization and failures.
Health checking, overflow, and autoscaling in GKE move traffic away from failed backends without waiting for a manual weight change. When the workload in gke-us fails, load balancing and health checking drain active connections and redirect traffic to the next closest cluster with healthy backends that have capacity. If more than 50% of backend Pods within a zone are unhealthy, traffic gradually fails over to other zones or regions independent of configured capacity.
Overflow to another region only occurs when all three conditions are present:
When those conditions hold, failure response follows a fixed order. First, health checking drains the failed cluster and sends its traffic to the next closest cluster. Then, if that cluster receives more traffic than its capacity, it sheds excess load to another cluster with spare capacity, such as gke-asia shedding to gke-eu. Then, the receiving cluster autoscales Pods to grow capacity where load increased, so regions grow to handle load as close to users as possible.
Steering with Cloud Monitoring metrics and safe ramp rates is the use of live signals and limited load increases to decide whether to hold, advance, or reverse a split. Operators use Cloud Monitoring metrics to watch dispatch rate, error rate, latency, backend utilization, and traffic growth proactively, and they configure alerts to detect dangerous increases before targets overload. This signal is critical after a rollback or resume, because a sudden jump from 0 TPS to full capacity can overload App Engine, Datastore, network targets, or newly enabled queues that hold a large backlog.
Safe recovery follows the 500/50/5 pattern: start new or resumed load at or below 500 transactions per second, increase by no more than 50% at a time, and wait at least 5 minutes between increases. The same rule applies to release-driven spikes, new high-TPS queues or queue groups, expanded queue groups, and unpausing high-TPS queues. For unpausing, stagger queue resumes or cap dispatch rate with maxDispatchesPerSecond; for bulk scheduled tasks, raise dispatch rate gradually with queue rate controls or spin up target capacity in advance; for increased fan-out where a new handler adds expensive downstream calls, use gradual rollout or traffic splitting to ramp up.
Traffic-based autoscaling complements manual steering by adding replicas as request rate grows. With a Service capacity of 10 requests per second (RPS) per Pod and a HorizontalPodAutoscaler target of 70% utilization, the controller targets replicas = ceiling[current traffic / (averageValue x maxRatePerEndpoint)], so 10 RPS computes to ceiling[10 / (0.7 x 10)] = 2 replicas and each replica receives about 5 RPS. In steady state the replica count approaches that target, but during spikes it is approximate, which is why monitoring plus explicit weight control remains necessary for rapid rollbacks.
Cloud Run revision and traffic tag management is the practice of controlling how client requests reach different deployed versions of the same service. A Cloud Run revision is an immutable snapshot of the container image, settings, and environment created by each deployment, while the Cloud Run service provides the stable endpoint that routes traffic across those revisions. Together, traffic percentages and revision tags let teams release with a gradual rollout (opens in a new tab), roll back quickly, compare versions, and test in isolation without changing the main service URL.
Cloud Run revisions as deployable versions are fixed snapshots that are created automatically each time a new container or configuration is deployed. Multiple revisions of the same Cloud Run service can exist at the same time, but only the revisions assigned traffic receive requests from the service URL. Older revisions remain available so traffic can be shifted back to them if the new revision has errors, which makes rollback a routing change rather than a redeployment. Choose revisions as the unit of release when each change needs to be tested, kept, or reversed independently.
Assigning traffic percentages to Cloud Run revisions means dividing incoming requests across two or more revisions by weight, such as 95% to the stable revision and 5% to the new revision. Traffic splitting applies to requests sent to the general service URL that does not target a specific revision, so the service endpoint distributes load while direct revision addresses bypass the split. A gradual rollout increases the new revision share in small stages while health, latency, and errors are monitored, and a rollback returns the stable revision to 100% to drain traffic from the faulty revision.
A safe gradual rollout follows an ordered sequence:
This staged pacing protects capacity and queue health because sudden jumps can overload downstream services such as databases and task handlers. When traffic must grow quickly, increase load by limited increments with time between steps so autoscaling and dependent systems can catch up before the next increase.
Using revision tags for A/B testing and isolated staging means attaching a custom name to a specific revision to create a directly addressable test URL. Traffic flows from the tag URL directly to the tagged revision, while traffic flows from the main service URL to revisions according to the configured percentages. Because the tagged address bypasses the percentage split, a revision can receive 0% of live traffic and still be fully testable by internal users, test automation, or stakeholders.
Revision tags support two common patterns. For A/B testing, two revisions each receive a share of live service traffic so behavior, performance, and conversion can be compared under real load. For isolated staging, the candidate revision receives no live traffic but is validated through its tag URL with production configuration before any percentage is shifted to it. In both cases, the tag preserves a stable test entry point even as percentage assignments change.
The following table compares the main traffic strategies on Cloud Run:
| Strategy | Purpose | Traffic setup |
|---|---|---|
| Gradual rollout | Release a feature with limited risk | Start with a small share to the new revision, then increase in stages |
| A/B testing | Compare two revisions under real traffic | Split service traffic between two revisions, such as 50% and 50% or 90% and 10% |
| Rollback | Restore stable behavior after a failure | Return 100% of service traffic to the previous stable revision |
Exam tip: Requests to the main service URL follow the traffic percentages, while requests to a tagged revision URL go directly to that revision and ignore the split.
GKE deployment patterns and service mesh integration is the use of Kubernetes traffic controls to run stable and experimental versions side by side and regulate how client requests move between them. In Google Kubernetes Engine (GKE) (opens in a new tab), Kubernetes Services expose groups of Pods as single network services, Ingress and Gateway resources connect those Services to load balancers, and Cloud Service Mesh manages traffic at the application layer for granular splitting. Together they support gradual rollouts that limit blast radius, immediate rollbacks by shifting weight back, and comparative testing by sending a portion of traffic to a new version.
Rolling deployments on GKE update servers one after another so the old and new versions of an app coexist until the rollout finishes. Traffic flows from the load balancer to a mix of old and new Pods, and as more servers are updated a larger share of users is routed to the new version until all users use the new version. GKE has built-in support for recreate and rolling deployment strategies through kubectl and Artifact Registry images, which keeps this pattern simple and resource efficient. Because two versions coexist, the pattern requires a load balancer configuration that avoids bouncing the same user between versions, and potential issues can be detected early before all users are affected.
Blue-green deployments on GKE run two full environments at once, where the blue environment serves production traffic while the green environment runs the new version for validation. When the green deployment is complete, all traffic switches from the old set of servers to the new set of servers at once, which avoids service interruption but temporarily requires up to twice the number of servers needed for production. A prerequisite is that the two versions can temporarily coexist without interfering, and for apps that access databases each schema change must stay backward compatible to at least the previous version.
A blue-green rollout on GKE follows an ordered sequence:
GKE primitives alone do not provide blue-green orchestration, so using a different strategy beyond recreate and rolling requires additional tooling. Spinnaker has built-in support for GKE and can orchestrate blue-green deployments, but Spinnaker is not a managed service and must be deployed and maintained separately on virtual machines or in a GKE cluster.
Canary releases and gradual traffic shifting on GKE move a small percentage of requests from the stable version to an experimental version, increase load while behavior is monitored, and perform a final cutover when confidence is high. The workflow depends on workloads first being validated with integration, load, compliance, and reliability testing and on observability signals from Google Cloud Observability, then being exposed to clients before any shift begins. After shifting starts, the team monitors how workloads behave as load increases and rolls back by shifting weight back to the stable version if error rates or latency degrade.
Three mechanisms from the documentation regulate that shift between a source and a GKE target:
| Mechanism | How traffic moves | Tradeoff |
|---|---|---|
| DNS resolution policy | DNS resolves a percentage of requests to IP addresses in the GKE environment and the rest to the source environment | Simple to implement, but lacks flexibility and depends on clients honoring DNS instead of aggressively caching past responses |
| Cloud Load Balancing with Network Endpoint Groups (NEGs) | A manually configured load balancer sends traffic to a standalone NEG that points to GKE workload instances and a hybrid connectivity NEG that points to source workload instances | Delegates balancing to Cloud Load Balancing, but requires manual load balancer configuration and complicates switching to a GKE-managed Gateway, Ingress, or LoadBalancer Service |
| Proxy with Cloud Service Mesh | A proxy deployed in the GKE environment forwards traffic to the source environment, and Cloud Service Mesh gradually shifts traffic from the proxy to GKE workload instances | Provides granular, application-layer control, but requires deploying and operating the proxy and mesh configuration |
Exam tip: GKE natively supports recreate and rolling strategies; use Spinnaker, Gateway HTTPRoute weights, or Cloud Service Mesh rules when you need blue-green or canary splitting between stable and experimental versions.
Regulating traffic with Services, Ingress, and Cloud Service Mesh means choosing the layer where the split between stable and experimental versions is enforced. Kubernetes Service resources expose an app running in a group of Pods as a single network service inside the cluster, while Ingress and Gateway resources connect those Services to external or internal load balancers that distribute client traffic into the cluster. Cloud Service Mesh, such as Istio, operates at the application layer above Services and Ingress and provides sophisticated traffic management with advanced routing policies and observability for percentage-based splits.
GKE load balancing choices map to the shape of the entry point:
For multi-cluster and fleet upgrades, multi-cluster Gateway is the recommended control plane. The GKE Gateway controller is a Google-hosted service that watches Gateway and HTTPRoute resources in a designated config cluster and provisions shared load balancing infrastructure, which decouples the load balancer lifecycle from individual GKE clusters. Operators implement blue-green upgrades by modifying weights in the HTTPRoute resource to move traffic from the blue cluster to the green cluster, rely on capacity-based balancing to protect a cluster from overload during rolling upgrades, and rely on health-based failover to reroute traffic away from a cluster being drained.
Weight is an explicit fixed proportion set in HTTPRoute that stays allocated even if backends fail or are over capacity, while capacity is an inherent Service property such as max rate per endpoint that reacts to utilization and health. Use weights to decide the intended split for a rollout or rollback and rely on capacity, health checking, and overflow to protect backends when that intent meets real load.
Requests to the main Cloud Run service URL are distributed according to the configured traffic percentages, while requests to a tagged revision URL go directly to that revision and bypass the split. A safe rollout keeps 100 percent on the stable revision, shifts a small share such as 1 percent to 5 percent to the new revision, monitors and expands in staged increments, then moves to 100 percent on the healthy revision or rolls back to 100 percent on the stable revision.
Health checking drains active connections from failed backends and redirects traffic to the next closest cluster with healthy backends that have capacity, with gradual failover to other zones or regions if more than 50 percent of backend Pods in a zone are unhealthy. Overflow to another region only occurs when the route uses a multi-cluster Gateway, the same Service is deployed across multiple clusters behind that Gateway, and capacities are configured so traffic exceeds capacity in one cluster but not in others.
Last updated