Drain Traffic, Add Capacity and Roll Back

Draining and redirecting traffic is the controlled removal of backends from serving and the shifting of user requests to healthy backends during deployments and incidents. Cloud Load Balancing (opens in a new tab) provides the control plane for this work through URL maps, backend services, and health checks. Traffic flows from pipeline automation to URL maps, from URL maps to backend services, and from backend services to specific backends, so each stage can be advanced, paused, or reversed without dropping in-flight work.

1.1Automated Drainage with Health Checks and Autohealing

Automated drainage makes traffic shifts a programmed pipeline stage instead of a manual cutover. A Compute Engine virtual machine (VM) can remain in the RUNNING state while its application has crashed, so autoscaling alone cannot guarantee responsiveness. Health checks probe each backend independently, traffic flows from the load balancer only to backends that respond successfully, and traffic stops flowing to backends that fail probes. When backends are managed instance groups (MIGs), autohealing adds a second flow where the MIG depends on its health check to identify unavailable VMs, deletes those VMs, and creates replacements so later pipeline stages have healthy targets.

1.2Weight-Based Splitting and Targeted Steering for Releases

Weight-based splitting is the mechanism for canary releases, A/B testing, and service migration on Application Load Balancers. The pipeline defines percentage-based splits across multiple backend services, so a new version receives a small share of live requests while the previous version serves the majority. Because the split is configuration, the pipeline can expand percentages after validation or reverse them instantly for rollback. A typical automated canary preserves order and validation gates:

1. Deploy new backend service: Bring up the new version alongside the previous version without sending it user traffic yet.
2. Split live traffic: Traffic flows from the load balancer to both backend services, for example 95% to the previous version and 5% to the new version.
3. Validate and expand: Only after health checks stay passing and canary analysis succeeds, gradually shift weights toward the new version.
4. Complete or roll back: Complete drainage when 100% reaches the new version, or reverse weights to the previous version when validation fails.

Traffic steering, actions, and policies drain targeted subsets. Steering routes by HTTP parameters such as host, path, and headers, so requests with user-agent:Mobile flow to mobile instances while other requests flow to defaults. Actions perform request or response operations such as redirects and header transformations, while policies fine-tune balancing behavior such as algorithms and request mirroring to a recording service.

1.3Connection Draining Timeout and Behavior by Load Balancer Type

Connection draining is the pattern that gracefully removes a backend so in-flight requests complete before shutdown. When an instance is removed from an instance group or an endpoint is removed from a zonal network endpoint group (zonal NEG), the backend service stops sending new connections to that backend while existing connections finish within a connection draining timeout of 0 to 3600 seconds. If no timeout is set, or the timeout is 0, existing connections end as quickly as possible. The timeout is set on the backend service, and when one instance group or zonal NEG is shared by multiple backend services, the largest timeout becomes the effective timeout.

Drainage behavior depends on whether the load balancer proxies the connection or only routes packets:

A passthrough Network Load Balancer cannot send RST or FIN itself. After its tracking entry is removed, closure depends on the newly selected backend sending an RST and the client reconnecting through normal backend selection.

1.4Global-to-Regional Failover with Cloud DNS Policies

Global-to-regional failover protects a global external Application Load Balancer with regional external Application Load Balancers and Cloud DNS. The primary is the global external Application Load Balancer that handles traffic normally, and each backup is a regional external Application Load Balancer that receives traffic when the primary fails health checks. Failover occurs when Cloud DNS routes traffic from primary to backups, while failback reverses routing after health checks pass. Regional load balancers work best as targets because they are self-contained within a region and isolated from global infrastructure in the same region.

Graceful degradation (opens in a new tab) keeps remaining backends available during shifts by serving reduced performance or accuracy under high load, then resuming full function when load falls. Pipeline orchestration combines independent replica throttling, early frontend drops before requests reach backends, partial-error handling with retries, and overload testing with spike monitoring. For globally served applications, Premium Tier carries traffic from the internet entry point of presence (PoP) closest to the sender across the Google network to the backend, which matters when traffic suddenly shifts regions.

Adding capacity is the combination of autoscaling, traffic shedding, and quota planning that keeps critical user journeys healthy when demand surges or a region loses backends. Cloud Load Balancing distributes traffic to the closest healthy backend with capacity, local autoscalers grow Pods or virtual machines (VMs) first, and only overflow that exceeds local capacity moves farther away. The goal is to prioritize critical traffic and degrade or redirect the rest rather than letting every journey fail.

2.1MIG Autoscaling on CPU, Serving Capacity, and Monitoring Metrics

A MIG autoscaling policy is the set of signals that tells the MIG autoscaler how many VMs a managed instance group (MIG) needs. The autoscaler collects usage for each signal, compares actual utilization to target utilization, and adds VMs when load rises or deletes VMs when load falls. When a policy has multiple signals, the autoscaler calculates a recommended count for each signal and sets group size to the largest recommendation. A new configuration includes average CPU utilization by default, with a 60-second initialization period that ignores data from initializing VMs.

MIG autoscaling supports four peer signal types that cover different load shapes:

• Average CPU utilization: maintains average CPU near a target such as 75%.
• HTTP(S) load balancing serving capacity: maintains a fraction of backend serving capacity defined in the load balancer backend service.
• Cloud Monitoring metrics: scales on built-in, Ops Agent, or custom application metrics.
• Schedules: scales on time-based schedules for predictable peaks.

Load-balancing-based scaling uses a balancing mode of UTILIZATION or RATE on the backend service. With UTILIZATION, the backend defines maximum backend utilization; with RATE, it defines target requests per second (RPS) per instance. For example, if capacity is 100 RPS per instance and the target is 0.8, the autoscaler holds load near 80 RPS per instance. Monitoring-based scaling uses utilization-target mode to hold a metric at a value, or single-instance-assignment to define work per VM. Pub/Sub queue-depth scaling must use pubsub.googleapis.com/subscription/num_undelivered_messages filtered to the subscription, because that metric counts unacknowledged messages including in-flight work and prevents scale-in while work is still running.

2.2GKE Pod, Node, and Hardware-Aware Scaling in GKE

GKE scaling splits responsibility between Pod replicas, Pod size, and node provisioning. The Horizontal Pod Autoscaler (HPA) changes replica counts from real-time load such as CPU, memory, or requests per second. A custom-metric pipeline moves data in order from Pods to the controller: first a PodMonitoring resource directs Google Cloud Managed Service for Prometheus to scrape Pod endpoints, then the Custom Metrics Stackdriver Adapter exposes the Cloud Monitoring metric to Kubernetes, then a HorizontalPodAutoscaler object targets the Deployment with minReplicas, maxReplicas, and an AverageValue target such as queue size. GPU-aware HPA follows the same flow using the container/accelerator/duty_cycle system metric reported every 60 seconds.

Node creation is not triggered by CPU load; it is triggered by an unfulfilled hardware request. A Pod requests hardware directly such as a GPU type or through a ComputeClass such as cloud.google.com/compute-class: premium-gpu, and auto-creation builds a node pool that satisfies those rules. Under surge the HPA first adds replicas, then cluster autoscaling adds nodes when replicas are unschedulable, while the VPA independently corrects Pod size.

2.3Service Capacity, Overflow, and Weight-Based Splitting

Service capacity is the implicit control that defines how much traffic a Service handles before the system autoscales or overflows. A common pattern sets max-rate-per-endpoint="10", meaning each Pod handles 10 RPS, so 2 replicas provide 20 RPS of capacity in that cluster. Traffic-based autoscaling grows replicas from the same signal with replicas = ceiling[ current traffic / ( averageValue * maxRatePerEndpoint )]. Overflow only occurs when all three conditions hold: a multi-cluster Gateway is used, the same Service runs across multiple clusters behind that Gateway, and capacity is exceeded in one cluster but not others. Excess then flows to the next closest region with capacity, while traffic within a region balances across zones in proportion to zone capacity.

Weight is explicit and capacity is implicit. Weight fixes exact proportions from HTTPRoute weights regardless of utilization, health, or client location, while capacity indirectly shapes shares from per-endpoint rate, utilization, and locality.

For example, a multi-cluster Gateway serves store from us-west1 and europe-west1 with 20 RPS per cluster. If Europe receives 30 RPS, 20 RPS stay in europe-west1 and 10 RPS overflow to us-west1. GKE Gateway Service Extensions add custom logic such as header modification, custom routing, and graceful degradation that routes non-essential journeys to cached or reduced responses to preserve critical capacity.

2.4Cloud Tasks Rate Controls and Gradual Ramp to Protect Targets

Cloud Tasks rate controls prevent sudden dispatch surges from overloading queues and downstream targets such as App Engine and Datastore. New or idle queues do not immediately dispatch at maxDispatchesPerSecond; dispatch ramps gradually toward the configured rate, and spiky traffic after long idle periods shows the ramp again as expected protection. For any queue dispatching more than 500 transactions per second (TPS), follow the 500/50/5 pattern: start at or below 500 TPS and increase by no more than 50% every 5 minutes. The pattern applies to the queue group as a whole when creating high-TPS queues, expanding ranges such as queue0000-queue0199 to queue0000-queue0399, launching releases that increase queue traffic, or resuming paused queues with large backlogs. App Engine traffic splitting applies the same ramp by starting with 1% to new queues and growing 50% every 5 minutes, while emergency expansion interleaves new queue names evenly in lexicographic order with no more than 50% new queues and each queue below 500 TPS.

2.5Quotas, Reservations, and Multi-Region Capacity Planning

Google Cloud quotas are per-project limits that prevent a few projects from monopolizing shared capacity. Resource quotas limit how many resources can be created, such as regional GKE clusters, while rate quotas limit API calls to a service in a period. Quotas apply at zonal, regional, or global scope, so exhaustion in one zone can block scaling there even when other locations have free capacity. Both types must be reviewed before peak load because autoscaling fails if either the VM count or the creation API rate is capped.

Proactive management forecasts need, watches consumption with Cloud Monitoring and load tests, and requests increases before an incident. Capacity Planner adds historical and forecasted use for VMs, Persistent Disk, and GPUs with reservation status and proximity to limits. The quota adjuster proactively requests increases during steady growth, while manual increases go through the console, a support case, or the account team. Denied console requests can be retried through support or the account team, but that retry is too slow during a critical incident. Reserved compute capacity assures Compute Engine capacity in a specific zone for chosen machine types, but an on-demand reservation succeeds only if capacity is available at request time, so it must be created before the surge. N+1 redundancy with three zones per region, at least two regions, and overprovisioning by a third lets the loss of one zone be absorbed without waiting for new physical capacity. Regional resources fail over across zones automatically, while multi-regional resources such as Cloud Storage and Spanner resist region loss, with traffic flowing from Cloud DNS and load balancing to healthy regional backends.

Rollback strategies define when a failed deployment must be reversed, how far the reversal reaches, and how the prior state is restored without losing data. Recovery time objective (RTO) answers how long after a disaster before service is running, while recovery point objective (RPO) answers how much data can be lost. A rollback is necessary when monitoring shows the new version cannot meet those objectives without prolonged degradation.

3.1Rollback Triggers Based on RTO, RPO, and Health Signals

Rollback triggers combine system health with business impact tolerance. System signals include automated consistency checks after each backup, backup health tracked by the Google Cloud Backup and DR service, adherence to backup intervals measured against RPO, and replication status and lag between primary and secondary environments. User signals include application performance degradation, failed synthetic scenarios in the recovered environment, and aggregate availability loss calculated as tier1_availability × tier2_availability × tierN_availability. Cloud Monitoring turns RPO into thresholds by alerting when backup intervals exceed the objective; if RPO is 15 minutes, backups must run at least every 15 minutes. Restoration testing must validate the full stack with restored data, not only the database, and production cutover must wait until both forward migration and fallback have been tested.

Automated thresholds act fast without human judgment, including Eventarc or Workflows triggering remediation from audit logs, autoscaling adding capacity during failover, MIGs recreating failed VMs, and regional load balancing shifting traffic. Manual or semi-automated decisions apply when replication is asynchronous across regions, when failover is constrained by data sovereignty, or when a migration task exceeds its timeboxed expected duration from a dry run. Increasing availability from 99% to 99.9% reduces downtime from 864 seconds to 86 seconds per day but increases cost, complexity, and operational overhead.

3.2Scoped Rollback and Full Fallback Execution

Rollback scope defines exactly what returns to a prior state, and narrow scope reduces blast radius. The Cloud Healthcare API method fhirStores.rollback illustrates scoping: the point-in-time parameter is mandatory and must be within the last 21 days in RFC 3339 format to the second with time zone, such as 2017-01-01T00:00:00Z. Scope can narrow by operation type, by resource type such as Patient, by identifiers such as Patient/123 stored in Cloud Storage, or by filters on resource.meta. The caller needs healthcare.fhirStores.rollback through Healthcare FHIR Store Administrator, the service agent needs write access to the Cloud Storage output destination, the output destination is required, production calls must set force to true, and a dry run writes predictions to Cloud Storage without changing store resources.

Fallback is the return to the source after cutover and must be treated as a full migration with planning and testing. Simple fallback switches applications back to the original source instance, which fits when downtime is acceptable or target transactions need not be preserved. When target transactions must be preserved, reverse replication synchronizes changes back to the source. A typical ordered recovery preserves integrity:

1. Create recovery state: Create resources from the deployment template, then apply the most recent backup and transaction logs from Cloud Storage.
2. Validate before traffic: Simulate user scenarios on the recovered environment with restored data.
3. Shift traffic: Configure Cloud DNS to point to the recovered web server or load balancer only after tests succeed.
4. Return when stable: Back up the recovery database, copy and apply it plus logs to production, block recovery connections, then point Cloud DNS back to production.
5. Restore protection: Verify copying to Cloud Storage and replication to standby, then delete the temporary deployment.

3.3Automated Infrastructure Recovery with IaC and Deployment Manager

Automated rollback restores prior infrastructure and application state without manual rebuilding. Infrastructure as code (IaC) tools like Terraform define and version the last known good configuration, so recovery re-applies a prior revision rather than editing live resources. Eventarc observes audit logs to trigger remediation, Workflows orchestrates multi-step recovery in order, and autoscaling absorbs shifted traffic during failover.

Google Cloud Deployment Manager (opens in a new tab) restores infrastructure from a template that defines servers, networks, and firewall rules using versioned custom images that match production. For databases, the image startup script copies the latest backup from Cloud Storage and invokes restore, while Compute Engine snapshots roll back persistent disks from multi-region locations. Traffic shifts to the recovered environment only after restoration and testing succeed, and returning to production reverses the flow with backup, copy, apply, DNS switch, verification, and cleanup.

3.4Post-Rollback Validation and HA Versus DR Tradeoffs

Post-rollback assessment confirms data is intact, recovery met time and loss targets, and procedures improve. Success is judged on data integrity of restored snapshots, RTO for resuming normal service, and RPO for recency of the restored point. Near-continuous options such as point-in-time recovery (PITR) for Cloud SQL or incremental backups for large datasets support tight RPOs, while active-active failover or cross-region replication improves RTO when backups alone are too slow. Operators must monitor open connections and request counts until draining completes, because pooled connections can still carry new requests to a draining VM and fail after deletion.

High availability (HA) generally means regional deployment with synchronous replication between zones and zero or near-zero RTO and RPO. Disaster recovery (DR) generally means multi-region deployment with typically asynchronous replication and non-zero RTO and RPO that require explicit planning and regular testing.

Key Facts

• Connection draining timeout on a backend service is 0 to 3600 seconds, where 0 disables draining, and the largest timeout wins when one instance group or zonal NEG is shared by multiple backend services.
• A Proxy Network Load Balancer closes all existing TCP connections on a draining backend within 10 minutes, the default idle timeout.
• Weighted traffic splitting across backend services takes precedence over session affinity for canary releases and rollback.
• Queues dispatching more than 500 TPS must follow the 500/50/5 pattern of starting at or below 500 TPS and increasing by no more than 50% every 5 minutes.
• High availability uses synchronous replication within a single region for zero or near-zero RTO and RPO, while disaster recovery uses typically asynchronous replication across regions with non-zero RTO and RPO.