Autoscaling MIGs, GKE Workloads and Reserved Capacity

Dynamic autoscaling in Google Cloud automatically adds capacity when load rises and removes it when load falls to absorb traffic surges. Managed instance groups (MIGs) (opens in a new tab) in Compute Engine use a MIG autoscaler, while Google Kubernetes Engine (GKE) (opens in a new tab) splits the work between Pod-level scaling and node-level scaling. Fine-tuning means choosing the metric that tracks real work — CPU, load-balancing serving capacity, Pub/Sub queue depth, or a custom Cloud Monitoring (opens in a new tab) metric — and setting a target that scales early enough to protect users.

1.1MIG Autoscaling Policy Fundamentals

A MIG autoscaling policy is the set of scaling signals that tells the MIG autoscaler how many virtual machine (VM) instances a managed instance group (MIG) needs. The MIG autoscaler continuously collects usage for each signal, compares actual utilization to the configured target utilization, and adds VMs when load rises or deletes VMs when load falls. When a policy contains multiple signals, the MIG autoscaler calculates the recommended VM count for each signal and sets the group size to the largest recommendation.

MIG autoscaling supports four peer signal types that cover different load shapes:

• Average CPU utilization: maintains average CPU near a target percentage, such as 75%.
• HTTP(S) load balancing serving capacity: maintains a fraction of the backend serving capacity defined in the load balancer backend service.
• Cloud Monitoring metrics: scales on built-in metrics, Ops Agent metrics, or custom metrics exported from the application.
• Schedules: scales on time-based schedules for predictable peaks.

A new MIG autoscaling configuration includes a CPU utilization signal by default, and that signal can be edited, removed, or combined with other signals. The target utilization level is interpreted differently per metric, and an initialization period defaulting to 60 seconds tells the MIG autoscaler to ignore data from VMs that are still initializing so startup load does not distort decisions.

1.2Scaling MIGs on Load-Balancing Serving Capacity

Load-balancing-based autoscaling is the MIG autoscaling mode for a MIG that serves as a backend for an external or internal HTTP(S) load balancer. The load balancer distributes requests to backend services according to a URL map, and each backend service defines serving capacity (opens in a new tab) for its instance-group backends using a balancing mode of UTILIZATION or RATE. With UTILIZATION, the backend defines maximum backend utilization; with RATE, the backend defines a target number of requests per second (RPS) per instance or per group.

Traffic flows from the forwarding rule to the load balancer, from the load balancer to healthy VMs with remaining capacity in the MIG backend, and capacity grows because the MIG autoscaler adds VMs to the MIG backend. For example, if serving capacity is defined as 100 RPS per instance and the autoscaler target is 0.8 or 80%, the MIG autoscaler adds or removes instances to hold load near 80 RPS per instance. If all VMs are already at capacity, the target utilization or rate is exceeded until new VMs become healthy and rejoin serving.

1.3Scaling MIGs on Cloud Monitoring and Pub/Sub Metrics

Cloud Monitoring-based autoscaling lets a MIG scale directly on application work rather than on CPU alone. The MIG autoscaler creates VMs when the metric value increases and deletes VMs when the value decreases, using one of two target modes. Utilization-target mode maintains a metric at a target value, such as received bytes or latency, while single-instance-assignment mode defines how much work each VM should handle, such as messages per VM.

Pub/Sub queue-depth scaling is the pattern for rapidly absorbing backlog surges. The MIG autoscaler uses the pubsub.googleapis.com/subscription/num_undelivered_messages metric filtered to resource.type = "pubsub_subscription" and a specific resource.labels.subscription_id, combined with a single-instance-assignment value for messages per VM. The num_undelivered_messages metric counts total unacknowledged messages, including messages being processed but not yet acknowledged. A metric that excludes in-flight messages can drop to zero while work is still running, causing the MIG autoscaler to scale in and interrupt that work. Related patterns use the same mechanism: compute.googleapis.com/instance/network/received_bytes_count for incoming traffic, agent.googleapis.com/disk/operation_count filtered by direction or device for disk I/O, and instance_group/size to scale a backend MIG proportionally to a frontend MIG.

1.4GKE Pod Scaling With HPA and Custom Metrics

The GKE Horizontal Pod Autoscaler (HPA) (opens in a new tab) is the controller that scales Pod replicas in or out by watching real-time load signals. The HPA reacts to current consumption with the logic that current Pods are overwhelmed and traffic must be distributed immediately, so the HPA changes the replica count of a Deployment to meet demand. CPU and memory utilization are available to the HPA out of the box, while load-balancer requests-per-second requires utilization-based load balancing and the Performance HPA profile, and business or server metrics require an explicit metrics pipeline through Google Cloud Managed Service for Prometheus.

A custom-metric HPA pipeline moves data from Pods to the HPA controller in an ordered sequence:

1. Export metrics to Cloud Monitoring: A PodMonitoring resource directs Google Cloud Managed Service for Prometheus to scrape Pod endpoints, such as port 8000 every 15 seconds for a text-generation inference server.
2. Expose metrics to Kubernetes: The Custom Metrics Stackdriver Adapter makes the exported Cloud Monitoring metric visible to the HPA controller.
3. Scale on the metric: A HorizontalPodAutoscaler object targets the Deployment with minReplicas, maxReplicas, and a pods metric such as prometheus.googleapis.com|tgi_queue_size|gauge or prometheus.googleapis.com|tgi_batch_current_size|gauge with an AverageValue target.

GPU-aware HPA follows the same flow using GPU duty cycle. When system metrics are enabled, the GKE cluster sends GPU utilization to Cloud Monitoring through the container/accelerator/duty_cycle system metric every 60 seconds, and the HPA can then scale inference Pods on GPU saturation in addition to queue size or batch size.

1.5Cluster Capacity and Hardware-Aware Autoscaling in GKE

Cluster-level autoscaling in GKE provisions nodes when Pods cannot be scheduled, which lets Pod-level scaling actually place new replicas during a surge. Node pool auto-creation is not triggered by CPU load; it is triggered by an unfulfilled hardware request when a Pod stays Pending because no existing node has the requested hardware. A Pod can request hardware directly, such as a GPU type, or request a capability through a ComputeClass, such as cloud.google.com/compute-class: premium-gpu.

The ComputeClass separates the Pod request from provisioning by holding prioritized rules that tell GKE how to fulfill the capability, and node pool auto-creation then creates a new node pool with hardware that satisfies those rules. The three GKE autoscaling behaviors divide responsibility by signal source and action:

The components compose under surge: the HPA first adds Pod replicas from real-time metrics, then cluster autoscaling adds nodes when those replicas are unschedulable, while the VPA independently corrects Pod size when historical usage shows the original CPU or memory request was wrong.

Traffic shedding and load balancing mitigation is the practice of keeping critical user traffic healthy when a region or backend runs out of capacity. Cloud Load Balancing distributes traffic to the closest healthy backend with capacity, GKE autoscales Pods to grow local capacity, and explicit traffic-splitting rules shed or shift the excess load that cannot be served locally. The goal is to prioritize critical journeys and degrade or redirect the rest rather than letting every journey fail.

2.1How Load Balancing, Autoscaling, and Capacity Work Together

Cloud Load Balancing with multi-cluster GKE Gateways is the system that sends each client to the closest Google point of presence (PoP) and then across the Google backbone to the closest healthy backend with remaining capacity. The GKE Gateway controller (opens in a new tab) is the control plane that watches Gateway, HTTPRoute, Service, and Policy resources and programs that distribution. When demand rises in one place, local Pods autoscale first so traffic stays close to users; only the overflow that exceeds local capacity moves to the next closest cluster.

A regional failure shows the ordered interaction:

1. Drain and redirect: Health checking on the failed gke-us cluster drains active connections and redirects traffic from clients to the next closest cluster.
2. Shed excess: The gke-asia cluster receives more traffic than its configured capacity allows, so it sheds the excess load to gke-eu.
3. Grow capacity: The gke-eu cluster receives more than its typical load and autoscales Pods to increase its traffic capacity.

GKE exposes this behavior through related capabilities:

• Service capacity: the amount of traffic a Service can receive before Pods autoscale or traffic overflows to other clusters
• Traffic-based autoscaling: scaling Pods based on HTTP requests per second
• Multi-cluster load balancing: balancing to Services across multiple GKE clusters or regions
• Traffic splitting: explicit, weight-based distribution across backends
• Session affinity (GCPTrafficDistributionPolicy): pinning sessions by cookie, header, or client IP for stateful performance
• Cross-Origin Resource Sharing (CORS): allowing and securing cross-origin requests
• Edge caching: caching with Cloud CDN to lower latency and reduce origin load
• Custom traffic management with Service Extensions: injecting custom logic for advanced control

2.2Service Capacity and Global Traffic Overflow

Service capacity is an implicit control that defines how much traffic a Service should handle before the system autoscales or overflows. A common pattern sets max-rate-per-endpoint="10", meaning each Pod can receive a maximum of 10 requests per second (RPS). Total Service capacity is then replicas multiplied by per-Pod capacity, so 2 replicas at 10 RPS provide 20 RPS of capacity in that cluster.

Traffic overflow only occurs when all three conditions are true: you are using a multi-cluster Gateway, you have the same Service deployed across multiple clusters served by that Gateway, and Service capacities are configured so traffic exceeds capacity in one cluster but not in others. Under normal conditions traffic terminates at the closest PoP and goes to the nearest healthy backend with capacity. If that region has no remaining capacity, excess traffic overflows to the next closest region with capacity. If more than 50% of backend Pods in a zone are unhealthy, traffic gradually fails over to other zones or regions independent of configured capacity.

For example, a multi-cluster Gateway serves the store Service from us-west1 and europe-west1 with 2 replicas each at 10 RPS per Pod, or 20 RPS per cluster. North American PoPs receive 6 RPS and all 6 RPS stay in us-west1. European PoPs receive 30 RPS, but europe-west1 holds only 20 RPS, so 10 RPS overflows to us-west1, which then serves 16 RPS total at 8 RPS per Pod. Within a single region, traffic is instead balanced across zones in direct proportion to available zone capacity, with each individual flow kept on one consistent backend Pod.

Traffic-based autoscaling grows replicas from the same capacity signal. The HorizontalPodAutoscaler targets utilization with the equation replicas = ceiling[ current traffic / ( averageValue * maxRatePerEndpoint )]. With capacity of 10 RPS and averageValue of 70, the target is 7 RPS per Pod, so 10 RPS computes as ceiling[ 10 / (0.7 * 10) ] = ceiling[1.4] = 2 replicas, leaving each replica at 5 RPS. In steady state replica counts approach the target, but rapid spikes cause the autoscaler to approximate.

Prevent overflow when distant backends would hurt latency-sensitive journeys by using any of these approaches:

• Use only single-cluster Gateways, which host Services in only a single cluster.
• Deploy the same application as separate Services across clusters, so the Gateway sees multi-cluster endpoints without aggregating them as one Service.
• Set Service capacities high enough that capacity is not realistically exceeded.

2.3Traffic Splitting With Explicit Weights

Traffic splitting is an explicit, weight-based rule that defines the proportion of HTTP requests sent to each Service. An HTTPRoute resource lists Services with relative weights, and the Gateway divides traffic by those weights. A route configured 90% to store-v1 and 10% to store-v2 sends 90% of client requests arriving at the store URL to store-v1 and 10% to store-v2, regardless of current utilization.

Traffic splitting works between Services in the same cluster and between Services in different clusters. Splitting between Services uses two Deployments and two Services, such as store-v1 and store-v2, to gradually shift traffic during version rollouts and canary releases. Splitting between ServiceImports uses multi-cluster Service representations to shift traffic to or from specific clusters for maintenance, migration, blue-green routing, or emergencies. Traffic splitting between single-cluster Gateways is generally available, but the gke-l7-gxlb GatewayClass does not support it and the gke-l7-gxlb-mc GatewayClass does not support multi-cluster splitting.

2.4Weight Versus Capacity

Weight and capacity both influence where traffic goes, but weight is explicit and capacity is implicit. Weight fixes exact proportions independent of incoming traffic, backend utilization, and client location. Capacity indirectly shapes proportions because the load balancer considers incoming volume, backend utilization, and source location against each Service's inherent capacity, which changes infrequently.

Use weight and capacity together for different purposes: weight to declare where traffic should go, and capacity to protect backends and trigger growth when that intent meets real load.

2.5Rate Controls and Gradual Ramp to Protect Targets

Cloud Tasks (opens in a new tab) rate controls prevent a sudden dispatch surge from overloading queues and downstream targets such as App Engine, Datastore, and network capacity. New or idle queues do not immediately dispatch at maxDispatchesPerSecond; Cloud Tasks gradually ramps the dispatch rate toward the configured rate. Spiky traffic separated by long idle periods will show this ramp again, which is expected protection rather than a defect.

For any queue dispatching more than 500 transactions per second (TPS), follow the 500/50/5 pattern: start at or below 500 TPS and increase by no more than 50% every 5 minutes. Apply the pattern to the queue group as a whole, not to individual queues, when creating new high-TPS queues, expanding a group such as queue0000-queue0199 to queue0000-queue0399, launching releases that increase queue traffic, resuming paused queues with large backlogs, or increasing dispatches that trigger downstream work. Monitor the increase with Cloud Monitoring metrics and alerts for Cloud Tasks, stagger queue resumes, cap maxDispatchesPerSecond, or pre-provision target capacity before bulk scheduled dispatches.

App Engine traffic splitting applies the same ramp to version rollouts that create or expand queues. Starting with 1% of traffic to the new version sends about 500 TPS to the new queues when total creation is 100,000 TPS, then traffic to the new version grows by 50% every 5 minutes until the old and new queue ranges each carry 50%. For emergency expansion without a gradual rollout, interleave new queue names evenly among existing names in lexicographic sort order, such as adding queue0000a, queue0002a alongside queue0000-queue0199, and send traffic immediately only when there are no more than 50% new interleaved queues and each queue stays below 500 TPS. For millions or billions of enqueues, use a double-injection pattern where an injector queue fans out 100 tasks per message and ramps from 5 TPS by 50% every 5 minutes. Avoid custom sequential task names such as timestamps because they add latency and errors; use a well-distributed prefix such as a content hash. Cap client retries with adaptive throttling when server-side errors rise, since aggressive retries slow queue recovery.

2.6Custom Traffic Management With Service Extensions

GKE Gateway Service Extensions are the extension point that injects custom logic into the Gateway request and response path. Service Extensions support GKE traffic management by modifying HTTP headers and payloads, implementing custom routing logic to control traffic flow, and integrating with external services for functions such as authorization. This extension point provides the mechanism for graceful degradation patterns, such as routing non-essential journeys to cached, reduced, or rejected responses while preserving capacity for critical journeys when physical limits are reached. Support for Service Extensions is in Preview.

Navigating resource quotas and multi-region capacity means ensuring that per-project limits and localized shortages do not block scaling during an incident. Google Cloud capacity is shared across customers, and quotas control how much of that shared capacity one project can use. A resilient design pairs proactive quota headroom with pre-provisioned, multi-region footprint so traffic can fail over to healthy zones and regions without waiting for new approvals or new VMs.

3.1Google Cloud Quotas and Capacity Limits

Google Cloud quotas are per-project limits that prevent a few projects from monopolizing shared resources. Each quota represents a specific countable resource, such as concurrent VMs, concurrent load balancers, API calls to a service, or projects that can be created. Quotas apply at zonal, regional, or global scope, so a limit exhausted in one zone or region can block scaling there even when other locations have free capacity. Enabling billing for a project increases quotas for most services, and quotas can grow further as sustained use expands.

Resource quotas block creation of infrastructure, while rate quotas block the control-plane calls used to manage it. Both must be reviewed for every service in the stack before peak load or an incident, because autoscaling that needs new VMs will fail if either the VM count or the creation API rate is capped.

3.2Proactive Quota Monitoring and Adjustment

Proactive quota management is the practice of forecasting need, watching consumption, and requesting increases before an incident makes the limit urgent. Cloud Monitoring (opens in a new tab) provides visibility into average and peak utilization, and load tests establish how much load the system can handle at target latency with fixed resources. Capacity Planner (opens in a new tab) extends this view with historical and forecasted usage for VM instances, Persistent Disk volumes, and GPUs, plus reservation status and proximity to quota limits with automatic increase requests.

The quota adjuster monitors resource consumption and proactively submits quota adjustment requests on behalf of the project, removing the need for manual requests in steady growth. When a manual increase is still needed, there are three peer paths:

• Through the Google Cloud console, which is fastest when a quick decision is needed
• Through a support case, with live chat as the quickest connection when creating the case
• Directly with the account team or Technical Account Manager (TAM), which suits requests needing deeper analysis

Most console requests are evaluated by automated systems based on resource availability, length of Google Cloud use, and other factors. Denied requests can be retried through a support case or the account team, but during a critical incident that retry path is too slow, which is why headroom must be secured in advance.

3.3Reserved and Redundant Compute Capacity

Reserved compute capacity is assured Compute Engine capacity in a specific zone for a specified number of VMs of a chosen machine type, including memory, vCPUs, and optional GPUs and Local SSD disks. An on-demand reservation is provisioned at request time only if the requested capacity is available, so it must be created before the surge or outage. A reservation can be specific to one project or shared across multiple projects.

Redundancy planning determines how much extra reserved and running capacity to hold. N+1 redundancy requires at least one redundant component beyond the minimum needed for forecast load in every tier of the stack. For zone and region resilience, the guidance is to use three zones per region rather than two, distribute across at least two regions, and overprovision by a third more than the estimate so the loss of one zone can be absorbed. This trades higher cost for the ability to recreate failed VMs in another zone without waiting for new physical capacity.

3.4Multi-Region Deployment to Route Around Local Limits

A multi-region deployment is an architecture that delivers service from more than one region so a zonal or regional shortage does not stop user traffic. Compute Engine VMs and Persistent Disk are zonal resources hosted in a single zone, where a zone-scoped interruption can make all of those resources unavailable. Regional resources are redundantly deployed across multiple zones and automatically serve from another zone if one backing zone is interrupted, while multi-regional resources are distributed within and across regions for still higher resistance to region loss.

To use this scope hierarchy for incident mitigation (opens in a new tab), prefer the widest practical scope and compose narrower resources across locations:

• Use regional resources instead of zonal resources, and multi-regional or global resources instead of regional resources, to avoid restore-from-backup.
• Run identical VMs in multiple zones with instance templates and managed instance groups (MIGs), which automatically recreate failed VMs.
• Replicate data with regional Persistent Disk synchronous replication across two zones, multi-region storage such as Cloud Storage (opens in a new tab) and Spanner, regional-to-secondary-region replication, and snapshots stored in multi-region locations.
• Distribute user traffic across regions with Cloud DNS plus a regional load balancer in each region, or with a global load balancer depending on criticality and requirements.

Traffic therefore flows from Cloud DNS and the load-balancing tier to healthy regional backends, and health checks steer traffic away from an impaired zone. Regional outage mitigation must be tested regularly, because an untested assumption of cross-region failover often fails when a real regional event occurs.

3.5High Availability Versus Disaster Recovery Tradeoffs

High availability (HA) and disaster recovery (DR) split responsibility for zone versus region failure. HA generally applies to regional deployments with synchronous replication between zones, supporting zero or near-zero recovery time objective (RTO) and recovery point objective (RPO) in an active-active topology with no single point of failure. DR generally applies to multi-region deployments, where replication without multi-regional services is typically asynchronous and requires explicit assessment of RTO, RPO, and manual or semi-automated failover steps.

Even a regional product designed for 99.99% availability can still incur nearly an hour of downtime per year, so critical applications need a multi-region DR plan if that duration is unacceptable. Multi-region products trade latency, consistency, and cost, most commonly through synchronous versus asynchronous replication where asynchronous lowers latency but risks data loss during outage. Choosing managed multi-regional services reduces manual recovery, while self-composed regional failover preserves control at the cost of greater operational complexity and regular testing burden.

Key Facts

• The MIG autoscaler calculates a recommended VM count for each configured signal and sets the managed instance group size to the largest recommendation.
• MIG autoscaling supports maximum RPS per instance but not maximum RPS per group, and Pub/Sub queue-depth scaling must use num_undelivered_messages so in-flight messages are not ignored.
• Weighted traffic splitting from HTTPRoute weights takes precedence over session affinity and ignores health and utilization, while Service capacity implicitly overflows only when the same Service is served by a multi-cluster Gateway and one cluster exceeds its per-endpoint rate.
• Queues dispatching more than 500 TPS must follow the 500/50/5 pattern of starting at or below 500 TPS and increasing by no more than 50% every 5 minutes.
• N+1 redundancy with three zones per region, at least two regions, and overprovisioning by a third lets the loss of one zone be absorbed without waiting for new physical capacity.