Autoscaling for GKE, MIGs and Cloud Run

Multi-dimensional autoscaling in Google Kubernetes Engine (GKE) (opens in a new tab) coordinates three independent control loops that scale different dimensions of capacity. The Horizontal Pod Autoscaler (HPA) changes the number of Pod replicas, the Vertical Pod Autoscaler (VPA) changes the size of each Pod, and the Cluster Autoscaler changes the number of nodes that host the Pods. Node pool auto-creation extends the Cluster Autoscaler by creating entirely new node pools when pending Pods require hardware that does not exist in the cluster.

The table shows why the layers must be aligned rather than operated alone. HPA creates demand for placement, VPA changes the size of that demand, and infrastructure automation supplies nodes only after the scheduler proves that existing nodes cannot place the Pods.

1.1Horizontal Pod Autoscaler (HPA)

The Horizontal Pod Autoscaler (HPA) (opens in a new tab) is the workload controller that adds or removes Pod replicas based on real-time demand. The HPA controller reads a metric, compares the current value to the target value, calculates a desired replica count bounded by minReplicas and maxReplicas, and updates the target Deployment or other scalable workload. Deleting an HPA object does not revert the workload to its original replica count, so the Deployment remains at its existing scale until it is manually scaled with kubectl scale.

HPA supports three metric families that cover different scaling signals:

• CPU and memory utilization: available out-of-the-box and calculated as a percentage of Pod request, so the Deployment must set CPU or memory requests to use averageUtilization targets.
• Load balancer requests-per-second (RPS): a direct traffic signal that reacts faster than CPU, enabled through utilization-based load balancing and the Performance HPA profile.
• Custom and external metrics: application or business signals such as active users, queue size, batch size, or GPU duty cycle, exposed through a metrics pipeline.

Custom-metric autoscaling in GKE depends on an ordered pipeline where each stage feeds the next:

1. Export metrics to Cloud Monitoring: Google Cloud Managed Service for Prometheus scrapes the workload through a PodMonitoring resource, typically every 15 seconds, or GKE system metrics emit container/accelerator/duty_cycle every 60 seconds.
2. Install the Custom Metrics Stackdriver Adapter: the adapter makes the Cloud Monitoring metric visible to the HPA controller through the custom or external metrics API.
3. Deploy the HPA resource: the autoscaling/v2 HPA references the Deployment in scaleTargetRef and defines the metric with type: Resource, type: Pods, or type: External plus an averageUtilization or averageValue target.

Configure a target such as 70% or 80% utilization to preserve headroom while new replicas start. The HPA algorithm applies a default 0.1 tolerance band around the target to dampen oscillation, so load testing with tools such as locust-load-inference is used to select an averageValue that scales before saturation without flapping. Use autoscaling/v2 for new objects because autoscaling/v1 is supported but no longer recommended and does not support features such as scaling to and from zero.

1.2Vertical Pod Autoscaler (VPA)

The Vertical Pod Autoscaler (VPA) is the workload controller that right-sizes containers by adjusting CPU and memory requests and limits up or down. VPA analyzes historical consumption to detect when initial requests are too large or too small, then re-provisions the Pod with a corrected configuration, for example from 1 CPU and 16 GB to 4 CPUs and 64 GB. Because re-provisioning restarts the Pod, VPA trades placement stability for long-term efficiency and prevention of resource contention.

VPA is complementary to HPA when each controls a different signal, with HPA controlling replica count from traffic and VPA controlling container size from historical use. VPA is especially useful for workloads that do not use HPA or where peak demand is unknown and requests cannot be set accurately in advance. To avoid erratic sizing, keep VPA in Off recommendation-only mode for at least 24 hours and ideally one week in a production-like environment, then set explicit minimum and maximum bounds in the VerticalPodAutoscaler object before enabling Initial or Auto modes.

1.3Cluster Autoscaler

The Cluster Autoscaler (opens in a new tab) is the infrastructure controller that adds or removes nodes in GKE Standard node pools based on scheduling simulation rather than metric load. When HPA creates new Pods that cannot be scheduled due to lack of CPU, memory, or available capacity on existing nodes, the Cluster Autoscaler provisions a new node in an existing node pool. When nodes become underutilized, the Cluster Autoscaler moves their Pods onto other nodes and terminates the empty node to reduce cost.

The Cluster Autoscaler depends on schedulability and disruption controls on both sides of the relationship. The workload depends on the Cluster Autoscaler for placement of unschedulable Pods, while the Cluster Autoscaler depends on Pod scheduling constraints and availability guards to scale down safely. Always configure a PodDisruptionBudget (PDB) for system and application Pods so consolidation of underutilized nodes does not evict too many replicas at once. Specify minimum node parameters to preserve baseline night capacity, and use the Performance HPA profile carefully because it increases gke-metrics-agent requests and restarts its Pods, which can force rescheduling on resource-constrained nodes.

1.4Node Pool Auto-Creation and ComputeClasses

Node pool auto-creation is the GKE capability that extends the Cluster Autoscaler from scaling existing node pools to automatically creating new node pools with specialized hardware. While the Cluster Autoscaler adds nodes of a type that already exists, node pool auto-creation provisions node pools whose shape, CPU count, memory, or accelerators precisely fit the scheduling parameters of pending Pods. This minimizes waste from placing small specialized Pods on oversized nodes and supplies GPUs or TPUs on demand for AI/ML inference without pre-provisioning them.

Node pool auto-creation is triggered by hardware availability, not by CPU load. A trigger occurs when a Pod requests specific hardware and no existing node can satisfy it, leaving the Pod in Pending, or when a Pod requests a ComputeClass such as cloud.google.com/compute-class: premium-gpu and no node provides those capabilities. A ComputeClass is a GKE resource that defines a prioritized list of rules for fulfilling a capability request, separating the Pod request from the provisioning action. Instead of the Pod demanding a rigid machine type such as a3-highgpu-8g or a device such as nvidia-l4-vws, the Pod requests a capability and GKE selects the implementation defined by the ComputeClass. If demand for specialized hardware is static and predictable, such as a long-running training job, manually creating or updating a node pool with a specific machine type remains appropriate.

Managed instance group (MIG) (opens in a new tab) autoscaling automatically adds virtual machine (VM) instances when load rises and deletes them when load falls. A MIG is a collection of VMs created from a common instance template, and only managed groups can attach an autoscaler. The autoscaler is a separate API resource — autoscalers for zonal MIGs and regionAutoscalers for regional MIGs — that acts on the target MIG using the Compute Engine Service Agent to create and delete VMs.

2.1Autoscaling Policy and Scaling Signals

An autoscaling policy defines one or more scaling signals and the minimum and maximum number of replicas for a MIG. When multiple signals are configured, the autoscaler calculates a recommended VM count for each signal and sets the group size to the largest recommendation. A policy must always contain at least one signal, and enabling autoscaling adds a CPU utilization signal by default that can be edited, removed, or supplemented.

Target utilization signals supported by the policy include average CPU utilization across all vCPUs in the group, HTTP load balancing serving capacity defined in the backend service, and Cloud Monitoring metrics including built-in, Ops Agent, and custom metrics. The autoscaler continuously collects the selected metric, compares actual utilization to the desired target, and scales out to add capacity or scales in to remove capacity. This comparison keeps utilization near the target, for example maintaining average CPU near 75% by adding VMs when usage exceeds the target and removing VMs when usage falls below it.

2.2CPU Utilization Scaling

CPU-based autoscaling maintains average CPU utilization across the MIG at a configured target such as 0.60 for 60%. Data flows from each VM's CPU usage to the autoscaler, which averages usage across the group and then changes the MIG size to hold that average near the target. If the application takes a long time to initialize, avoid setting the target at 85% or higher because existing CPUs can overload while new VMs are still starting and not yet serving traffic.

CPU scaling depends on the initialization period and the stabilization period. During the initialization period (coolDownPeriodSec), default 60 seconds, the autoscaler ignores metrics from VMs that are still starting so incomplete startup data does not skew decisions. During the stabilization period (stabilizationPeriodSec), default 600 seconds and allowed from 0 to 3600 seconds, the autoscaler delays scale-in deletions to prevent thrashing when utilization briefly dips. A shorter stabilization period deletes VMs faster and saves cost sooner, while a longer period protects availability for spiky or slow-draining workloads.

2.3Cloud Monitoring Metric Scaling Modes

Cloud Monitoring (opens in a new tab) metric scaling adjusts capacity from application measurements such as latency, network bytes, memory, disk operations, or queue length. Metrics can be built-in Monitoring metrics or exported custom metrics, but scaling is limited to up to 5 Monitoring metrics per MIG, only INT64 or DOUBLE value types, and the same metric cannot appear twice in one policy. If the metric originates from each VM, such as agent.googleapis.com/memory/percent_used or TCP connections, the autoscaler averages values across VMs and requires at least one running VM to publish data, so the MIG cannot scale to zero.

Per-VM utilization targets compare the group average to the target, while group-wide custom metrics compare the single global value to the target. A global custom metric such as custom.googleapis.com/example_average_latency with filter metric.labels.group_name="our-instance-group" AND resource.type="global" and utilizationTarget: 250.0 with utilizationTargetType: delta-per-second tells the autoscaler to hold the rate of change near 250 per second. When minNumReplicas is 0 and the metric is group-wide, a value of 0 means no work and the MIG can scale to zero until the metric rises.

2.4Pub/Sub Queue and Tiered Workload Patterns

Pub/Sub-driven autoscaling uses single instance assignment to match VM count to backlog. Data flows from the Pub/Sub subscription to Cloud Monitoring to the autoscaler to the MIG, where the autoscaler divides pubsub.googleapis.com/subscription/num_undelivered_messages filtered by resource.type="pubsub_subscription" AND resource.labels.subscription_id="SUBSCRIPTION_ID" by the configured messages-per-VM value. Use num_undelivered_messages because it counts both queued and in-flight but unacknowledged messages, while a metric that excludes in-flight work can drop to 0 while VMs are still processing and trigger a premature scale-in that interrupts work.

The same pattern supports tiered and infrastructure signals with a utilization target or assignment. Memory scaling uses agent.googleapis.com/memory/percent_used with filter metric.labels.state="used" and type GAUGE, which requires the Ops Agent, otherwise the autoscaler sums buffered, cached, free, slab, and used states. Network scaling uses compute.googleapis.com/instance/network/received_bytes_count, disk scaling uses agent.googleapis.com/disk/operation_count with optional direction or device filters, and backend-tier scaling uses compute.googleapis.com/instance_group/size from a frontend MIG so that, for example, one backend VM is added for every four frontend VMs.

2.5Scale-In Stability and Predictive Autoscaling

Predictive autoscaling (opens in a new tab) scales out ahead of forecasted load for applications with long startup times and predictable daily or weekly cycles. Without prediction the autoscaler reacts only to real-time load, so new VMs may finish initializing after the spike has already caused delays. With prediction enabled, the autoscaler combines real-time CPU data with history-based forecasts to start VMs early so capacity is ready when load arrives.

Predictive autoscaling works only with CPU utilization and requires 3 days of CPU-based autoscaling history before predictions begin. Predictions cover daily and weekly patterns but not monthly, annual, one-time events, or patterns shorter than 10 minutes, where schedule-based scaling is more appropriate. The feature itself is free, but optimizing for availability starts VMs earlier and runs them longer, which increases Compute Engine resource cost compared to purely reactive scaling.

Cloud Run (opens in a new tab) is an inherently elastic serverless container service that scales instances automatically based on incoming requests. Traffic flows from clients to the Cloud Run service, and the autoscaler adds container instances as concurrent demand rises and removes them as demand falls, scaling to zero instances when the service receives no traffic. Elasticity is bounded by configured minimum and maximum instance counts, shaped per-instance by concurrency, and fine-tuned by CPU and concurrency utilization targets.

3.1Request-Driven Elasticity and Instance Limits

Cloud Run request-driven scaling controls capacity between a minimum instance count and a maximum instance count. The autoscaler creates new instances before queued requests cause latency spikes, keeps average load near the configured utilization targets while instances remain below the maximum, and terminates unneeded instances when demand decreases. Maximum instances act as a hard ceiling for cost control and for protecting downstream dependencies such as databases and APIs from overload.

Instance limits can be set at both the service level and the revision level. Cloud Run enforces the lower of scaling.max_instance_count at the service level and template.scaling.max_instance_count at the revision level, so a global service limit always constrains every revision. For minimum instances, the revision-level template.scaling.min_instance_count overrides the service-level scaling.min_instance_count default when traffic is directed to a specific revision, which allows a new revision to be tested with a different warm pool size before promotion.

3.2Concurrency and Instance Sizing

Concurrency in Cloud Run is the maximum number of simultaneous requests a single container instance handles, configured as template.maxInstanceRequestConcurrency. When concurrent requests per instance approach the effective target, the autoscaler scales out by starting additional instances, and when concurrency per instance drops, the autoscaler scales in. A higher concurrency value lets each instance absorb more parallel requests, so fewer instances are needed for the same load, while a lower value forces faster scale-out with more instances.

The Cloud Run default maximum concurrency is 80 requests per instance. Starting at 80 and load-testing downward to find a stable limit is the most cost- and performance-efficient tuning method, because it maximizes utilization of each instance. Reusing a lower known-good value, such as 10 from App Engine standard environment, is safe but can significantly under-utilize Cloud Run instances and increase cost because the autoscaler creates more instances than required. If latency spikes occur under load, lowering maximum concurrency is the direct lever to force earlier scale-out.

3.3Custom CPU and Concurrency Utilization Targets

Custom scaling controls in Cloud Run define the utilization thresholds that trigger scale-up, configured as template.scaling.cpuUtilization and template.scaling.concurrencyUtilization. By default, the Cloud Run autoscaler maintains about 60% CPU utilization on active instances and about 60% of the maximum concurrency value per instance. When observed load exceeds either target, the autoscaler adds instances to bring average utilization back toward the target until the maximum instance count is reached.

The three streamlined scaling controls in Cloud Run are:

• Maximum concurrent requests per instance, which sets per-instance request capacity.
• Minimum and maximum instance counts, which set the warm baseline and hard ceiling.
• Custom CPU and concurrency utilization targets, set with --scaling-cpu-target and --scaling-concurrency-target, which set how densely each instance is packed before scaling.

These targets trade responsiveness against cost. Lower targets keep more headroom per instance and scale out earlier, which protects latency and Service Level Objectives during bursts but runs more instances. Higher targets pack more work onto each instance, which lowers cost but leaves less margin before queueing and CPU saturation.

3.4Minimum Instances, Cold Starts, and CPU Allocation

Minimum instances in Cloud Run keep a specified number of containers warm and running to handle baseline traffic and avoid cold starts. Cloud Run has no separate minimum or maximum idle-instance pool, so scaling.min_instance_count is the mechanism that guarantees warm capacity for immediate traffic. Setting minimum instances to 0 enables full scale-to-zero for maximum savings, while setting it to N pays for always-on capacity to preserve latency during sudden spikes.

CPU allocation and billing determine what a warm or idle instance can do between requests:

Choose request-based billing when work happens strictly inside the request path and idle CPU can be throttled for lowest cost. Choose instance-based billing when the application performs background work between requests, such as background threads, because request-based throttling would suspend that work. This cold-start versus cost tradeoff is central to elasticity design: minimum instances plus instance-based billing minimize startup latency and support background processing, while scale-to-zero plus request-based billing minimize idle spend.

3.5Manual Scaling and Scale-to-Zero Behavior

Manual scaling in Cloud Run disables autoscaling and runs a fixed number of instances, configured with scalingMode: MANUAL and manualInstanceCount: N at the service level. Traffic flows from clients to that fixed pool with no automatic scale-out or scale-in, so capacity does not respond to bursts or idle periods. Choose manual scaling only when deterministic capacity is required and elasticity must be disabled.

Automatic scaling is the default mode and replicates basic scale-to-zero behavior with scaling.min_instance_count: 0 plus a configured scaling.max_instance_count: N. Instances move through an ordered lifecycle where first a cold start provisions a container when demand exceeds warm capacity, then the instance serves concurrent requests up to its concurrency limit, then it remains alive and idle for up to 15 minutes after the last request, and only after that idle window does the autoscaler reclaim it when it is no longer needed. Cloud Run provides no configurable cooldown period for this cycle, so tuning focuses on minimum instances, maximum instances, concurrency, and utilization targets.

Key Facts

• The Horizontal Pod Autoscaler (HPA) changes Pod replica count from real-time metrics, the Vertical Pod Autoscaler (VPA) changes Pod CPU and memory size from historical use, and the Cluster Autoscaler adds nodes when Pods are unschedulable.
• A managed instance group (MIG) autoscaler sets group size to the largest recommendation when multiple scaling signals are configured.
• A MIG using per-VM metrics cannot scale to zero, while a MIG using a group-wide Cloud Monitoring metric can scale to zero when minNumReplicas is 0.
• Cloud Run scales request-driven container instances between minimum and maximum counts, defaults to 80 maximum concurrent requests per instance and 60% CPU and concurrency utilization targets, and scales to zero when minimum instances is 0.