Horizontal Pod Autoscaler Scaling and Metrics in GKE

Resource-based scaling is the process where the Horizontal Pod Autoscaler (HPA) (opens in a new tab) automatically changes the number of Pods in a workload to keep CPU or memory use near a target. Each HPA controls one scalable workload, such as a Deployment or StatefulSet, and it runs a repeating control loop that checks live Pod metrics against the target. When average use rises above the target, the HPA adds replicas, and when use falls below the target, it removes replicas. The replica count always stays between minReplicas and maxReplicas.

1.1Pod Specification Requirements for HPA

Pod specification requirements for HPA are the CPU and memory requests that must be set on every container when using percentage-based targets. The HPA controller queries the resource metrics API for the usage of each container in each Pod, then calculates average utilization as usage divided by requests. Because requests are the denominator in that math, the dependency flows from the HPA target to the container requests to the metrics API reading. If any container is missing a request for CPU or memory, a percentage target for that resource cannot be calculated and scaling for that metric fails.

Resource requests set the baseline that HPA uses for its utilization math, while resource limits set the maximum a container may use. HPA percentage calculations use requests only, not limits, so precise requests are what make scaling stable. Setting requests that match the real needs of the application keeps the percentage meaningful and lets the cluster schedule Pods with enough capacity. Requests that are too low cause constant over-scaling, while requests that are too high cause under-scaling.

1.2Configuring CPU and Memory Utilization Targets

CPU and memory utilization targets are the thresholds in the HPA manifest that define when the workload should scale. A percentage target is configured as type: Resource with resource.name: cpu or memory and target.type: Utilization with averageUtilization: 50, which tells the HPA to maintain 50% average use across Pods. A raw-value target uses a fixed amount instead of a percentage, and the HPA compares observed usage directly to that value without dividing by requests. The manifest also sets scaleTargetRef to identify the Deployment to scale and minReplicas and maxReplicas to bound the replica count.

The same target can be created with kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=10, which holds average CPU at 50% of request between 1 and 10 replicas. The autoscaling/v1 API supports only CPU utilization, while the autoscaling/v2 API is recommended because it supports CPU, memory, and custom or external metrics in one object. Choosing a lower percentage target adds capacity sooner and favors performance during spikes, while a higher percentage favors efficiency but leaves less headroom while new Pods start.

Scaling behaviors and stabilization policies control how fast the Horizontal Pod Autoscaler (HPA) (opens in a new tab) adds or removes Pods when metrics change in Google Kubernetes Engine (GKE). The HPA operates as a control loop that periodically compares observed metrics against targets and adjusts replicas between minReplicas and maxReplicas. Stabilization policies smooth those decisions over time so brief spikes or dips do not cause rapid Pod oscillation. This smoothing protects cluster stability during fluctuating traffic.

2.1How the HPA Control Loop Calculates Desired Replicas

The HPA control loop is the repeating process where one HPA object monitors one scalable workload, such as a Deployment or StatefulSet. The controller queries metric sources for each Pod, compares the current value to the configured target, and calculates a desired replica count that would bring use back to target. For per-Pod resources such as CPU, a percentage target is calculated as average utilization as a percentage of each container's requested CPU or memory, so requests must be set on every container. If the calculation calls for more or fewer Pods than allowed, the result is capped at maxReplicas or minReplicas and the workload stays at that bound.

2.2Controlling Scale-Up and Scale-Down Velocity

Scale-up and scale-down velocity is the separate speed limit applied to growing versus shrinking a workload. Scale-up is typically allowed to react quickly because adding Pods protects latency and prevents overload while new Pods start. Scale-down is typically slowed because removing Pods too quickly can cause thrashing, where the HPA scales down, metrics immediately spike, and the HPA must scale back up.

The Performance HPA profile changes reaction time at the controller level for large environments. The profile supports up to 1,000 HPA objects in GKE versions 1.31-1.32 and up to 5,000 objects in version 1.33 or later. It is automatically enabled on qualifying Autopilot clusters from version 1.32 and qualifying Standard clusters from version 1.33.

2.3Stabilization Windows That Prevent Pod Oscillation

A stabilization window in HPA is a look-back period that prevents a single noisy metric reading from reversing a recent scaling decision. The window holds recent desired replica counts in memory and keeps the more conservative count until the signal persists. For scale-down, this means the HPA keeps extra Pods running while traffic bounces up and down, and only reduces replicas after average demand has clearly fallen. For scale-up, the window is kept short so the workload can still grow without delay when load rises.

This protection matters for spiky signals such as load-balancer traffic. In traffic-based autoscaling, traffic flows from an internet Gateway to a Service and then to the Pods of the target Deployment, and the HPA compares Gateway traffic signals to per-Pod capacity defined by maxRatePerEndpoint on a GCPBackendPolicy. With maxRatePerEndpoint: 10 and an HPA target of averageValue: 70 for autoscaling.googleapis.com|gclb-capacity-fullness, the HPA aims for about 7 requests per second per Pod and leaves a 30% growth margin while new Pods start. Without stabilization, each brief dip below 7 requests per second would remove Pods that the next spike immediately requires.

2.4Bounds, Metric Signals, and Checking Paused Scaling

Bounds and metric signals in HPA define the limits and inputs that scaling behaviors act upon. The HPA supports three metric categories: actual resource usage such as CPU or memory, custom metrics such as requests per second, and external metrics from outside the cluster such as Pub/Sub queue size. When multiple metrics are configured in one autoscaling/v2 HPA object, the controller evaluates each metric separately and uses the largest recommended replica count, so the most stressed signal drives the decision. One HPA object must be dedicated to one Deployment and one Service, and a Deployment or Service cannot be referenced by more than one HPA.

Special bounds apply when scaling to and from zero. Scaling to zero with HPA requires GKE version 1.37 or later, at least one custom or external metric exposed through an AutoscalingMetric resource, and minReplicas: 0 in an autoscaling/v2 manifest. Resource metrics such as CPU cannot drive scale-from-zero because no Pods exist to report utilization, so a custom or external signal must activate the Pods. Checking status with kubectl describe hpa shows Metrics as current versus target values, Conditions for controller health, and Events for recent actions, where <unknown> means the metric has not been fetched and ScalingLimited means the HPA wants to scale but is capped by minReplicas or maxReplicas.

Integration of custom and external metrics is the practice of scaling a Horizontal Pod Autoscaler (HPA) (opens in a new tab) on application-specific signals instead of only CPU or memory in Google Kubernetes Engine (GKE). These signals include request latency, throughput, queue depth, and inference-server metrics such as backlog size or accelerator memory use. The signals flow from Pods to Google Cloud Managed Service for Prometheus, through a metrics adapter, to the HPA controller, which then adjusts the replica count of the target Deployment.

3.1How the Metric Pipeline Works

The metric pipeline in GKE connects workload telemetry to scaling decisions in a fixed order. First, a PodMonitoring resource tells Managed Service for Prometheus what to scrape, which port to use, and how often to collect it. Then, Managed Service for Prometheus stores those time series in Cloud Monitoring, where a metrics adapter can query them. Finally, the adapter exposes the values through the Kubernetes custom-metrics or external-metrics APIs so the HPA controller can read them and scale the target workload.

Setting up autoscaling with metrics follows an ordered workflow:

1. Export metrics to Cloud Monitoring: Create a PodMonitoring object that selects Pods by label and scrapes an endpoint every 15 seconds. Server metrics support scrape intervals as frequent as 5 seconds, while system metrics support intervals as frequent as 15 seconds.
2. Install a metrics adapter: Deploy an adapter that makes the exported Prometheus metrics visible to the HPA controller. If Workload Identity Federation for GKE is enabled, grant the roles/monitoring.viewer role to the adapter service account so it can read Monitoring data.
3. Create the HPA resource: Deploy a HorizontalPodAutoscaler object using autoscaling/v2 that sets minReplicas, maxReplicas, and a target average value for each metric. The HPA controller then compares the current metric value to the target and adds or removes Pod replicas.

3.2Choosing a Metrics Adapter for Prometheus Metrics

A metrics adapter for Prometheus metrics is the component that translates Managed Service for Prometheus data into metrics the HPA can consume. GKE supports three approaches for this translation, and only one adapter solution should be chosen per cluster. The Custom Metrics Stackdriver Adapter and the Prometheus Adapter cannot be used together in the same cluster because their resource definitions overlap.

3.3Configuring HPA with Pods and External Metric Types

HPA configuration for custom and external metrics defines what to scale, what signal to watch, and what value to maintain. Every HPA manifest names a scaleTargetRef such as a Deployment, sets MIN_REPLICAS as the lowest allowed replica count, sets MAX_REPLICAS as the highest allowed replica count, and sets TARGET as the desired average value across Pods. The HPA controller calculates the desired replica count from the ratio of the current value to the target value, so choosing an accurate target is critical for performance and resource efficiency.

The type field in spec.metrics controls how the metric is interpreted. A Pods metric represents the average value of a metric across the scaled Pods, such as prometheus.googleapis.com|jetstream_prefill_backlog_size|gauge for inference queue backlog or jetstream_slots_used_percentage for decode-slot use. An External metric represents a value that is not tied to a single Pod object, such as memory_used_percentage computed from accelerator memory or a queue-depth signal from an external system. Both types commonly use target.type: AverageValue with an averageValue, which tells HPA to maintain that average across all replicas.

GKE also provides built-in autoscaling based on Managed Service for Prometheus through an AutoscalingMetric resource. The AutoscalingMetric, Deployment, and HPA objects must be in the same namespace, and the HPA metric name uses the form autoscaling.gke.io|NAME|METRIC_NAME, where NAME is the AutoscalingMetric name. The HPA metric type must match the type defined in the AutoscalingMetric, with Pods for per-Pod signals and External for global or externally produced signals.

3.4Scaling on Multiple Metrics

Scaling on multiple metrics is the ability of a single HPA object to consider two or more signals at the same time. The multiple-metric HPA collects all entries from spec.metrics into one manifest, such as one Pods entry for jetstream_METRIC and one External entry for memory_used_percentage. Each entry keeps its own metric name and AverageValue target, and the HPA controller evaluates each metric independently. This combination lets a workload scale for both application throughput, such as prefill backlog, and hardware pressure, such as accelerator memory use, without creating competing HPA objects for the same Deployment.

Key Facts

• The Horizontal Pod Autoscaler (HPA) calculates CPU or memory percentage targets as average usage divided by container requests, so every container must define requests for percentage-based scaling to work.
• The autoscaling/v2 API supports CPU, memory, and custom or external metrics in one HPA object, while autoscaling/v1 supports only CPU utilization.
• Scale-up in HPA reacts quickly to meet demand, while scale-down uses a stabilization window that keeps extra Pods until low demand persists to prevent thrashing.
• In GKE, custom metric signals flow from Pods to Google Cloud Managed Service for Prometheus through PodMonitoring, through a metrics adapter, to the HPA controller.
• The Custom Metrics Stackdriver Adapter and the Prometheus Adapter cannot be installed together in the same cluster because their resource definitions overlap.