Prepare and test your skills
Prepare and test your skills
Deploying containerized applications on Google Kubernetes Engine (GKE) (opens in a new tab) is the process of describing workloads declaratively, exposing them to traffic, and delivering them through an automated pipeline. The same immutable container image moves from development to production while manifests, external configuration, load balancers, and delivery tools handle placement and routing. This section connects manifests and resource management, service exposure, CI/CD workflows, secure image management, and rollout strategies.
Configuration of Kubernetes Manifests and Resource Management is how an application is described, placed, and configured on GKE without rebuilding its image. A manifest states desired resources such as replica count, update strategy, and compute requests, while ConfigMaps and Secrets inject environment-specific settings at deploy time. The Kubernetes API then administers, scales, and monitors those workloads from source control to the cluster.
A Kubernetes deployment manifest is a declarative file based on the Kubernetes Resource Model (KRM) that states the desired state of an application. The application repository typically holds the kubectl configuration or Helm chart, an Application resource with type, version, and maintainers, and provider scripts for deployment. Tools such as Cloud Build (opens in a new tab) with the gke-deploy builder update the resource file to use the image digest instead of a tag, add recommended labels, retrieve credentials for target clusters, and wait until submitted resources are ready. Namespaces divide cluster resources among teams, an app instance name must be unique within its namespace, and namespaces ending in -system are reserved for GKE components and cannot host applications.
Resource management in GKE centers on how Pods, which are sets of containers, are placed onto nodes, which are Compute Engine virtual machines grouped to form a cluster. The Kubernetes control plane runs the API server and system controllers on Google-operated instances, so operations staff focus on applications while Google manages scheduling and system components. Because containers decouple applications from host infrastructure, the same Pod definition can run across cloud, on-premises, hybrid, and multicloud environments at a higher level of abstraction than virtual machines alone.
Cluster utilization and cost depend on the relationship between what running Pods request and what nodes provide:
| Cluster mode | Who manages nodes and control plane | Billing basis |
|---|---|---|
| GKE Autopilot | Google Cloud manages both nodes and control plane | Pay only for compute resources requested by running Pods |
| GKE Standard | Google Cloud manages control plane, customer manages nodes | Pay for all resources on nodes, regardless of Pod requests |
When using GKE Autopilot, scaling requirements must be checked against its scaling limits. Fault-tolerant workloads such as batch jobs can save cost by running on Spot Pods, which use spare capacity at a lower price.
Externalized configuration is the practice of injecting configuration files and secrets into workloads at deployment time instead of baking environment values into the image. ConfigMaps and Secrets provide that externalized configuration in Kubernetes, so the same immutable artifact can move from development to quality assurance to production with only its injected configuration changing. This approach lowers production risk because the exact image that passed testing is the image that is promoted.
Persistent data for stateful workloads is separate from injected configuration and uses storage services rather than ConfigMaps and Secrets:
Building images from tightly controlled base images, labeling each image with semantic versioning or Git commit hashes, and extending tests to cover both images and workloads keeps the immutable model reliable.
Service exposure on GKE is the practice of publishing containerized workloads so clients can reach them reliably from inside the cluster, from a private network, or from the internet. Three Kubernetes APIs divide this work by scope: Kubernetes Service groups Pod endpoints into one stable endpoint, Ingress routes external HTTP(S) traffic through a managed load balancer, and the Gateway API provides advanced Layer 7 routing through the GKE Gateway controller. All three depend on Cloud Load Balancing and on backend endpoints that GKE keeps updated as Pods start and stop.
A Kubernetes Service is a resource that groups a set of Pod endpoints into a single stable address. A client sends a request to the stable address, and the request is routed to one of the healthy Pods in the Service, so individual Pod restarts do not break connectivity. By default the Service gets a stable cluster IP address usable only inside the cluster, while other Service types widen that reach.
| Service type | Reachable from | How traffic flows |
|---|---|---|
| ClusterIP | Only inside the cluster | Traffic flows from an in-cluster client to the stable cluster IP, then to one Pod in the Service |
| NodePort | Inside the cluster and any client that can reach nodes | Traffic flows from the client to a high port number, for example 32640, opened on every node, then to the target Pod port such as 8080 |
| LoadBalancer | External clients outside the VPC network or internal clients, depending on configuration | GKE creates a Layer 4 TCP/UDP Passthrough Network Load Balancer, and traffic passes through the load balancer to nodes and then to Pods without terminating the connection |
| ExternalName | Only inside the cluster | An internal DNS name such as my-xn-service.default.svc.cluster.local redirects traffic to an external DNS name such as example.com |
A fifth type, Headless, is also part of the Service API. For HTTP(S) web traffic, a Service of type LoadBalancer handles TCP/UDP directly, while HTTP(S) routing uses a Service of type NodePort as the backend for Ingress.
Ingress is a Kubernetes resource that encapsulates rules and configuration for routing external HTTP(S) traffic to internal Services. On GKE, the GKE Ingress controller implements Ingress by automatically creating and managing an external Application Load Balancer, which is the recommended method for HTTP(S) services on GKE. To use Ingress, the external Application Load Balancer add-on must be enabled, and GKE clusters have this add-on enabled by default.
When an Ingress object is created, the controller builds the load balancer from the manifest in a fixed dependency chain. The load balancer's URL map uses host rules and path matchers that reference one or more backend services, where each backend service corresponds to a GKE Service of type NodePort. The backends for each backend service are either instance groups or network endpoint groups (NEGs), and GKE creates a Google Cloud health check for each backend service based on the readiness probe settings of the referenced workload. Load balancers created by GKE are billed according to regular Cloud Load Balancing pricing.
The GKE Gateway controller is Google's implementation of the Kubernetes Gateway API for Cloud Load Balancing, and it configures Layer 7 Application Load Balancers to expose HTTP(S) traffic. Gateway API is designed as a more expressive, flexible, and extensible successor to Ingress, with better support for advanced traffic management, multi-protocol use, and multi-tenancy. GKE Gateway is the recommended choice for new deployments with those needs, while GKE Ingress remains viable for simpler HTTP/HTTPS routing where migration effort outweighs the benefits.
Gateway API splits load balancer configuration across three cooperating resources:
Traffic therefore flows from the internet client to the Gateway listener, then through matching HTTPRoute rules to the target Service, with Policy controls applied to the infrastructure underneath.
Container-native load balancing is the practice of distributing traffic directly to individual Pod IP addresses rather than to nodes, using network endpoint groups (NEGs). With the GCE_VM_IP_PORT NEG type, each Pod endpoint is registered as a backend, which improves network performance, increases visibility, enables advanced load-balancing features, and enables use of Cloud Service Mesh. Because of these benefits, container-native load balancing is the recommended solution for load balancing through Ingress.
When NEGs are used with GKE Ingress, the Ingress controller facilitates creation of all Layer 7 load balancer aspects, including the virtual IP address, forwarding rules, health checks, and firewall rules. This direct-to-Pod path also protects connections during Cluster Autoscaler scale-downs, because for non-NEG load balancers programming and connection draining might not complete before node instances terminate. Container-native load balancing is enabled by default for Services when all of the following conditions are true:
Reliable access on GKE depends on how inbound traffic enters, how it is distributed to Pods, and how outbound traffic leaves the cluster. Service networking has three elements: the frontend defines where the load balancer listens with its network, region, IPs, ports, protocols, and TLS certificates; routing and load balancing define how traffic is processed by protocol, headers, and paths; and backends define the endpoint type and service-discovery integration that GKE updates as endpoints change. An external Application Load Balancer listens on the public internet through Google points of presence and terminates traffic at the edge close to clients, while an internal Application Load Balancer listens within a VPC network for private communication.
Inbound traffic to a workload follows an ordered path:
LoadBalancer, an Ingress, or a Gateway resource.kube-proxy on clusters without GKE Dataplane V2 or by eBPF programs on clusters that use GKE Dataplane V2, and the destination Pod may be on the same node or a different node.Two controls shape this distribution. VPC firewall rules control inbound traffic at the node level and work with Kubernetes Network Policies at the Pod level for defense in depth, while GKE creates default rules so the control plane can reach nodes. Setting the externalTrafficPolicy field to Local avoids an extra hop by sending traffic only to nodes with healthy Pods, with the tradeoff of uneven distribution if Pods are unevenly spread. For an external LoadBalancer Service, weighted load balancing distributes traffic based on serving Pods per node, so nodes with more Pods receive more traffic.
Outbound and cross-network connectivity sit on the other side of the boundary. All GKE clusters require outbound connectivity to *.googleapis.com, *.gcr.io, and *.pkg.dev, plus working connectivity to the control plane IP. In private clusters where Pods lack public IPs, Cloud NAT lets Pods initiate outbound internet access without exposing them to incoming connections, and Private Google Access lets the cluster reach services such as Cloud Storage or Cloud SQL without traversing the public internet. For hybrid traffic, Cloud VPN or Cloud Interconnect connects GKE to on-premises networks, and Multi-cluster Services enable discovery across clusters, regions, or projects.
Exam tip: GKE Autopilot bills on compute resources requested by running Pods, while GKE Standard bills on all node resources even when Pods request less.
Configuring CI/CD workflows for container deployment means connecting source code changes to running workloads on GKE through an automated chain of build, test, store, and deploy steps. In this chain, Cloud Build builds and tests container images, Artifact Registry stores the approved images, and Cloud Deploy (opens in a new tab) or the gke-deploy builder delivers them to GKE clusters. The workflow is triggered automatically when source code changes, so every commit can produce the same tested artifact and the same repeatable deployment.
Cloud Build is the managed build service that turns application source code and a Dockerfile into a tested container image. A Cloud Build trigger monitors the source repository, and when a change is committed to the main branch, the trigger starts the continuous integration (CI) pipeline that rebuilds the container, runs tests, and calls the delivery system to deploy to staging. This automation gives immediate feedback on each change and removes manual image production.
Producing each image follows the same ordered sequence:
Cloud Build also performs vulnerability scanning against a Google Cloud-maintained database and only approves the image to continue after successful scan results. Binary Authorization then signs the approved image to confirm it passed required checks before deployment.
Artifact Registry is the central Google Cloud service for storing container images and dependencies for GKE deployments. Cloud Deploy retrieves container images from Artifact Registry, so the registry sits between the CI pipeline that produces images and the continuous delivery (CD) pipeline that deploys them. To isolate the registry and its maintenance tasks, it is set up in a dedicated Google Cloud project.
Consistent naming makes each pipeline execution repeatable. Release images use semantic versioning tags, while images that still need work use an identifier tied to the codebase point, such as the Git commit hash from the main branch. Workload images are built from a small set of tightly controlled base images to avoid unsupported runtimes, and test suites are extended to cover the images themselves, not only the workloads inside them.
Automated delivery to GKE moves a stored image from Artifact Registry to running Pods on GKE nodes through a delivery pipeline. Cloud Deploy manages this CD work by registering a delivery pipeline and targets, generating a new release for each source change, and deploying the release across a progression of targets. Targets represent GKE clusters, Cloud Run services or jobs, or GKE attached clusters, and Cloud Deploy stores the Skaffold rendering source and rendered manifests in a Cloud Storage bucket.
For direct deployment from Cloud Build, the gke-deploy builder is a wrapper around kubectl that applies recommended practices. The gke-deploy builder updates the Kubernetes resource file to use the image digest instead of a tag, adds recommended Kubernetes labels, retrieves credentials for the target GKE clusters, and waits for submitted resources to become ready. If no extra functionality is needed, the kubectl builder can deploy directly, and Helm or Kustomize builders support templated files. Deployment requires the Kubernetes Engine Developer role for Cloud Build, and the gke-deploy run step must specify the resource file, cluster location, and cluster name.
Repeatable deployment patterns ensure the image tested in CI is the same image running in staging and production. The core rule is to build once and promote the immutable container image through environments, rather than rebuilding at each stage, because rebuilding can introduce minor differences or untested code. Promotion depends on keeping environment-specific configuration separate from the image, so configuration is externalized and read at runtime while secrets and config files are injected during deployment.
Two supporting patterns keep pipelines fast and traceable. GitOps stores declarative application and cluster configuration in Git repositories, so changes are reviewed through merge or pull requests, the full state is accessible at any point, and snapshots simplify recovery. Skaffold provides a common configuration format shared across development, staging, and production, which increases parity and lets Cloud Code with minikube, Cloud Build, and Cloud Deploy reuse the same definitions. Migrating workloads to Google Cloud (opens in a new tab) follows the same principle of promoting tested artifacts through consistent environments.
CI speed is managed with two pipeline types:
| Pipeline type | Typical duration | Choose when |
|---|---|---|
| Rapid pipeline | 10 minutes or less | Developers are iterating quickly on feature branches and need fast feedback, even if artifacts may be unstable |
| Full pipeline | Longer than 10 minutes | Changes are merged by pull request or committed to the main branch and need comprehensive tests and checks |
CD pipelines can run much longer than CI pipelines, especially with strategies such as blue-green deployments, and promotion from staging to production typically requires manual approval by an operator.
Pipeline security in GKE CI/CD means adding checks as early as possible so risks are found before artifacts are built or deployed. Early detection reduces the time and cost of fixing vulnerabilities and prevents risky images from reaching clusters. Security controls apply across the development environment, the CI system, and production governance. Hybrid and multicloud patterns (opens in a new tab) apply the same early-security idea when the same image must run consistently across cloud, on-premises, and multicloud environments.
Common early controls include:
Integration of secure image management and Workload Identity is the practice of storing container images in a trusted registry and giving GKE workloads controlled access to Google Cloud. Artifact Registry stores and protects the images that move through the deployment pipeline, while Workload Identity Federation for GKE lets each application authenticate to Google Cloud APIs with least privilege. Together they keep the same tested image moving from development to production without sharing broad credentials.
Artifact Registry is the managed Google Cloud service for storing container images and other build artifacts for deployment to GKE. Teams set up the registry in a dedicated Google Cloud project to isolate the registry and its maintenance tasks from workload projects. Images flow from the build system into Artifact Registry repositories, and deployment processes then pull those same images into GKE clusters across development, testing, and production environments. Existing images can be copied into Artifact Registry with tools such as gcrane during migration, which supports rollbacks while migration is in progress.
Automated image production is the pipeline that builds, tests, and stores a container image every time source code changes. The build is automatically triggered by a code change so developers get immediate feedback, and Cloud Build can run the full sequence and store the result in Artifact Registry only when tests succeed. Each image follows an ordered workflow:
Base images serve as the shared starting point for workload images, and the set of base images stays small and supported to avoid unsupported runtime environments. Tools such as Container Structure Tests and InSpec extend testing from application code to the structure and compliance of the image itself.
Tagging conventions make every pipeline execution repeatable and traceable from code to image:
| Image type | Use when | Identifier |
|---|---|---|
| Release image | The image is a versioned release ready to promote | Semantic version tag, such as 1.4.0 |
| Pre-release image | The image still needs work before release | Codebase identifier, such as Git commit hash from the main branch |
The same tagged, immutable artifact is promoted through development, quality assurance, and production instead of being rebuilt in each environment. Rebuilding between stages can introduce small differences or untested code, so promotion guarantees that the image tested is the image deployed.
Hardening images before deployment means combining repository controls, vulnerability scanning, and deployment policy so only trusted images reach GKE. Artifact Registry provides controls that protect artifacts at rest and in transit, and Binary Authorization enforces policy at deploy time and after deployment. The controls work together as layered defenses around the registry and the cluster:
Binary Authorization requires attestations, which are signed statements that an image completed a required process. Examples include passed vulnerability scan, passed quality assurance testing, and sign-off from the product owner. Continuous validation extends that check through the Pod lifecycle by logging policy conformance in Cloud Logging, so Pods that violate an updated policy or were deployed with dry run or breakglass are recorded.
Workload Identity Federation for GKE is the mechanism that lets a Kubernetes workload act as a Google Cloud Identity and Access Management (IAM) service account without managing keys. Traffic flows from a Pod using its Kubernetes service account to Google Cloud APIs, and Google Cloud trusts that identity through the federation between the cluster and IAM. The recommended pattern is to use a unique Kubernetes service account for each Kubernetes Service, and that account acts as its linked IAM service account. Because each application gets only the IAM permissions its own identity carries, the blast radius is limited if one workload is compromised.
The same isolation principle applies to the pipeline itself, where each deployment process step uses at least its own service account. One service account can produce the artifact while a different service account manages storage of that artifact in the repository, so the build system depends on separate identities with separate permissions. Using a distinct set of service accounts per environment makes development, testing, and production independent from each other. That separation adds operational complexity, but it lets teams test and validate changes to deployment processes in one environment without affecting the others.
Implementation of deployment strategies and health monitoring on GKE is the practice of updating containerized applications without losing service availability. GKE acts as the container orchestration tool that manages scaling and upgrading logic, so teams compose deployment logic with built-in mechanisms instead of building upgrade automation themselves. Health monitoring works with those mechanisms by observing production behavior and triggering rollback when a new version fails.
Rolling update deployment is a GKE update pattern that replaces application instances in small groups instead of all at once. Traffic continues to flow to the mix of old and new Pods while the controller first updates a subset of running instances, then progressively updates more instances until all instances run the new version. Because only a subset is unavailable or changing at any moment, the service stays available during the transition and a faulty update affects fewer users before it is stopped.
Recreate, blue-green, and canary patterns are alternative deployment patterns on GKE that trade downtime and risk differently from rolling updates. The choice depends on business goals, such as whether the team must deploy changes without any downtime or must expose changes to a subset of users before general availability.
GKE has built-in support for the recreate and rolling strategies, which can be triggered remotely with kubectl from a continuous integration (CI) system or a separate deployment step. More elaborate patterns such as blue-green and canary depend on additional tooling or scripting, such as Spinnaker, which has built-in support for GKE, or Cloud Deploy, which implements deployment processes for containerized workloads.
Maintaining service availability during transitions on GKE means combining a robust deployment strategy with production monitoring and automated rollback. No amount of pre-production testing guarantees correct behavior, because edge cases in data and user access patterns appear only in production, so teams must monitor the application in production and keep automated rollback and deployment mechanisms ready. Google Cloud Observability, with integrated logging and monitoring, provides the signals for that monitoring, and using a more robust strategy limits the blast radius so fewer end users are affected when an issue arises.
Continue with this study guide: Deploying containerized applications
Implementing Kubernetes health checks to increase application availability means configuring probes so GKE restarts truly stuck containers and stops sending traffic to Pods that are not ready. In GKE, the readiness probe controls whether a Pod receives traffic, the liveness probe controls whether a container is restarted, and the startup probe blocks the other two until initialization finishes. The load balancer health check used by Ingress runs outside the cluster, but it is derived from the readiness probe unless a BackendConfig custom resource supplies explicit settings.
Optimizing probe configuration parameters for stability means setting Kubernetes timing and thresholds so real failures are found quickly without restarting or isolating healthy containers. In GKE, the readiness probe controls whether a Pod receives traffic, the liveness probe controls whether a container is restarted, and the startup probe blocks the other two until initialization finishes. The load balancer health check used by Ingress (opens in a new tab) runs outside the cluster, but it is derived from the readiness probe unless a BackendConfig custom resource supplies explicit settings.
Probe roles in GKE divide availability work by the question each probe answers and the action Kubernetes takes on repeated failure. A startup probe asks whether the container has finished starting, a liveness probe asks whether a running container is stuck in an unrecoverable state such as a deadlock, and a readiness probe asks whether the instance should receive traffic right now. Data flows through a fixed gate: first the startup probe must succeed, then liveness and readiness checking begins.
| Probe | What it checks | Action after repeated failure |
|---|---|---|
| Startup probe | Container has finished starting and can accept traffic | Liveness probe and readiness probe stay disabled until startup succeeds |
| Liveness probe | Running container needs a restart to recover | Container is restarted to restore availability |
| Readiness probe | Instance should serve traffic now | Pod is removed from load-balancer endpoints without termination, and returns when the probe passes again |
Because passing startup is treated as ready to receive traffic, startup logic should also imply readiness. A new Pod should not receive requests before caches, connections, and frameworks inside the container are ready to serve them safely.
Balancing detection speed against churn means choosing timing values that react fast to deadlocks while tolerating slow startup and brief load spikes. The initialDelaySeconds setting delays the first probe until the application can realistically respond, the periodSeconds setting controls how often the kubelet re-checks the container, and the failureThreshold setting controls how many consecutive failures are required before Kubernetes restarts the container or removes the Pod from service. Short delays, frequent periods, and low thresholds detect failures faster, but they can turn a slow image pull, class load, cache warm-up, or traffic spike into repeated restarts and endpoint flapping.
Slow startup is a common cause of churn when liveness checking starts too early. Applications that load large caches or boot large frameworks can take minutes before an HTTP endpoint returns 200 OK. A startup probe protects this phase better than only lengthening the liveness delay, because liveness and readiness remain disabled until startup reports success. For readiness, the probe must report ready only after dependencies inside the container are ready, so scale-up and rolling updates do not send users to Pods that are still booting.
Stable probe endpoint design keeps the check local, simple, and independent of other systems. A good default is a lightweight HTTP endpoint on the container that returns 200 OK when the container itself can serve, rather than a deep check of connection pools or downstream services. Probe logic must never call other services, because a slow or failed dependency would then drive restarts and traffic removal for a Pod whose own process is healthy.
Configuration consistency across Pod, Service, and Ingress also controls stability. The probe must target the correct path and port where the application actually listens, and the Pod containerPort must match the Service targetPort and the Ingress backend port or probes fail with errors such as connection refused. Pod probe fields cannot be edited in place, and changing a readiness probe for Pods already behind an Ingress does not automatically update the external load balancer health check. To apply a new probe definition, redeploy the Pods and the Ingress resource so GKE recreates the load balancer and its health checks with the new path, port, and protocol.
Exam tip: Readiness failure removes a Pod from traffic without restarting it, while liveness failure restarts the container; use readiness for temporary not-ready states and liveness only for unrecoverable states.
Diverse probe handlers are the different mechanisms the kubelet uses to check whether a container is healthy and ready to serve traffic. The kubelet sends a probe request from the node to the container, the container responds through its application protocol or process state, and the kubelet then marks the container as healthy or unhealthy based on that response. Choosing the handler that matches how the application actually communicates prevents false restarts and supports consistent observability across web, network, microservice, and batch patterns.
HTTP GET probes check health by sending an HTTP GET request from the kubelet to a path and port inside the container. The request flows from the kubelet to the application web server, such as GET /healthz on port 8080, and success requires an HTTP status of 200 through 399. This handler verifies more than a listening port because the application must route the request, run health logic, and return a healthy code. Choose HTTP GET probes when the container serves HTTP or HTTPS traffic and can expose a dedicated health endpoint.
TCP socket probes check health by attempting to open a TCP connection from the kubelet to a container port. The connection attempt flows from the kubelet to the container network namespace, and success means the TCP handshake completes and the port accepts the connection. This handler sends no application data, so it cannot detect an application deadlock that still leaves the port open. Choose TCP socket probes when the workload uses non-HTTP protocols such as databases, message brokers, or raw TCP servers where an HTTP endpoint does not exist but a listening port proves the process is accepting work.
gRPC probes check health by invoking the standard gRPC health-checking service from the kubelet to the container. The check flows from the kubelet as a gRPC client to the application gRPC server on its configured port, and success requires a SERVING status for the requested service. This handler understands gRPC semantics over Protocol Buffers and HTTP/2, so it tests the real request path for microservices. Choose gRPC probes when the architectural pattern is service-to-service gRPC, where an HTTP probe would miss the real path and a TCP probe would only confirm the port is open.
Exec command probes check health by running a command inside the container filesystem and namespaces. The execution flows from the kubelet to the container runtime, which starts the specified process such as cat /tmp/healthy or a custom script, and success requires an exit code of 0. This handler depends on tools present in the container image, so the probe fails if the binary or file is missing even when the application is healthy. Choose Exec probes when health cannot be observed over the network, such as batch workers, file-based initialization, or applications with custom internal state that only a local command can validate.
Probe handlers differ by what the kubelet sends, what counts as success, and which architecture they fit. Configuring health checks (opens in a new tab) means matching the handler to the workload protocol so the check tests the real serving path.
| Handler | What the kubelet sends | Success condition | Choose when |
|---|---|---|---|
| HTTP GET probe | HTTP GET to path and port | HTTP status 200-399 | The app serves HTTP and has a health endpoint |
| TCP socket probe | TCP connection to port | Connection is accepted | The app uses non-HTTP TCP and has no HTTP endpoint |
| gRPC probe | gRPC health check call | Status is SERVING | The app communicates over gRPC services |
| Exec command probe | Command inside container | Exit code is 0 | Health is only visible to a local file or process |
All handlers share common tuning that controls how quickly an endpoint changes state. CHECK_INTERVAL is the time in seconds from the start of one probe to the start of the next, with a default of 5. HEALTHY_THRESHOLD is the number of sequential passing probes required to consider the endpoint healthy, with a default of 2. UNHEALTHY_THRESHOLD is the number of sequential failing probes required to consider the endpoint unhealthy, with a default of 2. Together, the handler defines how health is tested and the thresholds define how many consistent results are needed before routing changes or a restart occurs.
Kubernetes health checking in GKE uses three probes that divide container lifecycle work by the question asked. A startup probe asks whether the container has finished starting, a liveness probe asks whether the container is still healthy enough to keep running, and a readiness probe asks whether the container is ready to receive traffic. Together the probes provide self-healing through restarts and continuous availability through traffic control, with startup gating the other two during initialization.
A startup probe determines whether the container application has started and is ready to accept traffic after long initialization. While the startup probe is running, the kubelet disables liveness and readiness checks so slow initialization does not trigger a premature restart or traffic failure. If the startup probe does not succeed within its allowed window, defined by failure threshold multiplied by probe period, the container is shut down and restarted instead of being left in an unknown starting state.
A liveness probe determines whether to restart a container that is already running. The kubelet sends the configured check, such as an HTTP GET to a health endpoint or a gRPC health request, and repeated failure beyond the failure threshold causes the container to be killed and restarted. Liveness probes are intended for unrecoverable failures where no other recovery is possible, such as a deadlock where the process is running but cannot make progress. A database replica stuck in a deadlock is a typical example, because restarting the replica is the only way to restore progress without affecting healthy replicas.
A readiness probe determines when a Pod should serve traffic through a Kubernetes Service. The readiness check begins only after the startup probe has succeeded, and the Pod is added to Service endpoints while the probe passes and removed from endpoints while the probe fails. A readiness failure does not terminate or restart the container, and traffic flows back to the Pod automatically when the probe starts passing again. This behavior supports temporary unavailability, such as a database replica that must pause request handling during a backup and then resume serving without a restart.
The three probes differ by the lifecycle question they answer and by whether failure causes a restart or a traffic change. Assessing cluster and workload health (opens in a new tab) in the console shows these boundaries as restart events versus endpoint changes.
| Probe | Checks | Action on repeated failure |
|---|---|---|
| Startup probe | Has container finished initialization? | Container is shut down and restarted; liveness and readiness stay disabled until success |
| Liveness probe | Is running container still making progress? | Container is killed and restarted to recover from deadlock or corruption |
| Readiness probe | Is container ready to receive traffic now? | Pod is removed from Service endpoints with no restart; Pod returns to endpoints on recovery |
The table shows a one-way dependency from startup to the other probes. Startup success unblocks traffic routing and steady-state monitoring, liveness drives the restart path, and readiness drives the traffic path without changing container state.
The three probes run in a strict order that protects starting containers and then separates restarts from traffic routing. First, the startup probe controls the starting phase and liveness and readiness checks remain disabled until the startup probe succeeds. Then, once startup succeeds, the container becomes ready and the Service begins sending traffic from the Service to the Pod endpoints. Finally, during steady-state operation, the liveness probe monitors container health for restarts while the readiness probe continuously adds or removes the Pod from endpoints as its ability to serve traffic changes.
A readiness probe in GKE also shapes traffic flow through external load balancing, not only inside the cluster. When a Service is exposed through a GKE Ingress, the Ingress controller creates a Google Cloud health check for each backend service that corresponds to a Kubernetes Service, and it derives path, port, and protocol from the serving Pods' readiness probe. This load balancer health check is different from the Kubernetes probe because the Kubernetes probe is executed by the kubelet inside the cluster while the load balancer check is executed from outside the cluster against the backend. A BackendConfig custom resource provides precise control over that external check and overrides the derived settings when tighter tuning is needed. If readiness settings change after Ingress creation, GKE does not automatically update the load balancer check, so Pods and the Ingress resource must be redeployed to recreate the load balancer and align the external check with the new readiness definition.
Incorporating Horizontal Pod Autoscaler (HPA) (opens in a new tab) attributes means configuring how GKE automatically adds or removes Pods to keep performance stable and resource use efficient. Each HPA controls one scalable workload, such as a Deployment or StatefulSet, and it runs a repeating control loop that checks live Pod metrics against a target. The replica count always stays between minReplicas and maxReplicas, with stabilization rules that prevent rapid oscillation.
Resource-based scaling is the process where the HPA automatically changes the number of Pods in a workload to keep CPU or memory use near a target. Each HPA controls one scalable workload, such as a Deployment or StatefulSet, and it runs a repeating control loop that checks live Pod metrics against the target. When average use rises above the target, the HPA adds replicas, and when use falls below the target, it removes replicas. The replica count always stays between minReplicas and maxReplicas.
Pod specification requirements for HPA are the CPU and memory requests that must be set on every container when using percentage-based targets. The HPA controller queries the resource metrics API for the usage of each container in each Pod, then calculates average utilization as usage divided by requests. Because requests are the denominator in that math, the dependency flows from the HPA target to the container requests to the metrics API reading. If any container is missing a request for CPU or memory, a percentage target for that resource cannot be calculated and scaling for that metric fails.
Resource requests set the baseline that HPA uses for its utilization math, while resource limits set the maximum a container may use. HPA percentage calculations use requests only, not limits, so precise requests are what make scaling stable. Setting requests that match the real needs of the application keeps the percentage meaningful and lets the cluster schedule Pods with enough capacity. Requests that are too low cause constant over-scaling, while requests that are too high cause under-scaling.
CPU and memory utilization targets are the thresholds in the HPA manifest that define when the workload should scale. A percentage target is configured as type: Resource with resource.name: cpu or memory and target.type: Utilization with averageUtilization: 50, which tells the HPA to maintain 50% average use across Pods. A raw-value target uses a fixed amount instead of a percentage, and the HPA compares observed usage directly to that value without dividing by requests. The manifest also sets scaleTargetRef to identify the Deployment to scale and minReplicas and maxReplicas to bound the replica count.
| Target form | What the HPA compares | Pod specification dependency |
|---|---|---|
Percentage, such as averageUtilization: 50 | Average Pod usage divided by Pod requests for CPU or memory | Requires CPU or memory requests on every container in the Pod |
| Raw value, such as a fixed CPU or memory amount | Observed usage against the fixed value directly | Does not use requests as a denominator |
The same target can be created with kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=10, which holds average CPU at 50% of request between 1 and 10 replicas. The autoscaling/v1 API supports only CPU utilization, while the autoscaling/v2 API is recommended because it supports CPU, memory, and custom or external metrics in one object. Choosing a lower percentage target adds capacity sooner and favors performance during spikes, while a higher percentage favors efficiency but leaves less headroom while new Pods start.
Exam tip: A CPU or memory target expressed as a percentage requires requests on every container in the Pod; without requests the HPA status shows <unknown> and cannot calculate a replica count.
Scaling behaviors and stabilization policies control how fast the HPA adds or removes Pods when metrics change in GKE. The HPA operates as a control loop that periodically compares observed metrics against targets and adjusts replicas between minReplicas and maxReplicas. Stabilization policies smooth those decisions over time so brief spikes or dips do not cause rapid Pod oscillation. This smoothing protects cluster stability during fluctuating traffic.
The HPA control loop is the repeating process where one HPA object monitors one scalable workload, such as a Deployment or StatefulSet. The controller queries metric sources for each Pod, compares the current value to the configured target, and calculates a desired replica count that would bring use back to target. For per-Pod resources such as CPU, a percentage target is calculated as average utilization as a percentage of each container's requested CPU or memory, so requests must be set on every container. If the calculation calls for more or fewer Pods than allowed, the result is capped at maxReplicas or minReplicas and the workload stays at that bound.
Scale-up and scale-down velocity is the separate speed limit applied to growing versus shrinking a workload. Scale-up is typically allowed to react quickly because adding Pods protects latency and prevents overload while new Pods start. Scale-down is typically slowed because removing Pods too quickly can cause thrashing, where the HPA scales down, metrics immediately spike, and the HPA must scale back up.
| Direction | Goal | Behavior during fluctuating traffic |
|---|---|---|
| Scale-up | Meet demand quickly | Adds replicas promptly when current metric exceeds target, up to maxReplicas |
| Scale-down | Protect stability and avoid flapping | Waits for metrics to stay low over a stabilization period before removing replicas, down to minReplicas |
The Performance HPA profile changes reaction time at the controller level for large environments. The profile supports up to 1,000 HPA objects in GKE versions 1.31-1.32 and up to 5,000 objects in version 1.33 or later. It is automatically enabled on qualifying Autopilot clusters from version 1.32 and qualifying Standard clusters from version 1.33.
A stabilization window in HPA is a look-back period that prevents a single noisy metric reading from reversing a recent scaling decision. The window holds recent desired replica counts in memory and keeps the more conservative count until the signal persists. For scale-down, this means the HPA keeps extra Pods running while traffic bounces up and down, and only reduces replicas after average demand has clearly fallen. For scale-up, the window is kept short so the workload can still grow without delay when load rises.
This protection matters for spiky signals such as load-balancer traffic. In traffic-based autoscaling, traffic flows from an internet Gateway to a Service and then to the Pods of the target Deployment, and the HPA compares Gateway traffic signals to per-Pod capacity defined by maxRatePerEndpoint on a GCPBackendPolicy. With maxRatePerEndpoint: 10 and an HPA target of averageValue: 70 for autoscaling.googleapis.com|gclb-capacity-fullness, the HPA aims for about 7 requests per second per Pod and leaves a 30% growth margin while new Pods start. Without stabilization, each brief dip below 7 requests per second would remove Pods that the next spike immediately requires.
Bounds and metric signals in HPA define the limits and inputs that scaling behaviors act upon. The HPA supports three metric categories: actual resource usage such as CPU or memory, custom metrics such as requests per second, and external metrics from outside the cluster such as Pub/Sub queue size. When multiple metrics are configured in one autoscaling/v2 HPA object, the controller evaluates each metric separately and uses the largest recommended replica count, so the most stressed signal drives the decision. One HPA object must be dedicated to one Deployment and one Service, and a Deployment or Service cannot be referenced by more than one HPA.
Special bounds apply when scaling to and from zero. Scaling to zero with HPA requires GKE version 1.37 or later, at least one custom or external metric exposed through an AutoscalingMetric resource, and minReplicas: 0 in an autoscaling/v2 manifest. Resource metrics such as CPU cannot drive scale-from-zero because no Pods exist to report utilization, so a custom or external signal must activate the Pods. Checking status with kubectl describe hpa shows Metrics as current versus target values, Conditions for controller health, and Events for recent actions, where <unknown> means the metric has not been fetched and ScalingLimited means the HPA wants to scale but is capped by minReplicas or maxReplicas. Troubleshooting horizontal Pod autoscaling (opens in a new tab) uses those fields to distinguish metric failures from deliberate min/max caps.
Integration of custom and external metrics is the practice of scaling an HPA on application-specific signals instead of only CPU or memory in GKE. These signals include request latency, throughput, queue depth, and inference-server metrics such as backlog size or accelerator memory use. The signals flow from Pods to Google Cloud Managed Service for Prometheus, through a metrics adapter, to the HPA controller, which then adjusts the replica count of the target Deployment.
The metric pipeline in GKE connects workload telemetry to scaling decisions in a fixed order. First, a PodMonitoring resource tells Managed Service for Prometheus what to scrape, which port to use, and how often to collect it. Then, Managed Service for Prometheus stores those time series in Cloud Monitoring, where a metrics adapter can query them. Finally, the adapter exposes the values through the Kubernetes custom-metrics or external-metrics APIs so the HPA controller can read them and scale the target workload.
Setting up autoscaling with metrics follows an ordered workflow:
roles/monitoring.viewer role to the adapter service account so it can read Monitoring data.autoscaling/v2 that sets minReplicas, maxReplicas, and a target average value for each metric. The HPA controller then compares the current metric value to the target and adds or removes Pod replicas.A metrics adapter for Prometheus metrics is the component that translates Managed Service for Prometheus data into metrics the HPA can consume. GKE supports three approaches for this translation, and only one adapter solution should be chosen per cluster. The Custom Metrics Stackdriver Adapter and the Prometheus Adapter cannot be used together in the same cluster because their resource definitions overlap.
| Adapter | How it serves metrics to HPA | Key detail from documentation |
|---|---|---|
| KEDA (Kubernetes Event-driven Autoscaling) | Scales on Prometheus metrics through event-driven scalers | Most recently released option and a favored solution in the Prometheus community, with direct integration for Managed Service for Prometheus |
| Custom Metrics Stackdriver Adapter | Serves average values of individual metric queries from Managed Service for Prometheus | Developed and supported by Google Cloud; supports querying Managed Service for Prometheus starting with version v0.13.1 |
| Prometheus Adapter | Serves values computed by PromQL queries against Managed Service for Prometheus | Third-party library where custom queries and metric names are defined in a Helm values file |
Horizontal pod autoscaling with Managed Service for Prometheus (opens in a new tab) describes how PodMonitoring data flows through the chosen adapter to the HPA controller.
HPA configuration for custom and external metrics defines what to scale, what signal to watch, and what value to maintain. Every HPA manifest names a scaleTargetRef such as a Deployment, sets MIN_REPLICAS as the lowest allowed replica count, sets MAX_REPLICAS as the highest allowed replica count, and sets TARGET as the desired average value across Pods. The HPA controller calculates the desired replica count from the ratio of the current value to the target value, so choosing an accurate target is critical for performance and resource efficiency.
The type field in spec.metrics controls how the metric is interpreted. A Pods metric represents the average value of a metric across the scaled Pods, such as prometheus.googleapis.com|jetstream_prefill_backlog_size|gauge for inference queue backlog or jetstream_slots_used_percentage for decode-slot use. An External metric represents a value that is not tied to a single Pod object, such as memory_used_percentage computed from accelerator memory or a queue-depth signal from an external system. Both types commonly use target.type: AverageValue with an averageValue, which tells HPA to maintain that average across all replicas.
GKE also provides built-in autoscaling based on Managed Service for Prometheus through an AutoscalingMetric resource. The AutoscalingMetric, Deployment, and HPA objects must be in the same namespace, and the HPA metric name uses the form autoscaling.gke.io|NAME|METRIC_NAME, where NAME is the AutoscalingMetric name. The HPA metric type must match the type defined in the AutoscalingMetric, with Pods for per-Pod signals and External for global or externally produced signals. Optimizing Pod autoscaling based on metrics (opens in a new tab) shows how those AutoscalingMetric objects connect Prometheus signals to HPA targets.
Scaling on multiple metrics is the ability of a single HPA object to consider two or more signals at the same time. The multiple-metric HPA collects all entries from spec.metrics into one manifest, such as one Pods entry for jetstream_METRIC and one External entry for memory_used_percentage. Each entry keeps its own metric name and AverageValue target, and the HPA controller evaluates each metric independently. This combination lets a workload scale for both application throughput, such as prefill backlog, and hardware pressure, such as accelerator memory use, without creating competing HPA objects for the same Deployment. Configuring autoscaling for LLM workloads (opens in a new tab) uses this pattern to balance inference queue depth against accelerator memory pressure.
Google Kubernetes Engine Autopilot bills only for compute resources requested by running Pods, while Standard bills for all resources on nodes regardless of Pod requests. In Autopilot Google Cloud manages both nodes and the control plane, while in Standard Google Cloud manages the control plane and the customer manages nodes.
A Kubernetes Service groups Pod endpoints into one stable address so clients reach healthy Pods despite restarts, Ingress routes external HTTP(S) traffic through a managed Application Load Balancer, and the Gateway API provides more expressive Layer 7 routing with traffic splitting and header matching. GKE Gateway is the recommended choice for new deployments needing advanced traffic management and multi-tenancy, while GKE Ingress remains viable for simpler HTTP and HTTPS routing.
Cloud Build builds the container image from source and runs tests including vulnerability scanning, Artifact Registry stores the approved image with semantic version or commit-hash tags, and Cloud Deploy or the gke-deploy builder delivers the exact image digest to GKE clusters. The same immutable image is promoted through environments with only injected ConfigMaps and Secrets changing.
A startup probe asks whether the container has finished starting and keeps liveness and readiness disabled until it succeeds, a liveness probe asks whether a running container is stuck and restarts the container after repeated failure, and a readiness probe asks whether a Pod should receive traffic and removes it from endpoints without a restart. Use readiness for temporary not-ready states and liveness only for unrecoverable states such as deadlock.
Last updated