Prepare and test your skills

Prepare and test your skills

Node pool lifecycle operations are the creation, modification, and deletion of groups of nodes that supply compute capacity for a cluster. A Google Kubernetes Engine (GKE) (opens in a new tab) cluster consists of nodes, which are Compute Engine virtual machines (VMs), grouped together to form the cluster. You package apps into containers, deploy sets of containers as Pods to nodes, and Google Cloud manages the control plane for you. In GKE Standard mode, you pay for all resources on nodes regardless of Pod requests, so choosing the right node pool size and type directly affects cost and reliability.
Creating a node pool means defining a set of nodes with the same machine type, disk configuration, and VM pricing model to match a workload. Before creating nodes that use Spot VMs, check real-time Spot VM availability and view the preemption rate and pricing to confirm that the machine type fits the workload and cost needs. Node auto-provisioning can also create node pools automatically when scheduled Pods require Spot VMs through a nodeSelector or node affinity. GKE automatically adds the cloud.google.com/gke-spot=true:NoSchedule taint to nodes in those new pools, so only Pods with the matching toleration can run there.
A resilient cluster uses a mix of node pools that use Spot VMs and node pools that use standard Compute Engine VMs. This mix ensures that workloads and Jobs can still be processed when no Spot VMs are available, because Spot VMs have no availability guarantees. Keep at least one non-GPU node pool that uses standard VMs before adding a GPU node pool that uses Spot VMs.
Modifying a node pool means changing how its nodes scale, how workloads are placed on them, and how they upgrade. Cluster autoscaler automatically adds and removes nodes in node pools based on demand, and it can be configured to add new nodes with a preference for Spot VMs. Starting in GKE version 1.24.1-gke.800, you can define the autoscaler location policy, where the ANY policy attempts to provision Spot VM node pools when resources are available with lower preemption risk, while other VM types use the BALANCED distribution policy by default.
Modification also includes placement controls that keep critical Pods off Spot VMs. Use node taints and tolerations so critical Pods are not scheduled onto Spot VM pools, and add a nodeSelector or node affinity rule when a workload must run only on Spot VMs. If Standard node pools using Spot VMs use surge upgrades, GKE creates surge nodes with Spot VMs but does not wait for the Spot VMs to be ready before cordoning and draining the existing nodes, because Spot VMs provide no guarantee of availability.
Deleting a node pool removes its nodes from the cluster and removes the capacity where its Pods were running. Pods scheduled on the deleted pool must be placed on another pool with capacity, so workloads should be rescheduled before removal by updating tolerations, nodeSelector rules, or node affinity to direct Pods to the remaining pools. If a deployment requires Spot VMs but does not include the required toleration, its Pods remain in an unschedulable state and are never placed.
Removal is especially disruptive for Spot VM pools because reclamation is involuntary and data on the nodes is deleted when Google Cloud preempts the instance. Local SSD disks attached to these VMs do not persist after deletion, and internal and external IP addresses used by Spot VMs might change after recreation. Reclamation is not covered by the guarantees of PodDisruptionBudgets, so you might experience greater unavailability than the configured budget allows.
Exam tip: Use both a toleration and a nodeSelector or node affinity to force Pods onto Spot VMs only; a toleration alone lets GKE schedule Pods onto either Spot VMs or existing standard VMs with capacity.
The choice between Spot VMs and standard VMs is a tradeoff between lower cost and guaranteed availability. Spot VMs are priced lower and suit fault-tolerant workloads such as batch processing, stateless applications, or jobs that frequently save progress. Standard VMs cost more but provide the stable capacity needed for critical or stateful workloads.
| VM type | Availability | Best use |
|---|---|---|
| Spot VMs | No guarantee, can be reclaimed at any time | Fault-tolerant batch jobs and stateless apps that can restart |
| Standard VMs | Stable capacity for the cluster | Critical Pods, stateful workloads, and baseline capacity |
When stateful workloads must run on Spot VMs, test that they can gracefully terminate within 25 seconds of shutdown to minimize the risk of persistent volume data corruption. Follow Kubernetes Pod termination best practices so shutdown completes cleanly before the VM is reclaimed.
Cluster autoscaler configuration and management is the automatic resizing of Standard cluster node pools based on workload demand. The cluster autoscaler (opens in a new tab) works per node pool, where an administrator sets a minimum and maximum size and the autoscaler adds or removes nodes within those bounds. When demand is high, the autoscaler adds nodes to keep workloads available, and when demand is low, it removes nodes to control cost. No manual adding of nodes or over-provisioning is needed once the bounds are set.
The cluster autoscaler makes scaling decisions based on Pod resource requests, not actual CPU or memory utilization. It periodically checks Pods and nodes and changes the size of the underlying Compute Engine Managed Instance Group (MIG) (opens in a new tab) for the node pool by adding or removing virtual machine (VM) instances. New nodes are created in parallel rather than one at a time, so unschedulable Pods can become Active as quickly as possible.
Scaling behavior follows these rules:
The relationship between resource requests and autoscaling triggers directly affects accuracy and cost. If workloads request too little CPU or memory, or rely on low defaults, Pods may suffer shortages while the cluster autoscaler sees no need to add nodes. Setting explicit resource requests for every workload helps the cluster autoscaler add nodes when they are truly needed and remove them when they are not.
Exam tip: The cluster autoscaler scales on Pod resource requests, not on live utilization, so missing or too-small requests prevent correct scale-up.
Configuring minimum and maximum node pool size sets the bounds within which the cluster autoscaler can add or remove nodes. The configuration is per node pool with either per-zone or total-pool flags. The per-zone and total-pool flag pairs are mutually exclusive, and the total-pool flags require GKE version 1.24 or later. Enabling or disabling autoscaling can restart the control plane and take several minutes, with further configuration changes taking up to one minute to propagate.
| Configuration mode | Minimum flag | Maximum flag | Scope of limit |
|---|---|---|---|
| Per-zone limits | --min-nodes | --max-nodes | Minimum and maximum nodes in the node pool per zone |
| Total-pool limits | --total-min-nodes | --total-max-nodes | Minimum and maximum nodes for the entire node pool across all zones |
Autoscaling can be enabled at creation with --enable-autoscaling or later for an existing node pool, and it can be disabled with --no-enable-autoscaling. While autoscaling is enabled, a node pool cannot be manually resized by changing the minimum and maximum values. To manually resize an autoscaled node pool, follow this order:
Do not enable Compute Engine autoscaling for managed instance groups on cluster nodes. Compute Engine autoscaling is separate from the GKE cluster autoscaler, and the two controllers conflict and can prevent node pools from scaling up or down correctly.
Balancing cost and high availability means using the cluster autoscaler to add capacity during peaks and remove it during idle periods without breaking workloads. The cluster autoscaler improves availability during peaks and lowers cost during idle periods, but scale-down can briefly disrupt workloads. If a single-replica controller loses its node, its Pod must be rescheduled to another node, while a Deployment with multiple replicas can keep serving traffic during the move. Designing workloads to tolerate restarts, keeping critical Pods from interruption, and using Image streaming to start workloads faster on new nodes all improve resilience.
Cost control comes from the minimum size, scale-down to that minimum, and preference for less expensive capacity. The cluster autoscaler assumes replicated Pods can restart elsewhere, that no one is manually managing nodes, and that all nodes in one node pool share the same labels. It tries to expand the least expensive node pool, accounts for the lower cost of node pools with Spot VMs, but may still choose a more expensive pool when resources are unavailable in a zone.
Node pool customization and maintenance in Google Kubernetes Engine (GKE) Standard clusters is the practice of tagging groups of nodes to control where workloads run and keeping those nodes updated without breaking applications. A node pool (opens in a new tab) is a group of nodes in a cluster that share the same configuration, and customization applies Kubernetes metadata to every node in the pool. Maintenance replaces or moves those nodes while protecting running Pods from disruption.
Kubernetes node labels are key-value pairs in node metadata that attract Pods to particular nodes in a node pool. A workload uses a selector that matches a label, such as label1=value1, and the Kubernetes scheduler places the Pod only on nodes that carry that label. Kubernetes node labels are distinct from GKE cluster and resource labels, which are used for billing and organization rather than scheduling.
An existing Standard node pool can receive new labels without re-creating the pool or disrupting running workloads. The update is applied through the GKE API with gcloud container node-pools update --node-labels, and the new configuration is stored in GKE so future upgrades and newly added nodes in the pool use it. The update overwrites any previous user-specified values, so all labels that should remain must be included again. Label keys and values must meet Kubernetes syntax and character-set rules, and each value passed with --node-labels must be 1,024 characters or fewer.
Node taints are key-value metadata on nodes that repel Pods and steer them away from particular nodes in a node pool. A tainted node will not accept a Pod unless that Pod declares a matching toleration, which makes taints the opposite of labels for workload separation. Taints and labels work together to isolate workloads, such as keeping special jobs on dedicated nodes.
Like labels, taints on an existing node pool can be updated without re-creating the pool, and the new taint configuration is preserved for future upgrades and new nodes. The update is applied with gcloud container node-pools update --node-taints and overwrites any previous user-specified taints. Because the configuration persists in the node pool object, nodes created later by scaling or upgrades inherit the same placement rules.
| Option | Purpose | Scheduling behavior |
|---|---|---|
| Kubernetes node labels | Identify nodes for specific workloads | Scheduler places matching Pods onto labeled nodes in the node pool |
| Node taints | Protect nodes from general workloads | Scheduler keeps Pods away from tainted nodes unless Pods have a matching toleration |
The node pool depends on both settings to enforce placement: labels pull selected Pods toward the pool, while taints push all other Pods away from the pool.
A node pool upgrade is a lifecycle operation that moves the nodes in a Standard node pool to a new Kubernetes version for security and stability. The upgrade affects running workloads because nodes are replaced, so data that lives on the local node disk can be lost unless it is saved to persistent disks before upgrading. A node pool can also be downgraded to a prior version and can be deleted when it is no longer needed, with PodDisruptionBudgets (PDBs) controlling whether deletion respects availability protections.
Exam tip: Updating --node-labels or --node-taints overwrites all previous user-specified values, so include every label or taint you want to keep.
Workload migration between node pools is the ordered process of marking old nodes unschedulable, evicting their Pods gracefully, and letting Kubernetes reschedule those Pods onto other available nodes. Migration is used when changing machine types, replacing a pool, or preparing nodes for an upgrade, and it relies on kubectl cordon and kubectl drain applied node by node.
A left-to-right process flow showing prepare, cordon old nodes as unschedulable, drain with graceful termination, then reschedule Pods onto the new pool.
spec.terminationGracePeriodSeconds, which defaults to 30 seconds.This sequence depends on workload controllers such as Deployments, ReplicaSets, StatefulSets, Jobs, or DaemonSets, because standalone Pods are not rescheduled and will block drain unless removed. System Pods such as kube-proxy may require drain with the --force option. Two timing risks apply: a LoadBalancer Service with externalTrafficPolicy: Local can become unreachable when its nodes are cordoned, so change the policy to Cluster or redeploy first, and a new pool with cluster autoscaler enabled can scale back down if left underutilized for more than 10 minutes before the drain.
Spot VMs cost less but offer no availability guarantee and can be reclaimed at any time, while standard VMs cost more but provide stable capacity. Spot VMs suit fault-tolerant batch jobs and stateless applications that can restart, while standard VMs suit critical Pods, stateful workloads, and baseline capacity.
The cluster autoscaler adds nodes when Pods cannot be scheduled on current nodes up to the maximum node pool size and removes underutilized nodes when all Pods could still be scheduled with fewer nodes down to the minimum size. Scaling decisions are based on Pod resource requests, not actual CPU or memory utilization, so missing or too-small requests prevent correct scale-up.
Both a toleration and a nodeSelector or node affinity should be used to force Pods onto Spot VMs only. A toleration alone lets Google Kubernetes Engine schedule Pods onto either Spot VMs or existing standard VMs with capacity.
Workload migration follows cordon, then drain, then rescheduling by controllers onto the new pool. Cordoning marks old nodes as unschedulable while leaving existing Pods running, draining gracefully evicts workloads within the default 30-second grace period, and workload controllers such as Deployments reschedule the evicted Pods onto available nodes.
You are creating a new node pool in an existing Google Kubernetes Engine (GKE) cluster to host dedicated, high-priority workloads. You need to ensure that general application pods are not scheduled on these specific nodes, while allowing only dedicated pods configured with matching tolerations to run on them.
Which configuration should you apply to the node pool?
Last updated