Prepare and test your skills
Prepare and test your skills
The economic model of Spot VMs (opens in a new tab) trades very low price for no availability guarantee. Spot VMs are excess Compute Engine capacity sold at discounts of up to 91% off on-demand price, and Compute Engine can reclaim that capacity at any time to run standard VMs. When reclaimed, a Spot VM is either stopped or deleted, so Spot VMs fit only fault-tolerant work that can lose and replace capacity.
Spot pricing for Spot VMs reflects spare Compute Engine capacity rather than a fixed bid, so both price and availability shift with overall platform usage. Spot prices cover many machine types, GPUs, TPUs, and Local SSD disks, and prices can change up to once every day at midnight Pacific Time. Current prices are published on the Spot VMs pricing page, through the Cloud Billing Catalog API, and through pricing announcements, while historical trends are available through the preemption and pricing history view (opens in a new tab).
Billing follows the VM lifecycle state rather than preemption alone. If a preempted VM was stopped, the VM enters a TERMINATED state and accrues no VM-hour charges while it remains TERMINATED, but attached persistent disks still incur storage charges until deleted. If Compute Engine preempts a Spot VM less than one minute after creation, there is no charge for that VM use, and the same first-minute waiver applies to attached GPUs and Local SSDs. Premium operating systems are the exception: Spot VMs do not reduce premium OS cost, minimum-usage charges still apply, OS billing rounds up as normal, and OS charges continue even when preemption occurs in the first minute.
Preemption selection for Spot VMs describes which RUNNING Spot VMs Compute Engine reclaims when it needs capacity for standard VMs, system events, or recovery actions. The probability of preemption is generally low but varies from day to day, from zone to zone, and across machine families and resources, similar to resource-availability errors. Because Spot capacity is finite, Spot VMs also might not be available at creation time even if they were available before.
Only Spot VMs in a RUNNING state are considered for preemption, while Spot VMs in a TERMINATED state are not. Stopping a Spot VM and then restarting it therefore resets the preemption process because stopping leaves the VM in a TERMINATED state, and the VM can be stopped and restarted repeatedly as long as capacity exists. Resetting or rebooting instead leaves the VM in a RUNNING state and does not reset the process. The reason for any single preemption event is not visible, so planning must rely on aggregate signals such as expected uptime and historical preemption rate by machine type and location.
Exam tip: Stopping then restarting a Spot VM resets preemption because the VM passes through TERMINATED, while resetting or rebooting keeps it RUNNING and does not reset preemption.
The preemption process for Spot VMs is the ordered sequence Compute Engine follows to reclaim a Spot VM while giving the workload a brief chance to react. The workload detects preemption by monitoring instance metadata, handles it during a short shutdown window, and then the VM moves to the configured final state. The sequence runs in this order:
preempted value in VM default metadata to TRUE. The preemption notice duration controls the delay before shutdown begins and is either 120 seconds in Preview or 0 seconds by default.The final state depends on the termination action configured for each Spot VM:
| Termination action | Final state | What remains billable |
|---|---|---|
| STOP, the default when unspecified | VM transitions to TERMINATED and remains visible in the project | No VM-hour charge while TERMINATED; attached persistent disks still incur storage charges |
| DELETE | VM is deleted | No VM-hour charge after deletion; auto-delete persistent disks are deleted with the VM |
Historical preemption and pricing data for Spot VMs turn the spare-capacity model into a placement decision across machine types, zones, and regions. The historical preemption rate, or preemptionHistory, reports daily rates for a machine type and zone over the past 30 days, with day boundaries at midnight Pacific Time. Compute Engine calculates each daily value as total preempted Spot VMs in a day divided by total Spot VMs that stopped running that same day, where stops include user stops, programmatic stops, limited runtimes, deletions, suspensions, and preemptions.
The resulting rate ranges from 0.00 to 1.00, so 0.50 means Compute Engine preempted half of that machine type and zone combination that stopped that day across Google Cloud. Historical pricing, or priceHistory, shows hourly USD prices by machine type and region over the past year, with each entry covering the interval when that price was active. Comparing that Spot price history against the corresponding on-demand price shows the potential saving, up to the documented 91% maximum, while comparing preemption histories shows which combinations have been more stable. This history is not available for N1 machine types with GPUs attached, custom machine types, or TPUs, and Spot VMs carry no Service Level Agreement and are excluded from the Compute Engine SLA.
Operational management of Spot VMs (opens in a new tab) on Google Compute Engine means treating excess, preemptible capacity as a fleet rather than as individual servers. A fleet strategy combines Managed Instance Groups with mixed provisioning models for resilience, Terraform and instance templates for repeatable provisioning, and availability and preemption telemetry for cost-aware operations. Because Compute Engine can stop or delete Spot VMs at any time, the fleet must be designed to lose capacity and then automatically regain it when capacity returns.
A Managed Instance Group (MIG) is the control plane for a Spot VM fleet because the MIG continuously attempts to maintain its target size or autoscaler size from an instance template. When Compute Engine preempts one or more Spot VMs in the group, the MIG repeatedly tries to recreate those VMs from the same template, and replacement VMs return to RUNNING only when vCPU, memory, and accelerator capacity become available again. If Spot capacity remains limited in a zone, the MIG cannot resize or autoscale out until capacity returns, which is why fleet design must add flexibility rather than relying on a single machine type in a single zone.
Fleet flexibility comes from regional placement and machine-type flexibility. Regional MIGs spread Spot VMs across multiple zones so a zonal stock-out does not remove the whole fleet, while instance flexibility allows multiple machine types in the same MIG so the MIG can favor types with lower observed preemption rates. For Google Kubernetes Engine (GKE) (opens in a new tab), the same principle means maintaining a mix of Spot node pools and standard-VM node pools, keeping at least one non-GPU standard pool before adding a GPU Spot pool, and using node taints and tolerations so critical Pods avoid Spot nodes. Spot VM data, including attached Local SSD data, is deleted on preemption, so stateful work must checkpoint externally and be tested to terminate gracefully within 25 seconds on GKE.
A Terraform configuration is the repeatable way to provision Spot VMs because the same scheduling block defines both the economic model and the preemption behavior for every VM. The resource google_compute_instance depends on a boot disk image, a machine type, a zone, and a network interface, and fleet behavior depends on these scheduling fields:
provisioning_model = "SPOT" selects the spot economic modelpreemptible = true marks the VM as preemptible capacityautomatic_restart = false prevents automatic restart on host eventsinstance_termination_action set to STOP or DELETE controls the final state after preemptionSTOP is the default termination action and leaves a stopped VM that can be restarted manually when capacity exists, while DELETE removes the VM on preemption and relies on the MIG to recreate it. A metadata shutdown-script handles cleanup during the best-effort shutdown period, and the same template pattern is required when the fleet scales through MIGs or the bulk instance API. Like other VMs, Spot VMs start on creation and stopped Spot VMs can be restarted to RUNNING as long as capacity exists, but they cannot live migrate into standard VMs.
The same fields appear in imperative tooling, which lets a continuous integration and continuous delivery (CI/CD) pipeline promote one tested definition from development to production. The gcloud CLI creates a Spot VM with --provisioning-model=SPOT, --instance-termination-action, --preemption-notice-duration, and --metadata shutdown-script, while the Compute Engine API uses instances.insert with provisioningModel and instanceTerminationAction. For updates, the pipeline builds a new instance template with a new machine type or custom OS image family and then applies it to the MIG with automatic or selective update, so every VM in the fleet converges to a deterministic, versioned definition.
Exam tip: Handle preemption inside the workload when the VM uses a 120-second preemption notice duration, and handle preemption inside a shutdown script limited to about 30 seconds when the VM uses the default 0-second notice.
Capacity Advisor (opens in a new tab) is the pre-flight observability tool for Spot VM fleets because it answers whether a requested machine mix is obtainable before creation is attempted. The check takes a region or zone list, up to five machine types, a provisioning model of SPOT, a requested size, and a target distribution shape, and it returns scores for obtainability and estimatedUptime plus shards that map instanceCount and machineType to each destination zone. Historical preemption rate and historical pricing provide the cost-effectiveness side of the same decision, letting operators compare machine types against interruption risk and spot discounts of up to 91% for machine types, GPUs, TPUs, and Local SSDs.
The distribution shape controls where Compute Engine is allowed to place the fleet:
| Target shape | Placement behavior | Choose when |
|---|---|---|
| ANY | Places Spot VMs in one or more zones based on availability | The workload is batch and zones do not need to coordinate |
| ANY_SINGLE_ZONE | Places Spot VMs only in a single zone based on availability | The workload needs extensive inter-VM communication, such as AI or high performance computing |
| BALANCED | Places Spot VMs in one or more zones and spreads them as evenly as possible | The workload needs to minimize the impact of a zonal failure for serving or batch |
After the fleet is running, interruptions are observed from outside and inside the VM. From outside, zoneOperations with the filter operationType="compute.instances.preempted" lists preemption operations for a project and zone, optionally narrowed by targetLink to a specific VM, with status: DONE and statusMessage: Instance was preempted. From inside, preemption-handling code or a shutdown script performs cleanup such as stopping the application process and copying a checkpoint file to a Cloud Storage bucket before the operating system kills remaining processes. Fleet resilience is then verified by running simulated host maintenance events that force Spot VMs to preempt, by confirming the provisioning model through the console Management Availability policies view or gcloud compute instances describe showing provisioningModel: SPOT, and by confirming that the MIG or GKE cluster recreates capacity when resources return.
Resilient architecture for Spot VMs (opens in a new tab) assumes any or all Spot VMs can be reclaimed at any time because Spot VMs are excess Compute Engine capacity with no availability guarantee. A resilient design treats Spot VMs as ephemeral compute, keeps state outside the VM, saves progress continuously, and spreads replicas so a single zone or machine-type failure does not stop work. Because preemption patterns can shift suddenly, such as mass preemption during a partial zonal outage, workloads must tolerate both isolated and correlated losses.
Fault-tolerant design for Spot VMs means the application makes progress even when nodes disappear without warning and without a service level agreement. Spot VMs remain in a RUNNING state until Compute Engine reclaims them, then move to a TERMINATED state if the termination action is STOP or disappear if the action is DELETE. Stateless services, batch jobs, and checkpointed parallel work fit this model because a lost VM only slows the job instead of corrupting results. Stateful work that keeps authoritative data on the VM disk or on local SSD is unsafe because data on a preempted node is deleted and local SSD disks do not persist after deletion.
Resilient systems also plan for replacement capacity to be unavailable for a period. Compute Engine can deny new Spot VMs when excess capacity is exhausted, and a Managed Instance Group or autoscaler cannot resize until resources return. The documented fallback pattern is to retry Spot creation once or twice after preemption, then fall back to standard VMs, often by running mixed clusters of Spot VMs and standard VMs so baseline throughput continues. Large Spot fleets further improve odds by running during off-peak nights and weekends when data-center load is generally lowest.
Checkpoint-capable applications for Spot VMs save intermediate progress to durable storage so a preempted task resumes instead of restarting from scratch. A Spot VM workload should save progress at regular intervals and again when preemption is detected, because the shutdown window is short and best effort. The common Compute Engine pattern is a shutdown script that gracefully stops the program with SIGINT, waits for it to exit, then copies a checkpoint file to a Cloud Storage bucket with gcloud storage cp.
Checkpointing depends on detecting preemption from inside the VM through instance metadata. A workload can poll the metadata server for the current value of preempted, where TRUE means Compute Engine has started preemption and FALSE means normal operation. A workload that needs dedicated handling time can issue a hanging request with ?wait_for_change=true that returns only when preempted flips to TRUE, which triggers in-workload save logic outside a shutdown script. Simulated maintenance events can force this path for testing how detection and upload behave under time pressure.
Graceful shutdown for Spot VMs is the ordered sequence from preemption signal to VM stop or delete, and the chosen preemption notice duration controls where handling code runs. Compute Engine first sets preempted to TRUE in metadata, then after the notice duration sends an ACPI G2 Soft Off signal that starts the shutdown period and runs any shutdown script. The shutdown period is best effort and up to 30 seconds, shorter than for standard instances, and if the VM has not stopped Compute Engine sends ACPI G3 Mechanical Off and then applies the configured termination action.
The preemption flow proceeds in order:
preempted becomes TRUE.STOP, the default, or removes the VM when the action is DELETE.The two notice options split handling between the workload and the shutdown script:
| Preemption notice duration | Where handling runs | Shutdown-script requirement |
|---|---|---|
| 0 seconds, default | Shutdown script during the up-to-30-second shutdown period | Required to save progress on preemption |
| 120 seconds, Preview | In-workload code during the dedicated 120-second notice window, with optional shutdown script afterward | Optional, but workload must poll or wait on metadata |
Exam tip: Choose in-workload handling with 120 seconds when the task needs more than 30 seconds of dedicated save time; choose 0 seconds with a shutdown script when handling must fit inside the default best-effort shutdown period.
A Managed Instance Group (MIG) distributes Spot VM replicas and automatically attempts to restore target size after preemptions. A MIG creates Spot VMs from an instance template that sets the spot provisioning model, then continuously tries to maintain its target size or autoscaler size by recreating stopped instances when capacity returns. The MIG can also repair preempted Spot VMs through autohealing by recreating them, but it cannot grow during a capacity shortage. Stopping then restarting a stopped Spot VM leaves it in TERMINATED between operations, while reset or reboot leaves it in RUNNING and does not reset its exposure to preemption.
Flexibility across locations and machine types is the main lever for reducing both stockouts and correlated preemptions:
For example, capacity for n2-standard-16 is generally more likely than for n2-custom-24-96, which in turn is generally more likely than for n2-standard-32. Combining these choices with instance templates or the bulk instance API lets teams launch consistent replacements quickly, retry Spot placement, and keep standard VMs alongside Spot VMs for guaranteed progress.
Google Kubernetes Engine (GKE) (opens in a new tab) runs Spot VMs as nodes that behave like a MIG but without availability guarantees, so Kubernetes controllers must replace Pods faster than nodes disappear. When Compute Engine reclaims capacity, GKE receives the preemption notice and begins graceful node shutdown, during which the workload controller creates replacement Pods on surviving nodes. By default GKE allows 30 seconds total, split into 15 seconds for regular Pods followed by 15 seconds for system Pods with system-cluster-critical or system-node-critical priority classes. The total can be extended to 120 seconds in Preview by setting shutdownGracePeriodSeconds and shutdownGracePeriodCriticalPodsSeconds in node system configuration for Standard node pools or in kubeletConfig for custom ComputeClasses, where the critical-pods value must be smaller than the total.
Isolation and placement controls keep critical work off ephemeral nodes while letting tolerant work prefer Spot capacity. GKE adds the cloud.google.com/gke-spot=true:NoSchedule taint to Spot nodes, so only Pods with the matching toleration can schedule there, and a nodeSelector or node affinity for Spot VMs is required when Pods must land only on Spot nodes rather than any node with room. Best practice is to maintain a mix of Spot node pools and standard-VM node pools, keep at least one standard non-GPU pool before adding Spot GPU pools, and expect node names to persist while internal and external IPs may change after recreation. Reclamation is involuntary and is not covered by PodDisruptionBudgets, surge upgrades on Spot pools do not wait for Spot surge nodes to become ready, and stateful Pods on Spot should be tested to terminate within about 25 seconds to reduce persistent-volume corruption risk.
preempted to TRUE, sends ACPI G2 Soft Off to start an up-to-30-second shutdown period, sends ACPI G3 Mechanical Off if still running, then applies the STOP or DELETE termination action.cloud.google.com/gke-spot=true:NoSchedule taint to Spot nodes so only Pods with the matching toleration schedule there, and Spot reclamation is not covered by PodDisruptionBudgets.When a Spot VM is reclaimed, Compute Engine sets metadata preempted to TRUE, sends ACPI G2 Soft Off to start a best-effort shutdown period of up to 30 seconds, sends ACPI G3 Mechanical Off if the VM is still running, then applies the configured termination action. STOP is the default and transitions the VM to TERMINATED where it remains visible with no VM-hour charge while TERMINATED but attached persistent disks still incur storage charges, while DELETE removes the VM with no VM-hour charge after deletion and auto-delete persistent disks are deleted with the VM.
Yes, stopping then restarting a Spot VM resets the preemption process because stopping leaves the VM in a TERMINATED state and only RUNNING Spot VMs are considered for preemption. Resetting or rebooting instead leaves the VM in a RUNNING state and does not reset the process, and a stopped Spot VM can be stopped and restarted repeatedly as long as capacity exists.
Applications should poll instance metadata for the preempted value, where TRUE means Compute Engine has started preemption and FALSE means normal operation, or use a hanging request that returns only when preempted flips to TRUE, and should save progress to durable storage such as a Cloud Storage bucket. With the default 0 seconds, handling runs in a shutdown script during the up-to-30-second shutdown period and the script is required, while with 120 seconds in Preview, handling runs in workload code during the dedicated 120-second notice window and the shutdown script is optional.
An enterprise DevOps team is optimizing compute costs for a latency-sensitive, stateless microservice governed by Cloud Service Mesh. To implement FinOps cost-reduction practices, the team provisions Spot Virtual Machine (VM) Managed Instance Groups (MIGs) across multiple zones to serve as the primary compute tier, backed by standard on-demand VM MIGs to handle spillover and maintain high availability.
During periodic Spot VM preemption events in a specific zone, multiple instances are reclaimed simultaneously. The surviving Spot VM instances in that zone risk experiencing immediate traffic saturation and cascading failures before autoscaling can replace them.
Which traffic management configuration should the DevOps engineer implement in Cloud Service Mesh to protect degraded instance groups and ensure resilient traffic distribution?
Last updated