Prepare and test your skills
Prepare and test your skills
Dynamic Workload Scheduler (DWS) (opens in a new tab) is a queued provisioning mechanism for highly constrained accelerators such as GPUs and TPUs. A workload submits a request for GPU-powered VMs to a dedicated pool, and Compute Engine provisions the capacity when it becomes available instead of failing on stock-out. This model is designed for batch, AI/ML, and high-performance computing workloads that can tolerate a delayed start in exchange for higher obtainability and dense placement.
Obtainability modes in Google Cloud define how accelerator capacity is requested, guaranteed, and billed across Compute Engine, Google Kubernetes Engine (GKE) (opens in a new tab), Cloud Batch, and Agent Platform training and inference. On-demand VMs consume capacity without advance arrangement but can fail during stock-outs. Spot VMs offer the deepest discounts but can be preempted at any time with a 30-second warning and impose statelessness constraints. Reservations provide assured, exclusive capacity in a specific zone but require paying for the full reservation period.
| Obtainability mode | Provisioning guarantee | Key tradeoff |
|---|---|---|
| On-demand | Immediate if capacity is available, requires on-demand quota | Most flexible, but no guarantee and subject to stock-out |
| Spot | Immediate if capacity is available, requires preemptible quota | 61% to 90% discount, but preemptible at any time |
| Reservations | Assured exclusive capacity in a specific zone for the reservation period | Charged for full period whether used or not; cannot be used for Flex-start, Spot, sole-tenant, or f1-micro and g1-small VMs |
| Dynamic Workload Scheduler with flex-start | Queued and provisioned when capacity becomes available, runs uninterrupted for up to seven days | No reservation required, dense allocation with up to 53% discount on supported types; start time is not immediate |
In GKE, flex-start provisions GPU, TPU, and H4D nodes through the same queue, while Cloud Batch uses flex-start VMs for GPU jobs and Agent Platform uses flex-start VMs for training and inference. The workflow depends on quota: on-demand requests depend on on-demand quota, Spot and flex-start requests depend on preemptible quota, and failed accelerator creates in GKE return a Quota exceeded error when the matching quota is missing.
Clustered GPU consumption options in AI Hypercomputer (opens in a new tab) control duration, density, and approval path for large-scale training and multi-host inference. All clustered options place resources as close to each other as possible to minimize network latency, with future reservations through an account team adding topology-aware scheduling with enhanced monitoring and maintenance. Flex-start and Spot support all GPU machine types except A4X Max and A4X, while future reservations cover clustered GPUs and A3 Edge machine types.
| Consumption option | Best for duration and stability | How to request |
|---|---|---|
| Future reservations in AI Hypercomputer | Long-term stability for pre-training foundation models or multi-host inference, with up to 53% discount for vCPUs and GPUs | Request capacity through account team for a future start date and time, then create instances with the reservation-bound provisioning model |
| Future reservations in calendar mode | Scheduled capacity for 1 to 90 days for training or batch inference that needs high availability at start time, with up to 53% discount | Self-service search for available capacity by region and date; approval occurs within a minute, then consume with the reservation-bound model |
| Flex-start | Short-lived dense clusters lasting up to seven days without a reservation, with up to 53% discount or standard on-demand rates | Create request through Compute Engine, Cluster Director, Cluster Toolkit, or GKE; resources provision as soon as available |
| Spot | Fault-tolerant, batch, or short-lived work with deepest 61% to 90% discount | Specify Spot provisioning model at instance or node-pool creation; no reservation required |
At the end of a calendar-mode or AI Hypercomputer reservation period, Compute Engine deletes the reservation and stops or deletes consuming instances based on the termination action specified for the instances. Standard future reservations differ because they require sufficient quota both when the request is submitted for review and when the start time arrives, and Google Cloud review typically takes five days.
Accelerator quota in GKE restricts use of GPUs and TPUs by type, quantity, region, and consumption option to preserve availability for all users. Projects generally start without significant accelerator quota, quota must be requested separately for each consumption option, and nodes provisioned by custom ComputeClasses in Autopilot still consume project quota. Free Trial accounts cannot obtain meaningful GPU or TPU quota increases and must upgrade to a paid account.
Workload characteristics determine which consumption option fits best:
ComputeClasses orchestrate these options under resource constraints by expressing fallback priority across consumption modes. The control plane provisions nodes to meet ComputeClass requirements from the first available option, so a single workload can prefer reserved capacity and fall back to flex-start or Spot without redeployment.
Exam tip: Flex-start requires preemptible quota like Spot, but flex-start runs uninterrupted for up to seven days once provisioned, while Spot can be preempted at any time with a 30-second warning.
Reservation selection in Compute Engine follows the urgency, flexibility, and resource type of the workload. On-demand reservations attempt to reserve capacity immediately and can be consumed, modified, or deleted at any time. Standard future reservations reserve general GPUs for a future date after account-team review, while calendar-mode reservations reserve GPUs, H4D, or TPUs for up to 90 days after a self-service availability search. Future reservations in AI Hypercomputer or for H4D high-performance computing (HPC) clusters handle high-demand resources needed for more than 90 days.
The ordered workflow from quota to running resources is:
Quota is automatically increased before capacity is delivered for future reservations in AI Hypercomputer, while calendar-mode requests charge no quota action and standard future reservations fail or partially provision if quota is insufficient at start time.
Quota management in Google Cloud is the practice of controlling per-project limits so forecast capacity remains available without unexpected failures. Cloud Quotas (opens in a new tab) is the system that monitors consumption of Google Cloud products and services, restricts consumption when a limit is reached, and provides a way to request changes and automate adjustments. Quotas generally apply at the Google Cloud project level, where use in one project does not affect available quota in another project, and within a project quotas are shared across all applications and IP addresses. Enabling billing for a project increases quotas for most services, and quotas can also increase as use of Google Cloud expands over time.
Allocation quotas and rate quotas control two different dimensions of consumption: how much can exist and how fast APIs can be called. Resource quotas, also called allocation quotas, are the maximum resources that can be created, such as the number of regional GKE clusters in a region, the number of VMs used by a project at a given time, or the number of load balancers used concurrently. Allocation quotas do not reset over time and are released only when the resource is released, while system limits are fixed values that cannot be changed. Rate quotas limit the number of API requests that can be sent to a service in a specific period, such as requests per second or images per feature per day. Quotas can be zonal, regional, or global, so capacity planning must review both the resource quotas and the API rate quotas for every service used in the project.
Quota monitoring is the continuous comparison of current and forecast usage against quota limits to trigger an increase before exhaustion. Cloud Monitoring (opens in a new tab) provides visibility into application usage and capacity and the overall health of applications and infrastructure, and teams should evaluate average and peak utilizations of top workloads to determine how much over-provisioning is needed for traffic spikes. Capacity Planner extends this view with historical and forecasted usage of virtual machine (VM) instances, Persistent Disk volumes, and GPUs, plus reservations, proximity to quota limits, and automatic quota increase requests. Operational checks use the Quotas page in the Google Cloud console with Filter table to find a specific quota, gcloud compute project-info describe --project PROJECT_ID to check project quotas, and gcloud compute regions describe to check used quota in a region. Load tests and repeated benchmarking under normal and peak conditions validate how much load the system can handle while meeting latency targets for a fixed amount of resources.
Quota adjustment is the workflow that raises a quota value through manual requests or automated submission when consumption nears a limit. Quota adjuster monitors resource consumption and proactively submits quota adjustment requests on behalf of the project so manual requests are not needed. Most quota increase requests are evaluated by automated systems based on criteria including availability of resources, length of time Google Cloud has been used, and other factors, and requests that do not meet the criteria are denied and must be pursued through a support case or account team.
| Task | Required role |
|---|---|
| Check quotas for a project | Project Owner, Project Editor, or Quota Viewer (roles/servicemanagement.quotaViewer) |
| Modify quotas, request additional quota | Project Owner, Project Editor, Quota Administrator (roles/servicemanagement.quotaAdmin), or custom role with serviceusage.quotas.update permission |
There are three primary ways to submit a quota increase request:
Quota exhaustion is the condition where an attempt to consume more than the allowed amount is blocked and the task fails. A gcloud command that exceeds quota outputs a quota exceeded error message and returns with exit code 1, while an API request that exceeds quota returns HTTP status code 413 Request Entity Too Large. To avoid unexpected failures before a high-traffic event or peak season, quotas must be matched to resource requirements in advance because all projects start with the same quotas and some quotas increase automatically based on use. A quota does not guarantee resource availability, because even with available quota a new resource cannot be created if the underlying capacity is not available in that zone or region.
Exam tip: Available quota does not mean available capacity — a zonal stockout can still block creation, while region-wide unavailability is rare.
A Compute Engine reservation (opens in a new tab) is a zone-specific guarantee that capacity will be available to create virtual machine (VM) instances with matching hardware and optional accelerators. The reservation holds vCPUs and memory for a chosen machine type plus optional resources such as graphics processing units (GPUs), Local SSD disks, tensor processing units (TPUs), or H4D high-performance computing (HPC) resources. When a matching VM is later created from the reservation, no additional charge applies for the reserved vCPUs, memory, GPUs, or Local SSDs, while unreserved attachments such as disks or IP addresses are billed separately.
Compute Engine reservations provide assured capacity by holding resources exclusively for the owner, so other customers cannot consume them during growth, traffic spikes, large migrations, or backup and disaster recovery. A reservation inherits the same properties as its chosen machine family, which means a matching VM gets the expected performance and features without re-specifying hardware details. All reservation types share two hard boundaries: capacity is locked to a single zone, and reserved capacity cannot create f1-micro and g1-small machine types, Flex-start VMs, sole-tenant nodes, or Spot VMs and preemptible instances.
An on-demand reservation attempts to reserve capacity immediately, and Compute Engine creates the reservation only if the requested capacity is available in the zone at that moment. The created reservation serves instances that use the standard provisioning model, and it can be consumed, modified, or deleted at any time after creation. Choose on-demand reservations when timing is immediate and flexibility on how long to hold the capacity matters more than a guaranteed future start date.
Creating a single-project on-demand reservation follows an ordered flow from the owner project to the zone:
A future reservation is a request for capacity for a set future period, rather than an immediate hold. The owner creates a future reservation request and submits it to Google Cloud for review, which typically takes five days. If approved, Compute Engine automatically creates on-demand reservations with the requested capacity at the chosen date and time, and matching VMs that use the standard provisioning model then consume that delivered capacity. After the reservation period ends, the resulting reservations can be modified or deleted.
Standard future reservations carry a billing commitment that differs from on-demand flexibility. Once Google Cloud approves the request, the owner commits to pay for the requested resources for the entire reservation period regardless of usage. After an approved request enters the PROVISIONING state, it can no longer be cancelled or deleted. Google Cloud also manages quota for the workflow by automatically increasing quota at the start time if the project lacks quota for the reserved resources.
Future reservations in calendar mode and through an account team serve high-demand GPU, H4D, and TPU capacity with close placement and a reservation-bound provisioning model. In calendar mode, the owner first views when a specific count and type of GPU, H4D, or TPU resources are available in a region, then creates a request using only those confirmed-available properties for a period of up to 90 days. For large-scale artificial intelligence (AI) and machine learning workloads or H4D clusters with enhanced cluster management, the owner contacts the account team to create a draft request for machine types such as A4X, A4, A3 Ultra, or A3 Mega.
| Reservation path | Approval | Best fit | Provisioning model for consuming VMs |
|---|---|---|---|
| Standard future reservation | Typically five days after submit for review | General VM capacity needed for a set future period | Standard provisioning model |
| Future reservation in calendar mode | Within a minute when created with confirmed-available properties | GPU, H4D, or TPU capacity needed for up to 90 days | Reservation-bound provisioning model |
| Future reservation through account team | Immediate after the draft request is submitted | Large AI blocks or H4D clusters needing topology-aware scheduling, enhanced monitoring, and maintenance | Reservation-bound provisioning model |
After approval, Compute Engine automatically creates an empty reservation and reserves the requested resources as close to each other as possible to minimize network latency. At the reservation start time, the owner creates matching GPU, H4D, or TPU instances that explicitly bind to the reservation. At the end time, Compute Engine deletes the reservation and stops or deletes the consuming instances according to the termination action specified on those instances.
Share type controls which projects can consume a reservation, while consumption settings control which VMs can consume it. A single-project reservation uses LOCAL share type so only the owner project can use it, while a shared reservation uses SPECIFIC_PROJECTS so selected consumer projects in the same organization can use it. The owner project for a shared future reservation must be allowed to create and modify shared reservations, and overlapping reservation periods with the same instance properties and zone cause creation to fail.
Consumption control determines whether matching VMs draw capacity without extra configuration. With automatic use, any matching VM in the allowed projects consumes the reservation without naming it. With specific targeting, only VMs that name the reservation consume it, and targeting by name is required when the instance template specifies an A3 machine type or a compact placement policy. A GPU VM reservation can also enable sharing with Google Cloud services so Vertex AI custom training jobs or prediction jobs can consume the same held GPU capacity.
Exam tip: Standard future reservations are consumed with the standard provisioning model, while calendar-mode and account-team reservations must be consumed with the reservation-bound provisioning model.
Dynamic Workload Scheduler (DWS) with flex-start queues GPU and TPU requests until capacity becomes available and then runs uninterrupted for up to seven days, while Spot virtual machines (VMs) start immediately if capacity is available but can be preempted at any time with a 30-second warning. Both flex-start and Spot requests depend on preemptible quota, with Spot offering 61% to 90% discounts and flex-start offering dense allocation with up to 53% discount on supported types.
Compute Engine reservations should be used when assured exclusive capacity in a specific zone is needed for growth, traffic spikes, large migrations, or backup and disaster recovery, while Dynamic Workload Scheduler (DWS) with flex-start should be used for batch, AI/ML, and high-performance computing workloads that can tolerate a delayed start in exchange for higher obtainability. Reservations are charged for the full reservation period whether used or not and cannot be used for Flex-start, Spot, sole-tenant nodes, or f1-micro and g1-small virtual machines (VMs), whereas flex-start requires no reservation and provisions queued capacity when available.
Allocation quotas, also called resource quotas, limit how many resources can exist such as the number of regional Google Kubernetes Engine (GKE) clusters, virtual machines (VMs), or load balancers used concurrently, and they do not reset over time but are released only when the resource is released. Rate quotas limit how fast API requests can be sent to a service in a specific period such as requests per second, and quotas can be zonal, regional, or global.
An AI platform engineering team is designing a provisioning workflow on Google Kubernetes Engine (GKE) for large-scale, distributed training jobs that require multi-host TPU/GPU slices. The workload has the following operational requirements and constraints:
Which accelerator provisioning strategy should the team implement?
Last updated