Prepare and test your skills
Prepare and test your skills
Service management is the end-to-end control of a cloud service from planning and deployment through maintenance and retirement. A managed service (opens in a new tab) is the logical resource integrated with Service Infrastructure, with only two immutable properties — service name and producer project — plus service consumers, a history of immutable service configurations, and a history of immutable service rollouts. The configurations referenced by the latest rollout represent the current state, from display name to metrics definitions to rate limits, so every lifecycle change is made by pushing a new configuration and rollout.
The managed service lifecycle defines how a service moves from creation through active operation to deletion and possible recovery. Implementation is managed with gcloud endpoints services commands or the Service Management API for programmatic control, because the declarative model requires immutable configurations and rollouts to reach the desired state. Retirement is therefore a sequence of configuration, rollout, and cleanup steps, not a single delete.
gcloud endpoints services list to inventory what exists.gcloud endpoints services describe to confirm consumers, limits, and dependencies.gcloud endpoints services delete when it is no longer needed, which starts a mandatory retention window.gcloud endpoints services undelete within its retention window if retirement must be reversed.Isolation and ownership planning prevents shutdown from creating security gaps or orphaned billing dependencies. A dedicated producer project provides the best security and isolation and allows ownership to be transferred to another team or company during deprecation. Integration depends on paid products including the Service Control API, Cloud Logging API, and Cloud Monitoring API, so the plan must account for disabling or transferring those dependencies while Service Management API, Service Control API, and Cloud Billing remain enabled in the producer project.
Virtual Private Cloud (VPC) (opens in a new tab) networks and global external Application Load Balancers are global resources that are resilient to zone and region outages. Because teams often use a single instance of a global resource, that resource can become a single point of failure. Any modification to global resources, such as routes, firewall rules, or forwarding rules, is classified as a high-risk change that additional reviewers must verify and approve.
Data preservation ensures business and compliance needs survive after traffic stops. Retirement must define an exit plan for services with unique characteristics, including data retrieval and operational continuity, and verify that data can be exported to open formats like CSV, Parquet, and Avro. Open-technology support such as Google Kubernetes Engine (GKE) (opens in a new tab) support for Open Container Initiative format reduces lock-in during migration.
A controlled shutdown changes production infrastructure progressively and monitors availability throughout the retirement window. The change is first deployed to a few virtual machines in one zone, effects are monitored, and wider rollout proceeds only after verification. Availability is monitored through the Google Cloud Service Health Dashboard (opens in a new tab), which shows current status across regions, plus incident history with duration, affected zones and regions, affected services, and workarounds. Personalized Service Health filters incidents to a specific project and supports per-project or per-organization API access, alerts, and RSS feeds.
The progressive wind-down follows an ordered sequence:
Systematic decommissioning removes residual resources so no orphaned jobs, identities, or billable infrastructure remain. Deleting selected resources inside a project is preferred when only part of the environment retires, while deleting the whole project through the Manage resources page stops all contained resources together. Every deleted managed service has a mandatory 30-day deletion retention during which undelete can restore the service name, producer project binding, configurations, and rollouts.
Decommissioning a full project follows an ordered workflow where the project boundary controls all contained resources:
Exam tip: Every deleted managed service has a mandatory 30-day retention, and undelete works only within that window.
The shared responsibility model defines which maintenance tasks Google performs and which tasks the customer owns for GKE. Google protects hardware, firmware, kernel, operating system, storage, and network, and hardens and patches the GKE control plane and node components. The control plane runs on Google-operated, single-tenant Compute Engine instances and includes the API server, scheduler, controller manager, cluster certificate authority, etcd, and IAM authenticator, while node components include kubelet configuration, metadata server for Workload Identity, Container Network Interface plugin, persistent volume CSI driver, and logging and monitoring agents.
Customers own application code, build files, container images, data, role-based access control and IAM policy, and the containers and pods they run. Customers must rotate cluster credentials, monitor clusters and respond to alerts, keep logging and monitoring enabled, and provide environmental details for troubleshooting. Google makes patches for node operating system images such as Container-Optimized OS and Ubuntu available automatically when auto-upgrade is enabled or a release channel is used, but the operating system inside customer containers remains customer responsibility.
GKE upgrades preserve availability through redundancy, gradual rollout, and health-aware routing. A regional cluster distributes control plane components, nodes, and pods across multiple zones so the control plane stays reachable when one zone fails. Compute Engine managed instance groups distribute virtual machines across zones as a logical unit, regional persistent disks replicate data to two zones, and load balancers with health checks send traffic only to healthy backends. Even when virtual machines remain in RUNNING state, crashed application software is excluded from traffic by health checks.
Capacity management forecasts load from user counts, request rates, historical trends, seasonality, events, and growth, then provisions with N+1 redundancy, benchmarks under normal and peak load, manages zonal, regional, and global quotas, reserves Compute Engine capacity, and monitors utilization to remove idle resources. Validation uses escalating resilience tests — introduce intermittent failures, block dependencies, block all network communication, then terminate hosts — plus load tests that confirm latency, resource counts, and behavior at scaling limits. Blameless postmortems and continuous integration with Cloud Build then convert incidents into fixes for code, configuration, and automation.
Progressive deployment changes production in small, monitored stages to limit blast radius and preserve quick revert. The workflow depends on monitoring between stages so each expansion is gated by evidence that the workload still runs as expected. Availability targets shape launch scope because wider distribution across failure domains provides higher expected uptime, though increasing nines also increases complexity, latency considerations, and cost.
| Deployment location | Availability target | Estimated maximum downtime in 30 days |
|---|---|---|
| Single zone | 99.9% | 43.2 minutes |
| Multiple zones in a region | 99.99% | 4.3 minutes |
| Multiple regions | 99.999% | 26 seconds |
These percentages are design targets, not guarantees, since service-level agreements for specific services can differ. A launch plan should select dependencies whose designed-for targets exceed the application's intended reliability. Workload monitoring remains essential because software crashes or deleted forwarding rules can cause unavailability with no platform incident.
Capacity planning ensures forecast capacity remains available, reliable, and cost-effective through quotas, reservations, and queued provisioning. Dynamic Workload Scheduler (DWS) (opens in a new tab) is a queued provisioning mechanism for constrained accelerators such as GPUs and TPUs. A workload submits a request for GPU-powered virtual machines to a dedicated pool, and Compute Engine (opens in a new tab) provisions capacity when it becomes available instead of failing on stock-out, which suits batch, AI/ML, and high-performance computing workloads that tolerate delayed start for higher obtainability and dense placement.
Obtainability modes define how accelerator capacity is requested, guaranteed, and billed across Compute Engine, GKE, Cloud Batch, and Agent Platform training and inference. On-demand virtual machines consume capacity without advance arrangement but can fail during stock-outs. Spot virtual machines offer the deepest discounts but can be preempted at any time with a 30-second warning and impose statelessness constraints. Reservations provide assured exclusive capacity in a specific zone but require paying for the full period.
| Obtainability mode | Provisioning guarantee | Key tradeoff |
|---|---|---|
| On-demand | Immediate if capacity available, requires on-demand quota | Most flexible, but no guarantee and subject to stock-out |
| Spot | Immediate if capacity available, requires preemptible quota | 61% to 90% discount, but preemptible at any time |
| Reservations | Assured exclusive capacity in a specific zone | Charged for full period whether used or not; cannot be used for Flex-start, Spot, sole-tenant, or f1-micro and g1-small VMs |
| Dynamic Workload Scheduler with flex-start | Queued and provisioned when available, runs uninterrupted up to seven days | No reservation required, dense allocation with up to 53% discount; start is not immediate |
In GKE, flex-start provisions GPU, TPU, and H4D nodes through the same queue, while Cloud Batch uses flex-start virtual machines for GPU jobs. The workflow depends on quota: on-demand requests depend on on-demand quota, while Spot and flex-start depend on preemptible quota, and failed accelerator creates in GKE return a Quota exceeded error when matching quota is missing.
Clustered GPU consumption options in AI Hypercomputer control duration, density, and approval path for large-scale training and multi-host inference. All clustered options place resources as close as possible to minimize network latency, with future reservations through an account team adding topology-aware scheduling with enhanced monitoring and maintenance. Flex-start and Spot support all GPU machine types except A4X Max and A4X, while future reservations cover clustered GPUs and A3 Edge types.
| Consumption option | Best for duration and stability | How to request |
|---|---|---|
| Future reservations in AI Hypercomputer | Long-term pre-training or multi-host inference, up to 53% discount | Request through account team for future start, consume with reservation-bound model |
| Future reservations in calendar mode | Scheduled capacity for 1 to 90 days, up to 53% discount | Self-service search by region and date; approval within a minute |
| Flex-start | Short-lived dense clusters up to seven days, up to 53% discount or on-demand rates | Create request through Compute Engine, Cluster Director, Cluster Toolkit, or GKE |
| Spot | Fault-tolerant or short-lived work, 61% to 90% discount | Specify Spot model at instance or node-pool creation |
At the end of a calendar-mode or AI Hypercomputer reservation, Compute Engine deletes the reservation and stops or deletes consuming instances based on their termination action. Standard future reservations differ because they require sufficient quota both at review and at start time, and Google Cloud review typically takes five days.
Accelerator quota restricts GPU and TPU use by type, quantity, region, and consumption option to preserve availability. Projects generally start without significant accelerator quota, quota must be requested separately for each consumption option, and nodes provisioned by custom ComputeClasses in Autopilot still consume project quota. Free Trial accounts cannot obtain meaningful GPU or TPU increases and must upgrade to a paid account.
Workload characteristics determine the fit:
ComputeClasses orchestrate options under constraints by expressing fallback priority. The control plane provisions nodes from the first available option, so a workload can prefer reserved capacity and fall back to flex-start or Spot without redeployment.
Exam tip: Flex-start requires preemptible quota like Spot, but flex-start runs uninterrupted for up to seven days once provisioned, while Spot can be preempted at any time with a 30-second warning.
Allocation quotas (opens in a new tab) and rate quotas control how much can exist and how fast APIs can be called. Resource quotas, also called allocation quotas, are the maximum creatable resources such as regional GKE clusters, concurrent virtual machines, or load balancers. Allocation quotas do not reset over time and are released only when the resource is released, while system limits are fixed values that cannot be changed. Rate quotas limit API requests per period, such as requests per second, and quotas can be zonal, regional, or global.
Quota monitoring compares current and forecast usage against limits to trigger increases before exhaustion. Capacity Planner (opens in a new tab) extends Cloud Monitoring with historical and forecasted usage of virtual machine instances, Persistent Disk volumes, and GPUs, plus reservations, proximity to limits, and automatic increase requests. Operational checks use the Quotas page with Filter table, gcloud compute project-info describe --project PROJECT_ID for project quotas, and gcloud compute regions describe for regional use. Load tests under normal and peak conditions validate how much load the system handles while meeting latency targets.
Quota adjustment raises quota values through manual requests or automated submission when consumption nears a limit. Quota adjuster monitors consumption and proactively submits adjustment requests so manual requests are not needed. Most requests are evaluated by automated systems based on resource availability, length of Google Cloud use, and other factors, and denied requests must be pursued through a support case or account team.
| Task | Required role |
|---|---|
| Check quotas for a project | Project Owner, Project Editor, or Quota Viewer |
| Modify quotas, request additional quota | Project Owner, Project Editor, Quota Administrator, or custom role with serviceusage.quotas.update |
There are three primary ways to submit an increase: through the Google Cloud console for the fastest path, through a support case with Live chat for quick representative contact, and directly with the account team or Technical Account Manager when deeper analysis is needed. Quota exhaustion blocks consumption: a gcloud command outputs a quota exceeded error with exit code 1, while an API request returns HTTP 413 Request Entity Too Large. Quota does not guarantee resource availability, because zonal stock-out can still block creation even with available quota.
A Compute Engine reservation is a zone-specific guarantee that capacity will be available for virtual machines with matching hardware and optional accelerators. The reservation holds vCPUs and memory for a chosen machine type plus GPUs, Local SSDs, TPUs, or H4D resources. When a matching virtual machine is created, no additional charge applies for reserved vCPUs, memory, GPUs, or Local SSDs, while unreserved attachments such as disks or IP addresses are billed separately. All reservation types share two boundaries: capacity is locked to a single zone, and reserved capacity cannot create f1-micro, g1-small, Flex-start, sole-tenant, Spot, or preemptible instances.
An on-demand reservation attempts to reserve capacity immediately and is created only if capacity is available in the zone, then can be consumed, modified, or deleted at any time. A future reservation requests capacity for a set future period, is typically reviewed in five days, and automatically creates on-demand reservations at the start time. Once approved and in PROVISIONING state, a standard future reservation cannot be cancelled and commits the owner to pay for the entire period regardless of usage.
| Reservation path | Approval | Best fit | Provisioning model for consuming VMs |
|---|---|---|---|
| Standard future reservation | Typically five days | General capacity for a set future period | Standard provisioning model |
| Future reservation in calendar mode | Within a minute with confirmed-available properties | GPU, H4D, or TPU up to 90 days | Reservation-bound provisioning model |
| Future reservation through account team | Immediate after draft submitted | Large AI blocks or H4D clusters needing topology-aware scheduling | Reservation-bound provisioning model |
Share type controls which projects can consume a reservation, while consumption settings control which virtual machines can consume it. Single-project reservations use LOCAL share type for the owner project only, while shared reservations use SPECIFIC_PROJECTS for selected consumer projects in the same organization. With automatic use, any matching virtual machine consumes the reservation; with specific targeting, only virtual machines that name the reservation consume it, which is required for A3 machine types or compact placement policies.
Autoscaling maintains availability and efficiency by adding capacity when load rises and removing it when load falls. Managed instance group (MIG) (opens in a new tab) autoscaling, Google Kubernetes Engine autoscaling, and Cloud Run elasticity each implement this pattern with different signals and boundaries. The common tradeoff is responsiveness against cost: lower utilization targets and earlier scale-out protect latency but run more capacity, while denser packing lowers cost but leaves less margin before saturation.
The Horizontal Pod Autoscaler (HPA) is the workload controller that adds or removes Pod replicas based on real-time demand. The HPA controller reads a metric, compares current value to target, calculates desired replicas bounded by minReplicas and maxReplicas, and updates the target Deployment. Deleting an HPA object does not revert replica count, so the Deployment remains at its existing scale until manually scaled with kubectl scale.
HPA supports three metric families:
requests to use averageUtilization targets.Custom-metric autoscaling depends on an ordered pipeline where each stage feeds the next:
PodMonitoring, typically every 15 seconds, or system metrics emit duty cycle every 60 seconds.autoscaling/v2 HPA references Deployment in scaleTargetRef with averageUtilization or averageValue target.Configure targets such as 70% or 80% to preserve headroom while replicas start. The algorithm applies a default 0.1 tolerance band to dampen oscillation, and autoscaling/v2 is required for modern features such as scaling to and from zero.
The Vertical Pod Autoscaler (VPA) is the workload controller that right-sizes containers by adjusting CPU and memory requests and limits from historical use. VPA detects when initial requests are too large or small, then re-provisions the Pod with corrected configuration, for example from 1 CPU and 16 GB to 4 CPUs and 64 GB. Because re-provisioning restarts the Pod, VPA trades placement stability for long-term efficiency. Keep VPA in Off recommendation-only mode for at least 24 hours and ideally one week, then set minimum and maximum bounds before enabling Initial or Auto.
The Cluster Autoscaler is the infrastructure controller that adds or removes nodes in GKE Standard node pools based on scheduling simulation. When HPA creates Pods that cannot be scheduled for lack of CPU, memory, or capacity, the Cluster Autoscaler provisions a new node in an existing pool. When nodes become underutilized, it moves Pods onto other nodes and terminates the empty node. The workload depends on the Cluster Autoscaler for placement, while the Cluster Autoscaler depends on PodDisruptionBudgets, minimum node parameters, and scheduling constraints to scale down safely without evicting too many replicas at once.
Exam tip: Do not configure HPA and VPA on the same CPU or memory metrics, because HPA adding replicas lowers per-Pod utilization while VPA simultaneously lowers per-Pod requests and the controllers conflict.
Node pool auto-creation extends the Cluster Autoscaler from scaling existing pools to creating new pools with specialized hardware. While the Cluster Autoscaler adds nodes of an existing type, auto-creation provisions pools whose shape, CPU, memory, or accelerators precisely fit pending Pods. This minimizes waste from placing small specialized Pods on oversized nodes and supplies GPUs or TPUs on demand without pre-provisioning. If demand is static and predictable, such as long-running training, manually creating a node pool with a specific machine type remains appropriate.
Auto-creation is triggered by hardware availability, not CPU load. A trigger occurs when a Pod requests specific hardware and no node can satisfy it, leaving the Pod in Pending, or when a Pod requests a ComputeClass such as cloud.google.com/compute-class: premium-gpu and no node provides those capabilities. A ComputeClass defines a prioritized list of rules for fulfilling a capability request, separating the Pod request from provisioning. Instead of demanding a3-highgpu-8g or nvidia-l4-vws, the Pod requests a capability and GKE selects the implementation.
Autoscaler (opens in a new tab) configuration for a MIG automatically adds virtual machines when load rises and deletes them when load falls. A MIG is a collection of virtual machines from a common instance template, and only managed groups can attach an autoscaler. The autoscaler is a separate API resource — autoscalers for zonal MIGs and regionAutoscalers for regional MIGs — that acts on the target MIG using the Compute Engine Service Agent.
An autoscaling policy defines one or more scaling signals plus minimum and maximum replicas. When multiple signals are configured, the autoscaler calculates a recommended count for each signal and sets group size to the largest recommendation. A policy must contain at least one signal, and enabling autoscaling adds a CPU utilization (opens in a new tab) signal by default that can be edited or removed. Supported targets include average CPU across vCPUs, HTTP load balancing serving capacity defined in the backend service (opens in a new tab), and Cloud Monitoring metrics.
CPU-based autoscaling maintains average utilization near a target such as 0.60 for 60% by adding virtual machines when usage exceeds target and removing them when below. Data flows from each virtual machine's CPU usage to the autoscaler, which averages usage and changes MIG size. Avoid targets at 85% or higher for slow-initializing applications because existing CPUs can overload while new virtual machines start. During initialization period coolDownPeriodSec, default 60 seconds, metrics from starting virtual machines are ignored. During stabilization period stabilizationPeriodSec, default 600 seconds and allowed 0 to 3600 seconds, scale-in deletions are delayed to prevent thrashing.
Cloud Monitoring metric scaling adjusts capacity from application measurements such as latency, network bytes, memory, disk operations, or queue length. Scaling is limited to up to 5 Monitoring metrics per MIG, only INT64 or DOUBLE value types, and the same metric cannot appear twice in one policy. If the metric originates from each virtual machine, the autoscaler averages values and requires at least one running virtual machine to publish data, so the MIG cannot scale to zero.
| Mode | What it maintains | How size is calculated | Choose when |
|---|---|---|---|
| Utilization target | Average or global metric value at target with type GAUGE, DELTA_PER_SECOND, or DELTA_PER_MINUTE | Adds virtual machines when value above target, removes when below | Signal is rate or saturation, such as latency or memory percent used |
| Single instance assignment | Fixed amount of work per virtual machine | Divides total metric by assignment, for example 100 messages / 5 per VM = 20 VMs | Signal is countable backlog, such as Pub/Sub queue length |
Pub/Sub-driven autoscaling uses single instance assignment where data flows from the Pub/Sub subscription to Cloud Monitoring to the autoscaler to the MIG. The autoscaler divides pubsub.googleapis.com/subscription/num_undelivered_messages filtered by subscription ID by messages-per-virtual-machine. Use num_undelivered_messages because it counts queued plus in-flight unacknowledged messages, while a metric excluding in-flight work can drop to zero during processing and trigger premature scale-in. The same pattern supports memory with agent.googleapis.com/memory/percent_used, network with compute.googleapis.com/instance/network/received_bytes_count, and backend-tier scaling with compute.googleapis.com/instance_group/size.
Predictive autoscaling scales out ahead of forecasted load for applications with long startup and predictable daily or weekly cycles. With prediction enabled, the autoscaler combines real-time CPU data with history-based forecasts to start virtual machines early. Predictive autoscaling works only with CPU utilization, requires 3 days of CPU history before predictions begin, covers daily and weekly patterns but not monthly, annual, one-time, or sub-10-minute patterns, and increases Compute Engine cost because it runs capacity earlier for availability.
Cloud Run is an inherently elastic serverless container service that scales instances automatically based on incoming requests. Traffic flows from clients to the Cloud Run service, and the autoscaler adds container instances as concurrent demand rises and removes them as demand falls, scaling to zero when minimum instances is 0. Elasticity is bounded by minimum and maximum instance counts, shaped per-instance by concurrency, and fine-tuned by CPU and concurrency utilization targets.
Request-driven scaling keeps average load near configured targets while below maximum, and maximum instances act as a hard ceiling for cost control and downstream protection. Cloud Run enforces the lower of service-level scaling.max_instance_count and revision-level template.scaling.max_instance_count. For minimum instances, revision-level template.scaling.min_instance_count overrides service-level default when traffic is directed to a specific revision. Concurrency, configured as template.maxInstanceRequestConcurrency, is the maximum simultaneous requests per instance, with a default of 80. Starting at 80 and load-testing downward maximizes utilization, while reusing a lower value such as 10 under-utilizes instances and increases cost.
Custom scaling controls define thresholds that trigger scale-up, configured as template.scaling.cpuUtilization and template.scaling.concurrencyUtilization. By default the autoscaler maintains about 60% CPU and 60% of maximum concurrency per instance. Lower targets scale out earlier to protect latency but run more instances, while higher targets pack more work per instance to lower cost but leave less margin before queueing.
| Billing mode | CPU allocation | Idle behavior |
|---|---|---|
| Request-based billing | CPU only during request processing, throttled to zero otherwise | Instance can remain alive up to 15 minutes after last request, but idle CPU throttled |
| Instance-based billing | CPU remains allocated outside requests | Instance stays ready for background work without suspension |
Choose request-based billing when work happens strictly inside the request path, and instance-based billing when background threads must run between requests. Manual scaling with scalingMode: MANUAL and manualInstanceCount: N disables autoscaling and runs a fixed pool that does not respond to bursts or idle periods. Automatic scaling follows an ordered lifecycle where first a cold start provisions a container, then the instance serves concurrent requests, then it remains idle up to 15 minutes, and only after that window is it reclaimed, with no configurable cooldown period.
Continue with this study guide: Autoscaling (e.g., managed instance groups, Cloud Run, GKE)
Allocation quotas limit how many resources can exist and do not reset over time until the resource is released, while rate quotas limit how fast APIs can be called such as requests per second. Available quota does not guarantee available capacity because a zonal stock-out can still block creation even with available quota.
Dynamic Workload Scheduler (DWS) with flex-start is suited for batch, AI/ML, and high-performance computing workloads that tolerate delayed start because requests are queued until densely allocated capacity becomes available and then run uninterrupted for up to seven days. Spot virtual machines start immediately if capacity is available and offer 61% to 90% discounts but can be preempted at any time with a 30-second warning, while both flex-start and Spot depend on preemptible quota.
A managed service has only two immutable properties, service name and producer project, so every lifecycle change is made by pushing a new configuration and rollout. Deleting a service starts a mandatory 30-day retention window during which undelete can restore the service name, producer project binding, configurations, and rollouts.
Last updated