Site Reliability Practices with SLOs and Error Budgets

Balancing change, velocity, and reliability (opens in a new tab) is the practice of measuring user-facing performance, setting internal targets stricter than external promises, and using remaining error budget to decide release speed. Service level indicators (SLIs) provide the measurement, service level objectives (SLOs) provide the internal target, and service level agreements (SLAs) provide the legal promise. Error budgets and the cost of additional availability connect those targets to daily operational decisions.

1.1Defining SLIs (e.g., availability, latency), SLOs, and SLAs

Defining SLIs, SLOs, and SLAs creates a hierarchy from measurement to internal target to external promise. The SLI is the technical measurement of successful events, the SLO is the internal operational target for that measurement over a compliance period, and the SLA is the legal commitment to users with penalties such as service credits. SLI data flows from services, load balancers, and probes into Cloud Monitoring, Cloud Monitoring evaluates that telemetry against SLO goals, and SLO discipline protects SLA commitments. The hierarchy only works when the internal SLO is deliberately stricter than the external SLA for the same capability.

Promise-based SLIs count kept promises as successful valid events divided by total valid events multiplied by 100, measured over a rolling window such as 5 minutes, 1 hour, or 30 days. Filtering out invalid events such as client-caused 4xx errors and planned maintenance keeps the SLI focused on service behavior, and the gap between the SLI value and 100% defines the error budget. An SLI specification defines what to measure, such as home page loads under 100 ms, while an SLI implementation defines how to measure it. One specification can have many implementations with different tradeoffs in fidelity, coverage, and cost:

• Latency field in the application server request log.
• Metrics exported directly by the application server.
• Metrics exported by a load balancer in front of the application servers.
• Black-box monitoring service that sends synthetic requests and times valid responses.
• Application code running in the customer browser that records timing and sends it to a collection service.

Fidelity improves when measurement moves closer to the user, so browser code reflects perceived latency better than server logs. The tradeoff is that browser data also includes variation from the public internet, so browser signals are good proxies for user happiness but may not isolate a fixable service problem. Separate SLIs are needed when read requests must be fast but write requests must be successful, and when fast errors, slow successes, and slow errors have different user tolerance.

Load balancer telemetry provides SLI data from the edge without custom code in every service. SLI data flows from Google Kubernetes Engine (GKE), Cloud Run, load balancer logs, OpenTelemetry pipelines, and Google Cloud Managed Service for Prometheus into Cloud Monitoring. For non-UDP services, availability is a request-based ratio using TimeSeriesRatio on an External Application Load Balancer under the https_lb_rule resource with metric loadbalancing.googleapis.com/https/request_count. The total filter often excludes 400 class responses because they indicate client errors, while the good filter counts only 200 class responses. For multi-backend traffic, loadbalancing.googleapis.com/https/backend_request_count filtered by backend_target_name isolates one backend target.

Latency is expressed with a distributionCut that counts the share of requests inside a range such as 0 to 100 ms. Both latency metrics have value type DISTRIBUTION, with histogram buckets built from powers of 1.4:

A single latency objective cannot capture both typical performance and tail degradation, so teams pair a typical-performance objective with a tail objective such as 99.9% of requests under 1000 ms over a rolling 1-hour window. Windowed SLIs further expose short violations, such as requiring 95th percentile latency under 100 ms in 99% of one-minute windows over 28 days.

Service level objectives control reliability by setting a specific, measurable, achievable, relevant, and time-bound target over a compliance period, such as 99.9% availability or 99.5% of payments under 400 ms over 30 days. If the service misses its SLO, the response is internal, such as freezing releases and shifting work to resilience fixes. Service level agreements require a formal commitment that documents services provided, expected level such as minimum uptime or maximum resolution time, responsibilities of provider and customer, and penalties such as a 5% monthly credit per 0.1% below minimum availability. The safety margin is the numerical gap between the stricter SLO and looser SLA, such as a 99.995% SLO protecting a 99.9% SLA.

1.2Error budgets (e.g., Cloud Service Mesh definitions)

Error budgets connect remaining allowable unreliability to decisions about release velocity and operational response. Cloud Service Mesh acts as the control plane that distributes health, routing, and backend information to sidecar proxies or proxyless gRPC clients, while Cloud Monitoring collects request counts and connectivity state for budget tracking. When failed or slow requests consume the allowed budget, operators visualize consumption per service, per region, and per backend to decide whether to allow releases or freeze changes.

The relationship between SLIs, SLOs, and error budgets defines how reliability is measured, targeted, and governed. An SLI is measured performance over time, an SLO is the target such as 99.9% in 30 days, and the error budget is the allowed shortfall. A 99.9% SLO over 1,000,000 requests allows 1,000 failures. If monitoring measures 99.5% compliance, or 5,000 failures, the service has consumed five times its budget, while 99.95% compliance consumes half the budget and leaves 500 failures remaining. When performance meets the SLO, teams continue releases at normal velocity. When performance falls below the SLO, policy requires slowing releases, freezing non-essential changes, and prioritizing reliability work until the budget recovers. Cloud Monitoring visualizes this with an SLO chart showing health, compliance, absolute error budget, or fractional error budget, and a custom dashboard can show all SLOs for a set of services with up to 100 widgets per dashboard.

Error budget consumption measures how much allowed unreliability was used within a window. A burn rate alert forecasts exhaustion by comparing consumption percentage against time, where Lookback Duration defines the window of recent performance and Threshold defines the unacceptable consumption percentage. A short lookback with a high threshold detects fast burns needing immediate response, while a longer lookback with a lower threshold detects slow burns that will exhaust the budget before the period ends. The alert re-evaluates on a Retest window with a minimum of 30 minutes, which prevents flapping from brief spikes. Creating the burn rate condition from the Cloud Service Mesh dashboard follows an ordered workflow:

1. Name the condition: Click the Suggested title link or enter a custom name.
2. Set the evaluation window: Enter the lookback period in Lookback Duration, or use the default.
3. Set the exhaustion signal: Enter the consumption percentage in Threshold with position Above threshold.
4. Save the condition: Click Save to return to the Create new alerting policy window.

An alerting policy connects the burn condition to notifications and runbooks. The policy carries the condition forward, applies trigger logic, and delivers the alert to the team that must act. Completing the policy follows an ordered workflow:

1. Name the policy: Enter a name that identifies service, SLO, and burn severity.
2. Confirm conditions: Keep the populated condition, optionally adding another for multi-window coverage.
3. Set trigger logic: With one condition, leave Policy triggers at Any condition is met.
4. Route the response: Configure Notifications channels and Documentation so responders know whether to slow releases, roll back, or start incident response.
5. Activate the policy: Click Save to create the policy and open Policy details.
6. Return to operations view: Use the Navigation menu to go to Anthos Services.

Tracking SLI compliance starts from the resource type that matches the configuration APIs. For service routing APIs, the mesh signal comes from the Gateway Scope or Mesh resource with the connected_clients metric, which shows whether Envoy proxies remain connected to the control plane. For request success, the signal comes from the Compute Engine Backend Service resource with Request count, which flows from each sidecar proxy to the backend service. A drop in connected_clients means routing updates stop flowing from the control plane to those clients over xDS APIs, while an error spike in Request count means the data plane between proxies and backends is failing and directly consuming budget. Cross-region consumption is visualized with Monitoring Query Language (MQL) by fetching gce_backend_service and trafficdirector.googleapis.com/xds/server/request_count, filtering where source region does not equal destination region, and grouping by source region, destination region, and backend service ID with a one-minute rate. Grounding budgets in routing state links failures to the path from the user through an external global load balancer to the Front End microservice, then from the Front End proxy to Shopping Cart, then to Payments. If health checks report Shopping Cart VMs in us-central1 unhealthy, Cloud Service Mesh instructs Front End proxies to fail over to asia-southeast1 and notifies the managed instance group there to scale.

1.3Opportunity cost of risk and reliability (e.g., number of “nines”)

Opportunity cost of risk and reliability is the tradeoff between the business benefit of higher availability and the engineering cost of achieving it. AlloyDB Omni illustrates the tradeoff through standard, enhanced, and premium availability architectures that survive progressively larger failure scopes at progressively higher cost. Every extra nine reduces downtime but demands more redundancy, complexity, and delivery friction.

Availability percentage directly promises allowable yearly downtime, where downtime equals 1 minus availability multiplied by about 525,600 minutes per year. A 99.9% target allows about 0.1% downtime, or about 526 minutes per year, which is about 8.8 hours, while a 99.99% target allows about 0.01% downtime, or about 52.6 minutes per year. Revenue loss can be estimated as downtime minutes multiplied by revenue per minute for the affected flow. User impact grows when downtime blocks critical journeys like checkout or autoscaling during peak demand, and reputation cost grows when failures are visible or repeated.

Recovery targets prioritize components by business impact. Recovery time objective (RTO) sets how quickly service must be restored, and recovery point objective (RPO) sets how much data loss can be tolerated. Components with high developer and operations impact need the lowest, strictest RTO and RPO. Artifact Registry and source control sit in the highest-impact quadrant because an outage blocks builds, deployments, and autoscaling, so Artifact Registry needs lower targets than the infrastructure as code (IaC) pipeline. Test data with moderate developer impact and low operations impact accepts higher, cheaper objectives.

AlloyDB Omni offers standard, enhanced, and premium architectures to protect against planned and unplanned outages. Standard protects data with backups for development, Enhanced adds zonal replication for high availability within one region, and Premium adds zonal plus regional replication for high availability and disaster recovery. Failure scopes drive cost from VM to node to zone to region:

Moving from Standard to Enhanced buys survival of a full data-center loss through replicas in other zones. Moving from Enhanced to Premium buys survival of a full regional loss through replicas in a second region.

A cost-benefit framework compares cost of unavailability, such as lost deployments and developer downtime, against cost of prevention, such as duplicate infrastructure and continuous monitoring. For continuous integration and continuous delivery (CI/CD), active/passive mirrors the toolchain in a different region but stays scaled-down, and recovery updates the repository trigger to switch regions. Choose active/passive when only a small amount of downtime is tolerable, and choose backup/restore when slower restoration and higher RTO and RPO are acceptable for lower cost. The same tradeoff appears in data pipelines: a single pipeline receiving all payments from Pub/Sub or Kafka and writing to BigQuery with streaming inserts must meet the stricter less-than-10-minute SLO for all payments, while separate pipelines isolate high-priority payments to streaming and standard payments to daily batch load jobs with more Dataflow workers on the urgent path. Observability reduces impact without full duplication through metrics as numerical measurements, logs as time-stamped event records including Cloud Storage access logs and VPC Flow Logs, and traces showing a single transaction journey. Deployment pipelines must be treated like production and protected with emergency-access accounts and cached artifacts for use when npm, Maven Central, or NuGet Gallery is unavailable.

Managing service lifecycle is the end-to-end control of a cloud service from planning and deployment through maintenance, retirement, and capacity assurance. A managed service (opens in a new tab) is the logical resource integrated with Service Infrastructure, with only two immutable properties — service name and producer project — plus consumers, immutable configurations, and immutable rollouts. The configurations referenced by the latest rollout represent the current state, so every lifecycle change is made by pushing a new configuration and rollout.

2.1Service management (e.g., planning, deployment, maintenance, retirement)

Service management controls a managed service from creation through active operation to deletion and possible recovery. Implementation uses gcloud endpoints services commands or the Service Management API, because the declarative model requires immutable configurations and rollouts to reach the desired state. Retirement is therefore a sequence of configuration, rollout, and cleanup steps, not a single delete.

• List services in a producer project with gcloud endpoints services list to inventory what exists.
• Describe a service with gcloud endpoints services describe to confirm consumers, limits, and dependencies.
• Delete a service with gcloud endpoints services delete when no longer needed, which starts a mandatory retention window.
• Undelete a service with gcloud endpoints services undelete within its retention window if retirement must be reversed.

Isolation and ownership planning prevents shutdown from creating security gaps or orphaned billing dependencies. A dedicated producer project provides the best security and isolation and allows ownership transfer during deprecation. Integration depends on paid products including Service Control API, Cloud Logging API, and Cloud Monitoring API, so the plan must account for those dependencies while Service Management API, Service Control API, and Cloud Billing remain enabled in the producer project. Virtual Private Cloud (VPC) networks and global external Application Load Balancers are global resources resilient to zone and region outages. Because teams often use a single instance of a global resource, it can become a single point of failure. Any modification to global resources, such as routes, firewall rules, or forwarding rules, is a high-risk change requiring additional reviewer verification.

Data preservation ensures business and compliance needs survive after traffic stops. Retirement must define an exit plan including data retrieval and operational continuity, and verify export to open formats like CSV, Parquet, and Avro. Open-technology support such as Google Kubernetes Engine (GKE) support for Open Container Initiative format reduces lock-in.

• Use managed multi-regional storage such as Spanner and Cloud Storage while zonal or regional compute is removed.
• Take snapshots of persistent disks and store snapshots in multi-region locations before deleting zonal disks.
• Replicate regional or zonal resources to another region, to multi-regional Cloud Storage, or to hybrid options such as GKE and Google Distributed Cloud.
• Export datasets and artifacts to open formats so data remains usable after APIs are deleted.

A controlled shutdown changes production progressively and monitors availability throughout. The change is first deployed to a few virtual machines in one zone, effects are monitored, and wider rollout proceeds only after verification. Availability is monitored through the Google Cloud Service Health Dashboard, which shows current status plus incident history with duration, affected zones and regions, affected services, and workarounds. Personalized Service Health filters to a specific project and supports per-project or per-organization API access, alerts, and RSS feeds. The progressive wind-down follows an ordered sequence:

1. Reduce scope first: Apply the change to a few virtual machines or a single zone and monitor behavior.
2. Revert on signal: Revert quickly if errors, crashes, or missing forwarding rules appear.
3. Diagnose and resolve: Identify application error, configuration deletion, or infrastructure incident before retrying.
4. Expand gradually: Roll across remaining zones, regions, and global resources only after verification.

Systematic decommissioning removes residual resources so no orphaned jobs, identities, or billable infrastructure remain. Deleting selected resources is preferred when only part of the environment retires, while deleting the whole project through the Manage resources page stops all contained resources together. Every deleted managed service has a mandatory 30-day deletion retention during which undelete can restore name, producer binding, configurations, and rollouts. Decommissioning a full project follows an ordered workflow where the project boundary controls all contained resources:

1. Inventory first: List services, virtual machines, buckets, scheduler jobs, and service accounts.
2. Delete workload resources: Delete Cloud Scheduler jobs, Cloud Run services, and workload service accounts.
3. Open Manage resources: Select the project to delete and click Delete.
4. Confirm shutdown: Type the project ID and click Shut down.

The shared responsibility model defines which maintenance tasks Google performs and which the customer owns for GKE. Google protects hardware, firmware, kernel, operating system, storage, and network, and hardens and patches the GKE control plane and node components. The control plane runs on Google-operated, single-tenant Compute Engine instances and includes the API server, scheduler, controller manager, cluster certificate authority, etcd, and IAM authenticator, while node components include kubelet configuration, metadata server for Workload Identity, Container Network Interface plugin, persistent volume CSI driver, and logging agents. Customers own application code, build files, container images, data, role-based access control and IAM policy, and the containers and pods they run. Customers must rotate cluster credentials, monitor clusters, keep logging enabled, and provide environmental details for troubleshooting. Google makes node operating system patches for Container-Optimized OS and Ubuntu available automatically when auto-upgrade or a release channel is used, but the operating system inside customer containers remains customer responsibility.

GKE upgrades preserve availability through redundancy, gradual rollout, and health-aware routing. A regional cluster distributes control plane components, nodes, and pods across zones so the control plane stays reachable when one zone fails. Compute Engine managed instance groups distribute virtual machines across zones, regional persistent disks replicate data to two zones, and load balancers with health checks send traffic only to healthy backends. Even when virtual machines remain in RUNNING state, crashed software is excluded by health checks. Capacity management forecasts load from user counts, request rates, trends, seasonality, and growth, then provisions with N+1 redundancy, benchmarks under normal and peak load, manages quotas, reserves capacity, and removes idle resources. Validation uses escalating resilience tests — introduce intermittent failures, block dependencies, block all network communication, then terminate hosts — plus load tests and blameless postmortems with Cloud Build.

Progressive deployment changes production in small, monitored stages to limit blast radius and preserve quick revert. Each expansion is gated by monitoring evidence. Availability targets shape launch scope, though increasing nines also increases complexity and cost:

These percentages are design targets, not guarantees, since service SLAs can differ. A launch plan should select dependencies whose designed-for targets exceed the application reliability goal.

2.2Capacity planning (e.g., quotas, limits, reservations, Dynamic Workload Scheduler)

Capacity planning ensures forecast capacity remains available, reliable, and cost-effective through quotas, reservations, and queued provisioning. Dynamic Workload Scheduler (DWS) is a queued provisioning mechanism for constrained accelerators such as GPUs and TPUs. A workload submits a request for GPU-powered virtual machines to a dedicated pool, and Compute Engine provisions capacity when it becomes available instead of failing on stock-out, which suits batch, AI/ML, and high-performance computing workloads that tolerate delayed start for higher obtainability and dense placement.

Obtainability modes define how accelerator capacity is requested, guaranteed, and billed across Compute Engine, GKE, Cloud Batch, and Agent Platform. On-demand virtual machines consume capacity without advance arrangement but can fail during stock-outs. Spot virtual machines offer the deepest discounts but can be preempted at any time with a 30-second warning. Reservations provide assured exclusive capacity in a specific zone but require paying for the full period.

In GKE, flex-start provisions GPU, TPU, and H4D nodes through the same queue, while Cloud Batch uses flex-start virtual machines for GPU jobs. On-demand requests depend on on-demand quota, while Spot and flex-start depend on preemptible quota, and failed accelerator creates in GKE return a Quota exceeded error when matching quota is missing.

Clustered GPU consumption options in AI Hypercomputer control duration, density, and approval path. All clustered options place resources as close as possible to minimize network latency, with future reservations through an account team adding topology-aware scheduling. Flex-start and Spot support all GPU machine types except A4X Max and A4X, while future reservations cover clustered GPUs and A3 Edge types.

At the end of a calendar-mode or AI Hypercomputer reservation, Compute Engine deletes the reservation and stops or deletes consuming instances based on termination action. Standard future reservations differ because they require sufficient quota both at review and at start time, and review typically takes five days.

Accelerator quota restricts GPU and TPU use by type, quantity, region, and consumption option. Projects generally start without significant accelerator quota, quota must be requested separately for each consumption option, and nodes provisioned by custom ComputeClasses in Autopilot still consume project quota. Free Trial accounts cannot obtain meaningful increases and must upgrade to paid. Workload characteristics determine fit:

• Training workloads need high-performance memory and have well-defined lifespan, favoring future reservations or flex-start.
• Inference workloads prioritize scalability and lower cost but need large memory during spikes, favoring on-demand or Spot with autoscaling.
• Implementation phase from proof of concept to production determines commitment length.
• Time to provision determines immediacy, where only on-demand, on-demand reservations, and Spot start immediately.

ComputeClasses orchestrate options by expressing fallback priority. The control plane provisions nodes from the first available option, so a workload can prefer reserved capacity and fall back to flex-start or Spot without redeployment.

Allocation quotas and rate quotas control how much can exist and how fast APIs can be called. Allocation quotas are the maximum creatable resources such as regional GKE clusters or load balancers. Allocation quotas do not reset over time and are released only when the resource is released, while system limits are fixed values that cannot be changed. Rate quotas limit API requests per period, and quotas can be zonal, regional, or global. Quota monitoring compares current and forecast usage against limits. Capacity Planner extends Cloud Monitoring with historical and forecasted use of virtual machine instances, Persistent Disk volumes, and GPUs, plus reservations and proximity to limits. Operational checks use the Quotas page, gcloud compute project-info describe --project PROJECT_ID for project quotas, and gcloud compute regions describe for regional use.

Quota adjustment raises values through manual requests or automated submission. Quota adjuster proactively submits requests during steady growth. Most requests are evaluated by automated systems, and denied requests must be pursued through a support case or account team.

There are three primary ways to submit an increase: through the console for the fastest path, through a support case with Live chat, and directly with the account team when deeper analysis is needed. Quota exhaustion blocks consumption: a gcloud command outputs a quota exceeded error with exit code 1, while an API request returns HTTP 413 Request Entity Too Large. Quota does not guarantee resource availability, because zonal stock-out can still block creation even with available quota.

A Compute Engine reservation is a zone-specific guarantee for virtual machines with matching hardware and optional accelerators. The reservation holds vCPUs and memory plus GPUs, Local SSDs, TPUs, or H4D resources. When a matching virtual machine is created, no additional charge applies for reserved vCPUs, memory, GPUs, or Local SSDs, while unreserved attachments such as disks or IP addresses are billed separately. All reservation types share two boundaries: capacity is locked to a single zone, and reserved capacity cannot create f1-micro, g1-small, Flex-start, sole-tenant, Spot, or preemptible instances. An on-demand reservation attempts to reserve capacity immediately and can be consumed, modified, or deleted at any time. A future reservation requests capacity for a future period and automatically creates on-demand reservations at start time. Once approved and in PROVISIONING state, a standard future reservation cannot be cancelled and commits the owner to pay for the entire period.

Share type controls which projects can consume a reservation, while consumption settings control which virtual machines can consume it. Single-project reservations use LOCAL share type for the owner project only, while shared reservations use SPECIFIC_PROJECTS for selected consumer projects in the same organization. With automatic use, any matching virtual machine consumes the reservation; with specific targeting, only virtual machines that name the reservation consume it, which is required for A3 machine types or compact placement policies.

2.3Autoscaling (e.g., managed instance groups, Cloud Run, GKE)

Autoscaling maintains availability and efficiency by adding capacity when load rises and removing it when load falls. Managed instance group (MIG) autoscaling, Google Kubernetes Engine autoscaling, and Cloud Run elasticity each implement this pattern with different signals and boundaries. The common tradeoff is responsiveness against cost: lower utilization targets protect latency but run more capacity, while denser packing lowers cost but leaves less margin before saturation.

The Horizontal Pod Autoscaler (HPA) is the workload controller that adds or removes Pod replicas based on real-time demand. The HPA controller reads a metric, compares current value to target, calculates desired replicas bounded by minReplicas and maxReplicas, and updates the target Deployment. Deleting an HPA object does not revert replica count, so the Deployment remains at its existing scale until manually scaled with kubectl scale. HPA supports three metric families:

• CPU and memory utilization: calculated as percentage of Pod request, so Deployments must set requests to use averageUtilization targets.
• Load balancer requests-per-second: direct traffic signal that reacts faster than CPU, enabled through utilization-based load balancing and Performance HPA profile.
• Custom and external metrics: application signals such as active users, queue size, or GPU duty cycle exposed through a metrics pipeline.

Custom-metric autoscaling depends on an ordered pipeline where each stage feeds the next:

1. Export metrics to Cloud Monitoring: Managed Service for Prometheus scrapes workload via PodMonitoring, typically every 15 seconds, or system metrics emit duty cycle every 60 seconds.
2. Install the Custom Metrics Stackdriver Adapter: adapter makes the metric visible through custom or external metrics API.
3. Deploy the HPA resource: autoscaling/v2 HPA references Deployment in scaleTargetRef with averageUtilization or averageValue target.

Configure targets such as 70% or 80% to preserve headroom while replicas start. The algorithm applies a default 0.1 tolerance band to dampen oscillation, and autoscaling/v2 is required for modern features such as scaling to and from zero.

The Vertical Pod Autoscaler (VPA) is the workload controller that right-sizes containers by adjusting CPU and memory requests and limits from historical use. VPA detects when initial requests are too large or small, then re-provisions the Pod with corrected configuration, for example from 1 CPU and 16 GB to 4 CPUs and 64 GB. Because re-provisioning restarts the Pod, VPA trades placement stability for long-term efficiency. Keep VPA in Off recommendation-only mode for at least 24 hours and ideally one week, then set minimum and maximum bounds before enabling Initial or Auto. The Cluster Autoscaler is the infrastructure controller that adds or removes nodes in GKE Standard node pools based on scheduling simulation. When HPA creates Pods that cannot be scheduled, the Cluster Autoscaler provisions a new node. When nodes become underutilized, it moves Pods and terminates the empty node. The workload depends on the Cluster Autoscaler for placement, while the Cluster Autoscaler depends on PodDisruptionBudgets and scheduling constraints to scale down safely.

Node pool auto-creation extends the Cluster Autoscaler from scaling existing pools to creating new pools with specialized hardware. While the Cluster Autoscaler adds nodes of an existing type, auto-creation provisions pools whose shape or accelerators precisely fit pending Pods. This minimizes waste from placing small specialized Pods on oversized nodes. If demand is static, manually creating a node pool remains appropriate. Auto-creation is triggered by hardware availability, not CPU load. A trigger occurs when a Pod requests specific hardware and no node can satisfy it, leaving the Pod in Pending, or when a Pod requests a ComputeClass such as cloud.google.com/compute-class: premium-gpu and no node provides those capabilities. A ComputeClass defines a prioritized list of rules, separating the Pod request from provisioning.

Autoscaler configuration for a MIG automatically adds virtual machines when load rises and deletes them when load falls. A MIG is a collection of virtual machines from a common instance template, and only managed groups can attach an autoscaler. The autoscaler is a separate API resource — autoscalers for zonal MIGs and regionAutoscalers for regional MIGs — that acts on the target MIG using the Compute Engine Service Agent. An autoscaling policy defines one or more scaling signals plus minimum and maximum replicas. When multiple signals are configured, the autoscaler calculates a recommended count for each signal and sets group size to the largest recommendation. A policy must contain at least one signal, and enabling autoscaling adds a CPU utilization signal by default. CPU-based autoscaling maintains average utilization near a target such as 0.60 by adding virtual machines when usage exceeds target and removing them when below. Data flows from each virtual machine CPU usage to the autoscaler, which averages usage and changes MIG size. Avoid targets at 85% or higher for slow-initializing applications because existing CPUs can overload while new machines start. During initialization period coolDownPeriodSec, default 60 seconds, metrics from starting machines are ignored. During stabilization period stabilizationPeriodSec, default 600 seconds and allowed 0 to 3600 seconds, scale-in deletions are delayed to prevent thrashing.

Cloud Monitoring metric scaling adjusts capacity from application measurements such as latency, network bytes, memory, disk operations, or queue length. Scaling is limited to up to 5 Monitoring metrics per MIG, only INT64 or DOUBLE value types, and the same metric cannot appear twice in one policy. If the metric originates from each virtual machine, the autoscaler averages values and requires at least one running machine to publish data, so the MIG cannot scale to zero.

Pub/Sub-driven autoscaling uses single instance assignment where data flows from the Pub/Sub subscription to Cloud Monitoring to the autoscaler to the MIG. The autoscaler divides pubsub.googleapis.com/subscription/num_undelivered_messages filtered by subscription ID by messages-per-virtual-machine. Use num_undelivered_messages because it counts queued plus in-flight unacknowledged messages, while a metric excluding in-flight work can drop to zero during processing and trigger premature scale-in. Predictive autoscaling scales out ahead of forecasted load for applications with long startup and predictable daily or weekly cycles. With prediction enabled, the autoscaler combines real-time CPU data with history-based forecasts to start machines early. Predictive autoscaling works only with CPU utilization, requires 3 days of CPU history, covers daily and weekly patterns but not monthly, annual, one-time, or sub-10-minute patterns, and increases cost because it runs capacity earlier.

Cloud Run is an inherently elastic serverless container service that scales instances automatically based on incoming requests. Traffic flows from clients to the Cloud Run service, and the autoscaler adds container instances as concurrent demand rises and removes them as demand falls, scaling to zero when minimum instances is 0. Elasticity is bounded by minimum and maximum instance counts, shaped per-instance by concurrency, and fine-tuned by CPU and concurrency utilization targets. Request-driven scaling keeps average load near configured targets while below maximum, and maximum instances act as a hard ceiling for cost control. Cloud Run enforces the lower of service-level scaling.max_instance_count and revision-level template.scaling.max_instance_count. For minimum instances, revision-level template.scaling.min_instance_count overrides service-level default when traffic is directed to a specific revision. Concurrency, configured as template.maxInstanceRequestConcurrency, is the maximum simultaneous requests per instance, with a default of 80. Starting at 80 and load-testing downward maximizes utilization, while reusing a lower value such as 10 under-utilizes instances and increases cost. Custom scaling controls define thresholds as template.scaling.cpuUtilization and template.scaling.concurrencyUtilization. By default the autoscaler maintains about 60% CPU and 60% of maximum concurrency per instance. Lower targets scale out earlier to protect latency but run more instances, while higher targets pack more work per instance but leave less margin.

Choose request-based billing when work happens strictly inside the request path, and instance-based billing when background threads must run between requests. Manual scaling with scalingMode: MANUAL and manualInstanceCount: N disables autoscaling and runs a fixed pool that does not respond to bursts. Automatic scaling follows an ordered lifecycle where first a cold start provisions a container, then the instance serves concurrent requests, then it remains idle up to 15 minutes, and only after that window is it reclaimed, with no configurable cooldown period.

Mitigating incident impact on users is the combination of draining traffic from unhealthy backends, adding capacity for surges, and rolling back failed changes without losing data. Cloud Load Balancing provides the control plane through URL maps, backend services, and health checks. Traffic flows from automation to URL maps, from URL maps to backend services, and from backend services to specific backends, so each stage can be advanced, paused, or reversed.

3.1Draining/redirecting traffic

Draining and redirecting traffic is the controlled removal of backends from serving and the shifting of user requests to healthy backends during deployments and incidents. Cloud Load Balancing provides the control plane for this work through URL maps, backend services, and health checks. Traffic flows from pipeline automation to URL maps, from URL maps to backend services, and from backend services to specific backends, so each stage can be advanced, paused, or reversed without dropping in-flight work.

Automated drainage makes traffic shifts a programmed pipeline stage instead of a manual cutover. A Compute Engine virtual machine (VM) can remain in the RUNNING state while its application has crashed, so autoscaling alone cannot guarantee responsiveness. Health checks probe each backend independently, traffic flows from the load balancer only to backends that respond successfully, and traffic stops flowing to backends that fail probes. When backends are managed instance groups (MIGs), autohealing adds a second flow where the MIG depends on its health check to identify unavailable VMs, deletes those VMs, and creates replacements so later pipeline stages have healthy targets.

Weight-based splitting is the mechanism for canary releases, A/B testing, and service migration on Application Load Balancers. The pipeline defines percentage-based splits across multiple backend services, so a new version receives a small share of live requests while the previous version serves the majority. Because the split is configuration, the pipeline can expand percentages after validation or reverse them instantly for rollback. A typical automated canary preserves order and validation gates:

1. Deploy new backend service: Bring up the new version alongside the previous version without sending it user traffic yet.
2. Split live traffic: Traffic flows from the load balancer to both backend services, for example 95% to the previous version and 5% to the new version.
3. Validate and expand: Only after health checks stay passing and canary analysis succeeds, gradually shift weights toward the new version.
4. Complete or roll back: Complete drainage when 100% reaches the new version, or reverse weights to the previous version when validation fails.

Traffic steering, actions, and policies drain targeted subsets. Steering routes by HTTP parameters such as host, path, and headers, so requests with user-agent:Mobile flow to mobile instances while other requests flow to defaults. Actions perform request or response operations such as redirects and header transformations, while policies fine-tune balancing behavior such as algorithms and request mirroring to a recording service.

Connection draining gracefully removes a backend so in-flight requests complete before shutdown. When an instance is removed from an instance group or an endpoint is removed from a zonal network endpoint group (zonal NEG), the backend service stops sending new connections to that backend while existing connections finish within a connection draining timeout of 0 to 3600 seconds. If no timeout is set, or the timeout is 0, existing connections end as quickly as possible. The timeout is set on the backend service, and when one instance group or zonal NEG is shared by multiple backend services, the largest timeout becomes the effective timeout. Drainage behavior depends on whether the load balancer proxies the connection or only routes packets:

A passthrough Network Load Balancer cannot send RST or FIN itself. After its tracking entry is removed, closure depends on the newly selected backend sending an RST and the client reconnecting through normal backend selection.

Global-to-regional failover protects a global external Application Load Balancer with regional external Application Load Balancers and Cloud DNS. The primary is the global external Application Load Balancer that handles traffic normally, and each backup is a regional external Application Load Balancer that receives traffic when the primary fails health checks. Failover occurs when Cloud DNS routes traffic from primary to backups, while failback reverses routing after health checks pass. Regional load balancers work best as targets because they are self-contained within a region and isolated from global infrastructure in the same region.

Graceful degradation keeps remaining backends available by serving reduced performance or accuracy under high load, then resuming full function when load falls. For globally served applications, Premium Tier carries traffic from the internet entry point of presence (PoP) closest to the sender across the Google network to the backend, which matters when traffic suddenly shifts regions.

3.2Adding capacity

Adding capacity is the combination of autoscaling, traffic shedding, and quota planning that keeps critical user journeys healthy when demand surges or a region loses backends. Cloud Load Balancing distributes traffic to the closest healthy backend with capacity, local autoscalers grow Pods or virtual machines (VMs) first, and only overflow that exceeds local capacity moves farther away. The goal is to prioritize critical traffic and degrade or redirect the rest rather than letting every journey fail.

A MIG autoscaling policy is the set of signals that tells the MIG autoscaler how many VMs a managed instance group (MIG) needs. The autoscaler collects usage for each signal, compares actual utilization to target utilization, and adds VMs when load rises or deletes VMs when load falls. When a policy has multiple signals, the autoscaler calculates a recommended count for each signal and sets group size to the largest recommendation. A new configuration includes average CPU utilization by default, with a 60-second initialization period that ignores data from initializing VMs. MIG autoscaling supports four peer signal types that cover different load shapes:

• Average CPU utilization: maintains average CPU near a target such as 75%.
• HTTP(S) load balancing serving capacity: maintains a fraction of backend serving capacity defined in the load balancer backend service.
• Cloud Monitoring metrics: scales on built-in, Ops Agent, or custom application metrics.
• Schedules: scales on time-based schedules for predictable peaks.

Load-balancing-based scaling uses a balancing mode of UTILIZATION or RATE on the backend service. With UTILIZATION, the backend defines maximum backend utilization; with RATE, it defines target requests per second (RPS) per instance. For example, if capacity is 100 RPS per instance and the target is 0.8, the autoscaler holds load near 80 RPS per instance. Monitoring-based scaling uses utilization-target mode to hold a metric at a value, or single-instance-assignment to define work per VM. Pub/Sub queue-depth scaling must use pubsub.googleapis.com/subscription/num_undelivered_messages filtered to the subscription, because that metric counts unacknowledged messages including in-flight work and prevents scale-in while work is still running.

GKE scaling splits responsibility between Pod replicas, Pod size, and node provisioning. The Horizontal Pod Autoscaler (HPA) changes replica counts from real-time load such as CPU, memory, or requests per second. A custom-metric pipeline moves data in order from Pods to the controller: first a PodMonitoring resource directs Google Cloud Managed Service for Prometheus to scrape Pod endpoints, then the Custom Metrics Stackdriver Adapter exposes the Cloud Monitoring metric to Kubernetes, then a HorizontalPodAutoscaler object targets the Deployment with minReplicas, maxReplicas, and an AverageValue target such as queue size. GPU-aware HPA follows the same flow using the container/accelerator/duty_cycle system metric reported every 60 seconds.

Node creation is not triggered by CPU load; it is triggered by an unfulfilled hardware request. A Pod requests hardware directly such as a GPU type or through a ComputeClass such as cloud.google.com/compute-class: premium-gpu, and auto-creation builds a node pool that satisfies those rules. Under surge the HPA first adds replicas, then cluster autoscaling adds nodes when replicas are unschedulable, while the VPA independently corrects Pod size.

Service capacity is the implicit control that defines how much traffic a Service handles before the system autoscales or overflows. A common pattern sets max-rate-per-endpoint="10", meaning each Pod handles 10 RPS, so 2 replicas provide 20 RPS of capacity in that cluster. Traffic-based autoscaling grows replicas from the same signal with replicas = ceiling[ current traffic / ( averageValue * maxRatePerEndpoint )]. Overflow only occurs when all three conditions hold: a multi-cluster Gateway is used, the same Service runs across multiple clusters behind that Gateway, and capacity is exceeded in one cluster but not others. Excess then flows to the next closest region with capacity, while traffic within a region balances across zones in proportion to zone capacity. Weight is explicit and capacity is implicit. Weight fixes exact proportions from HTTPRoute weights regardless of utilization, health, or client location, while capacity indirectly shapes shares from per-endpoint rate, utilization, and locality.

For example, a multi-cluster Gateway serves store from us-west1 and europe-west1 with 20 RPS per cluster. If Europe receives 30 RPS, 20 RPS stay in europe-west1 and 10 RPS overflow to us-west1. GKE Gateway Service Extensions add custom logic such as header modification, custom routing, and graceful degradation that routes non-essential journeys to cached responses to preserve critical capacity.

Cloud Tasks rate controls prevent sudden dispatch surges from overloading queues and downstream targets such as App Engine and Datastore. New or idle queues do not immediately dispatch at maxDispatchesPerSecond; dispatch ramps gradually toward the configured rate, and spiky traffic after long idle periods shows the ramp again as expected protection. For any queue dispatching more than 500 transactions per second (TPS), follow the 500/50/5 pattern: start at or below 500 TPS and increase by no more than 50% every 5 minutes. The pattern applies to the queue group as a whole when creating high-TPS queues, expanding ranges such as queue0000-queue0199 to queue0000-queue0399, launching releases that increase queue traffic, or resuming paused queues with large backlogs. App Engine traffic splitting applies the same ramp by starting with 1% to new queues and growing 50% every 5 minutes, while emergency expansion interleaves new queue names evenly in lexicographic order with no more than 50% new queues and each queue below 500 TPS.

Google Cloud quotas are per-project limits that prevent a few projects from monopolizing shared capacity. Resource quotas limit how many resources can be created, while rate quotas limit API calls in a period. Quotas apply at zonal, regional, or global scope, so exhaustion in one zone can block scaling there even when other locations have free capacity. Both types must be reviewed before peak load because autoscaling fails if either the VM count or the creation API rate is capped. Proactive management forecasts need, watches consumption with Cloud Monitoring and load tests, and requests increases before an incident. Capacity Planner adds historical and forecasted use for VMs, Persistent Disk, and GPUs with reservation status and proximity to limits. The quota adjuster proactively requests increases during steady growth, while manual increases go through the console, a support case, or the account team. Denied console requests can be retried through support or the account team, but that retry is too slow during a critical incident. Reserved compute capacity assures Compute Engine capacity in a specific zone, but an on-demand reservation succeeds only if capacity is available at request time, so it must be created before the surge. N+1 redundancy with three zones per region, at least two regions, and overprovisioning by a third lets the loss of one zone be absorbed without waiting for new physical capacity. Regional resources fail over across zones automatically, while multi-regional resources such as Cloud Storage and Spanner resist region loss, with traffic flowing from Cloud DNS and load balancing to healthy regional backends.

3.3Rollback strategies

Rollback strategies define when a failed deployment must be reversed, how far the reversal reaches, and how the prior state is restored without losing data. Recovery time objective (RTO) answers how long after a disaster before service is running, while recovery point objective (RPO) answers how much data can be lost. A rollback is necessary when monitoring shows the new version cannot meet those objectives without prolonged degradation.

Rollback triggers combine system health with business impact tolerance. System signals include automated consistency checks after each backup, backup health tracked by the Google Cloud Backup and DR service, adherence to backup intervals measured against RPO, and replication status and lag between primary and secondary environments. User signals include application performance degradation, failed synthetic scenarios in the recovered environment, and aggregate availability loss calculated as tier1_availability × tier2_availability × tierN_availability. Cloud Monitoring turns RPO into thresholds by alerting when backup intervals exceed the objective; if RPO is 15 minutes, backups must run at least every 15 minutes. Restoration testing must validate the full stack with restored data, not only the database, and production cutover must wait until both forward migration and fallback have been tested. Automated thresholds act fast without human judgment, including Eventarc or Workflows triggering remediation from audit logs, autoscaling adding capacity during failover, MIGs recreating failed VMs, and regional load balancing shifting traffic. Manual decisions apply when replication is asynchronous across regions, when failover is constrained by data sovereignty, or when a migration task exceeds its timeboxed expected duration from a dry run. Increasing availability from 99% to 99.9% reduces downtime from 864 seconds to 86 seconds per day but increases cost, complexity, and operational overhead.

Rollback scope defines exactly what returns to a prior state, and narrow scope reduces blast radius. The Cloud Healthcare API method fhirStores.rollback illustrates scoping: the point-in-time parameter is mandatory and must be within the last 21 days in RFC 3339 format to the second with time zone, such as 2017-01-01T00:00:00Z. Scope can narrow by operation type, by resource type such as Patient, by identifiers such as Patient/123 stored in Cloud Storage, or by filters on resource.meta. The caller needs healthcare.fhirStores.rollback through Healthcare FHIR Store Administrator, the service agent needs write access to the Cloud Storage output destination, the output destination is required, production calls must set force to true, and a dry run writes predictions to Cloud Storage without changing store resources. Fallback is the return to the source after cutover and must be treated as a full migration with planning and testing. Simple fallback switches applications back to the original source instance, which fits when downtime is acceptable or target transactions need not be preserved. When target transactions must be preserved, reverse replication synchronizes changes back to the source. A typical ordered recovery preserves integrity:

1. Create recovery state: Create resources from the deployment template, then apply the most recent backup and transaction logs from Cloud Storage.
2. Validate before traffic: Simulate user scenarios on the recovered environment with restored data.
3. Shift traffic: Configure Cloud DNS to point to the recovered web server or load balancer only after tests succeed.
4. Return when stable: Back up the recovery database, copy and apply it plus logs to production, block recovery connections, then point Cloud DNS back to production.
5. Restore protection: Verify copying to Cloud Storage and replication to standby, then delete the temporary deployment.

Automated rollback restores prior infrastructure and application state without manual rebuilding. Infrastructure as code (IaC) tools like Terraform define and version the last known good configuration, so recovery re-applies a prior revision rather than editing live resources. Eventarc observes audit logs to trigger remediation, Workflows orchestrates multi-step recovery in order, and autoscaling absorbs shifted traffic during failover. Google Cloud Deployment Manager restores infrastructure from a template that defines servers, networks, and firewall rules using versioned custom images that match production. For databases, the image startup script copies the latest backup from Cloud Storage and invokes restore, while Compute Engine snapshots roll back persistent disks from multi-region locations. Traffic shifts to the recovered environment only after restoration and testing succeed, and returning to production reverses the flow with backup, copy, apply, DNS switch, verification, and cleanup.

Post-rollback assessment confirms data is intact, recovery met time and loss targets, and procedures improve. Success is judged on data integrity of restored snapshots, RTO for resuming normal service, and RPO for recency of the restored point. Near-continuous options such as point-in-time recovery (PITR) for Cloud SQL or incremental backups for large datasets support tight RPOs, while active-active failover or cross-region replication improves RTO when backups alone are too slow. Operators must monitor open connections and request counts until draining completes, because pooled connections can still carry new requests to a draining VM and fail after deletion.

High availability (HA) generally means regional deployment with synchronous replication between zones and zero or near-zero RTO and RPO. Disaster recovery (DR) generally means multi-region deployment with typically asynchronous replication and non-zero RTO and RPO that require explicit planning and regular testing.

Key Facts

• A service level indicator is the technical measurement, a service level objective is the internal target, and a service level agreement is the legal commitment, with the internal SLO kept stricter than the external SLA to preserve a safety margin.
• A 99.9% SLO leaves a 0.1% error budget, so over 1,000,000 requests the service allows 1,000 failures, and burn alerts fire only when consumption is Above threshold with a retest window of at least 30 minutes.
• Every deleted managed service has a mandatory 30-day retention during which undelete can restore the service, while Dynamic Workload Scheduler with flex-start runs uninterrupted for up to seven days and Spot virtual machines can be preempted at any time.
• Connection draining timeout on a backend service is 0 to 3600 seconds where 0 disables draining, and a Proxy Network Load Balancer closes all existing TCP connections on a draining backend within 10 minutes.
• High availability uses synchronous replication within a single region for zero or near-zero RTO and RPO, while disaster recovery uses typically asynchronous replication across regions with non-zero RTO and RPO.