Prepare and test your skills
Prepare and test your skills
Observability and Site Reliability Engineering (SRE) metrics are the practice of measuring how a cloud service behaves from the user's view and using those measurements to guide operations. Holistic observability connects reliability (opens in a new tab) goals, layered metrics, and continuous improvement into one loop. In Google Cloud, the work moves through three connected stages in order, where each stage depends on the previous one so scaling and maintenance become data-driven.
Service-level indicators (SLIs) (opens in a new tab) are measurements of performance that describe one aspect of good service. Cloud Monitoring collects the underlying metrics, such as request counts with 2xx or 5xx responses and response latencies for successful responses. For known service types like Cloud Service Mesh, Istio on Google Kubernetes Engine (GKE), and App Engine, useful metrics are identified automatically, while other workloads can define a custom service type and select its metrics.
Standard SLIs are ratios built from those metrics. An availability SLI is the ratio of successful responses to all responses, while a latency SLI is the ratio of calls below a latency threshold to all calls. Service-specific SLIs can also track AI behavior, such as success rate of user requests, time to first token (TTFT) and token streaming for large language models (LLMs), rate of irrelevant or harmful responses, and model FLOPS utilization (MFU) or tokens per second for training.
SLIs fall into two patterns that count goodness differently:
| SLI type | What is counted | Example of good service |
|---|---|---|
| Request-based | Atomic units of service | Number of successful HTTP requests out of all requests |
| Windows-based | Time periods where a criterion is met | Number of minutes where latency stays below a threshold |
Choose a request-based SLI when each request matters equally, and choose a windows-based SLI when sustained performance over time matters more than any single request.
Service-level objectives (SLOs) are target values for SLIs measured over a compliance period. An SLO in Cloud Monitoring combines the SLI that measures performance, a performance goal that states the desired level, and a compliance period that defines the measurement window. You can create up to 500 SLOs for each service, which lets one service track availability, latency, and quality separately.
Concrete SLO targets make reliability measurable from the user's perspective. Examples include 99.9% of API calls must return a successful response, 95th percentile inference latency must be below 300 ms, TTFT must be below 500 ms for 99% of requests, and rate of harmful output must be below 0.1%. Other requirements take the form of latency can exceed 300 ms in only 5 percent of requests over a rolling 30-day period, or the system must have 99% availability over a calendar week.
The compliance period controls how service monitoring (opens in a new tab) judges SLO compliance. A rolling period, such as the last 30 days, continuously drops old data and adds new data, so the window always ends at the present time. A calendar period, such as a week or month, measures from a fixed start to a fixed end and extrapolates expected traffic from elapsed time. Aligning each SLO with a business key performance indicator (KPI), such as satisfaction with a conversational assistant or revenue from recommendations, keeps engineering effort focused on behavior that affects users and the business.
Exam tip: A useful SLO targets less than 100% to leave an error budget, and the highest SLO value you can set in Cloud Monitoring is 99.9%.
An error budget is the amount of imperfect performance allowed by an SLO, starting at 1 minus the SLO and declining as actual performance misses the target. For example, a 99.9% SLO starts with a 0.1% error budget, and the budget drops as failed or slow requests consume it, triggering an SLO violation when the budget drops below 0. Changes in SLO compliance can signal the onset of failures, so alerting policies typically monitor burn rate to give warning before a problem cascades.
The error budget does not always move in only one direction. If a request-based SLO uses a calendar period and traffic increases, expected total traffic rises and remaining budget can rise because the system extrapolates from calls so far. If an SLO uses a rolling period and a past period of poor compliance rolls out of the window while the present is compliant, the budget can also rise, with a budget at or above 0 meaning compliant and below 0 meaning non-compliant. Monitoring burn rate with alerting policies turns these movements into decisions: fast burn pauses risky releases and triggers scaling or fixes, while slow burn allows feature work to continue.
Error-budget signals guide infrastructure scaling and maintenance when combined with saturation and traffic data. Saturation shows how close critical resources like CPU, memory, GPU, or TPU accelerators are to capacity limits, while traffic shows request volume and load patterns over time. Historical metrics stored beyond short-term retention help validate scalable and resilient apps (opens in a new tab) designs, plan capacity for peak events, and confirm that scaling handled past high-traffic periods without over-provisioning.
Signals and tools that feed observability are the layered metrics and Google Cloud services that supply reliable data to SLIs, SLOs, and scaling decisions. Infrastructure monitoring provides baseline health such as processor usage for CPU, GPU, and TPU, memory usage, network throughput and latency, and disk I/O for both managed environments and self-managed resources like GKE nodes and Cloud Run instances. App-level, service, and end-to-end monitoring add request flows between microservices and user-perceived availability, which can reveal errors or latency that infrastructure metrics alone miss.
The Google SRE practice organizes user-facing monitoring around four golden signals for operational excellence (opens in a new tab):
Averages can hide wide imbalances, so distributions such as the 95th or 99th percentile are often more informative than the mean for latency and health. Cloud Monitoring collects, stores, and visualizes these signals with pre-built and custom dashboards, and metrics from serving frameworks like vLLM or NVIDIA Triton Inference Server flow into Cloud Monitoring through Google Cloud Managed Service for Prometheus. Cloud Logging adds event and error context for troubleshooting, Cloud Trace follows request flows across distributed inference pipelines, and Cloud Profiler identifies bottlenecks inside function blocks so teams can optimize resource usage and execution time before scaling out.
Release management and deployment risk mitigation is the practice of validating every code and infrastructure change before it reaches production and recovering quickly if a release fails. This work combines continuous integration and continuous delivery (CI/CD) pipelines with isolated environments, health-aware traffic management, and automated recovery for reliability (opens in a new tab). Together these controls limit the blast radius of a bad release and protect system availability.
CI/CD validation means that automated tests run in the pipeline and a change is promoted only after it passes. Integrating automated testing into CI/CD pipelines validates application changes before deployment and significantly reduces the risk of errors and regressions that could cause operational disruptions. The same approach applies to infrastructure code, where a script can automatically run Terraform validate and plan commands on every commit to support operational excellence (opens in a new tab).
The pipeline typically validates a change through an ordered workflow:
Pipeline test coverage spans multiple peer levels:
Performance and load testing adds a further gate by simulating realistic traffic patterns and stressing the system to its limits. Results from Cloud Load Balancing simulations and load-testing services are used to adjust resource allocation, tune application configuration, and identify bottlenecks before release.
Isolated release environments are separate dev, QA, and staging instances where changes are tested without affecting production. Deploying a separate instance per Google Distributed Cloud (GDC) project allows teams to test changes in isolation, using the ResourceManager API to declaratively create and tear down Project resources such as ticketing-system-dev, ticketing-system-qa, and ticketing-system-staging. Combined with chart packaging and infrastructure managed as code, developers can iterate quickly alongside production and run regular regression testing for Compliance Manager frameworks (opens in a new tab).
Infrastructure as code (IaC) makes those environments repeatable and rollback-capable. Defining infrastructure with Terraform ensures consistency, enables rollbacks, and removes the risks of manual management. Teams organize Terraform configurations as modular templates, including Google-curated blueprints from Cluster Toolkit for GKE and high-performance clusters, and manage ongoing configuration declaratively with Config Connector manifests applied to a GKE cluster. Because infrastructure and application configuration live in version control with IAM policies and service accounts defined as code, any environment can be recreated consistently and a failed change can be reverted to a prior version.
Health-aware traffic management is the system that sends user traffic only to healthy instances during and after a deployment. A load balancer pulls health status from each instance through a configured health check, and only instances that pass the check continue to receive traffic. In Kubernetes, liveness and readiness probes help the orchestrator decide how to manage pods and requests, while load-balancer health checks can autoheal Compute Engine managed instance groups (MIGs) by recreating unhealthy virtual machines (VMs).
Resilience against zone failure comes from spreading redundant resources across locations. Regional MIGs and Google Kubernetes Engine (GKE) (opens in a new tab) regional clusters are resilient to zone outages, and critical topologies use at least two regions with three zones in each region. Traffic flows from Cloud DNS and regional or global load balancers across those zones and regions to evenly distribute load, and capacity is overprovisioned by a third more than the estimate so surviving zones can absorb failover traffic. Data-layer redundancy follows the same pattern through Cloud SQL high availability (HA) configurations, regional persistent disks with synchronous replication, and multi-region services like Spanner and Cloud Storage.
Aggregate availability explains why limiting deployment blast radius matters. The aggregate availability of chained tiers is the product of each tier, or tier1_availability x tier2_availability x tierN_availability. Four tiers at 99.9% each yield only 99.6% aggregate availability, and two 99% services chained together yield 98%. By contrast, parallelized services use 1 - (1 - A) x (1 - B), so two 99% services in parallel yield 99.99% because each runs independently. In availability terms, 99.9% allows about 86 seconds of downtime in 24 hours, while 99% allows 864 seconds, or 10 times more.
Automated rollback and recovery is the set of triggers and replication mechanisms that restore service without manual steps when a release or zone fails. Automation reduces human error and shortens recovery time by triggering remedial actions directly from observed signals. For example, Eventarc or Workflows can automatically trigger remediation in response to issues observed through audit logs, while autoscaling increases cloud capacity during failover to preserve AI and ML reliability (opens in a new tab).
The recovery path depends on whether the failure is contained in one region or spans regions:
| Approach | Scope and replication | Recovery objective |
|---|---|---|
| High availability (HA) | Regional deployment with synchronous replication between zones | Zero or near-zero recovery time objective (RTO) and recovery point objective (RPO) with active-active topology |
| Disaster recovery (DR) | Multi-region deployment with typically asynchronous replication | RTO and RPO depend on defined replication strategy plus manual or semi-automated failover |
For data loss, recovery depends on consistent, restorable backups tested in a non-production environment. Backups are scheduled to align with RPO, so a 15-minute RPO requires backups at least every 15 minutes, with near-continuous options like point-in-time recovery (PITR) for Cloud SQL for critical applications. Backup health is tracked with Google Cloud Backup and DR service, with replication across regions for resilience, and Cloud Monitoring alerts fire if backup intervals exceed the defined RPO. A full restoration test must include the entire application stack with restored data, judged by data integrity, RTO, and RPO.
Exam tip: HA applies to regional deployments with synchronous replication and zero or near-zero RTO and RPO, while DR applies to multi-region deployments with typically asynchronous replication and explicitly planned RTO and RPO.
Automated testing and continuous integration (CI) validation is the practice of automatically building, testing, and checking every code change before it is released. In Google Cloud, Cloud Build runs this validation when code is committed, then hands a tested container image to Cloud Deploy for progressive rollout under a defined deployment methodology (opens in a new tab). This early checking finds defects in code, container structure, and security while the change is still in the build, which protects code integrity through the rest of the pipeline.
The CI validation workflow is the ordered sequence that moves code from a commit to a deployable release. Data flows from the source repository through Cloud Build to Artifact Registry and then to Cloud Deploy, and each step depends on the success of the previous step for continuous integration and delivery to Google Kubernetes Engine (opens in a new tab):
A six-step process flow showing the CI validation workflow: a code commit triggers Cloud Build, which builds and pushes the image to Artifact Registry, runs automated tests, scans the container structure and vulnerabilities, signs the image with Binary Authorization, and creates a release in Cloud Deploy.
The workflow enforces a quality gate between build and delivery. The image digest created in Artifact Registry becomes the artifact that is promoted through environments, so the exact image that was tested is the image that is deployed.
Functional and structural testing in the build checks that code works correctly and that the container is assembled correctly. Validation happens in every stage of the pipeline, not only during code review or CI, with linters and static analyzers checking style and quality during review and automated tests running after the change is integrated. Teams that follow test-driven development gain more complete coverage because tests are written alongside each feature and Cloud Build automatically runs the tests, evaluates results, and notifies owners about build issues when they migrate to Google Cloud and deploy workloads (opens in a new tab).
Container Structure Tests validate the contents of the built image before it is approved:
These checks catch packaging errors that unit tests alone would miss, such as a missing config file or a wrong startup command. Because the checks run inside Cloud Build before approval, a broken image never reaches Binary Authorization signing or Cloud Deploy.
Security testing in the build identifies known weaknesses in code and in container images before deployment. Artifact Analysis vulnerability scanning automatically scans images pushed to the registry for known exposures from Common Vulnerabilities and Exposures (CVE) sources, and assigns each finding a severity of critical, high, medium, low, or minimal based on Common Vulnerability Scoring System (CVSS) scores. Web Security Scanner extends this to running web applications on App Engine, Compute Engine, and Google Kubernetes Engine (GKE) by checking for cross-site scripting, misconfigurations, and vulnerable resources, with scans run on demand, on a schedule, or automatically through APIs as part of the build pipeline.
A continuous vulnerability program keeps images safe over time under Assured Workloads frameworks (opens in a new tab):
Security results are visible in the Security insights panel for each build, which shows Artifact Analysis vulnerabilities and the build's level of security assurance under Supply-chain Levels for Software Artifacts (SLSA) guidelines. Testing for weaknesses such as unescaped inputs or raw input passed to SQL queries belongs in this same early stage, so insecure code is fixed before an image is promoted.
Binary Authorization is the Google Cloud service that provides software supply-chain security by allowing only trusted container images to deploy. The service uses policies and rules to define requirements, notes to store trusted metadata, attestations to certify that a required check passed, attestors to represent trusted approvers, and signers to cryptographically sign the attestation. A typical policy requires attestations for passed vulnerability scans, passed quality assurance testing, or product-owner sign-off, which is especially useful when images move from development to testing to production for PCI DSS compliance on GKE (opens in a new tab).
At deployment time, the Binary Authorization policy enforcer checks the image provenance before the container is allowed to deploy to GKE. If the image satisfies all required constraints and has signatures from all required attestors, the service allows the deployment; otherwise deployment is blocked until the image becomes compliant. In an emergency, an operator can bypass the policy with the breakglass workflow, and every breakglass use is recorded in Cloud Audit Logs for later review.
Exam tip: Vulnerability scanning detects issues in Cloud Build, but only Binary Authorization blocks a non-compliant image at deployment time.
Service-Level Indicators (SLIs) are measurements of performance that describe one aspect of good service as ratios such as successful responses to all responses for availability and calls below a latency threshold to all calls for latency, while Service-Level Objectives (SLOs) are target values for SLIs measured over a compliance period. An SLO in Cloud Monitoring combines the SLI that measures performance, a performance goal that states the desired level, and a compliance period that defines the measurement window, and you can create up to 500 SLOs for each service. A useful SLO targets less than 100 percent to leave an error budget, and the highest SLO value you can set in Cloud Monitoring is 99.9 percent.
Choose a request-based Service-Level Indicator (SLI) when each request matters equally, and choose a windows-based Service-Level Indicator (SLI) when sustained performance over time matters more than any single request. A request-based SLI counts atomic units of service such as the number of successful HTTP requests out of all requests, while a windows-based SLI counts time periods where a criterion is met such as the number of minutes where latency stays below a threshold.
An error budget is the amount of imperfect performance allowed by a Service-Level Objective (SLO), starting at 1 minus the SLO and declining as actual performance misses the target. Monitoring burn rate with alerting policies turns these movements into decisions where fast burn pauses risky releases and triggers scaling or fixes, while slow burn allows feature work to continue. Error-budget signals guide infrastructure scaling and maintenance when combined with saturation data showing how close critical resources are to capacity limits and traffic data showing request volume and load patterns.
High Availability (HA) is regional deployment with synchronous replication between zones for zero or near-zero recovery time objective (RTO) and recovery point objective (RPO) with active-active topology, while Disaster Recovery (DR) is multi-region deployment with typically asynchronous replication and explicitly planned recovery time objective (RTO) and recovery point objective (RPO). HA provides zero or near-zero recovery objectives within the region, while DR recovery depends on the defined replication strategy plus manual or semi-automated failover.
An enterprise runs a business-critical order processing microservice on Google Cloud. The Site Reliability Engineering (SRE) team has established a 30-day rolling Service Level Objective (SLO) target of 99.9% availability based on a Service Level Indicator (SLI) measuring successful requests.
During an aggressive deployment cycle, custom Cloud Monitoring dashboards indicate that 90% of the 30-day error budget was consumed in just 4 days due to transient server errors and latency spikes.
According to SRE best practices, how should the team use these SLI/SLO metrics to guide their upcoming operational and development decisions?
Halt non-critical feature releases and redirect engineering effort toward infrastructure reliability, performance tuning, and automated scaling fixes until the error budget stabilizes
Ignore the rolling error budget and evaluate infrastructure health exclusively using raw virtual machine CPU and memory utilization thresholds
Lower the SLO target from 99.9% to 95.0% for the remainder of the compliance window to prevent alerting policies from triggering during deployments
Switch the SLI calculation from request-based availability to a short 1-minute windowed SLI and reset the 30-day compliance period to zero
Last updated