Prepare and test your skills

Prepare and test your skills

Site Reliability Engineering (SRE) and DevOps are related ways of working that keep cloud applications reliable while still releasing changes quickly. SRE applies engineering practices to operations, using clear reliability targets, automation, and learning from failures. DevOps brings development and operations teams together with shared processes and tools to build, deploy, and maintain applications, dependencies, and infrastructure.
What SRE and DevOps aim to achieve is shared responsibility for reliability between the people who build software and the people who run it. An application is reliable when it meets its objectives for availability and resilience to failure, where availability is the percentage of time an application is usable or the share of requests it serves successfully. Reliability in Google Cloud depends on application design, the secondary systems it depends on, the infrastructure resources it runs on, provisioned capacity and scaling, and the DevOps processes and tools used to maintain it. Observability guides this work by showing system health and where to act. Google Cloud Observability (opens in a new tab) provides integrated monitoring, logging, tracing, and profiling for applications on Google Cloud and beyond, including Cloud Monitoring for performance data and Cloud Logging for log data and events. Together with progressive rollouts, automated testing, and defined incident response, observability helps teams detect problems early, respond in a coordinated way, and improve over time.
Reducing toil means replacing repetitive manual operational work with automation so systems stay consistent and human error drops. Automated testing in continuous integration and continuous delivery (CI/CD) pipelines validates unit, integration, and end-to-end changes before production, which reduces errors and regressions that could cause disruptions. Progressive infrastructure changes follow the same idea: deploy to a few virtual machines (VMs) in one zone first, monitor the effect, revert quickly if there is a problem, and only then expand gradually. Automation is also built into the platform and into regular validation. Building blocks of reliability in Google Cloud (opens in a new tab) include hardware redundancy and replication, live migration of Compute Engine VMs during maintenance, geographically separated regions, and change management across the software lifecycle. Teams add regular performance and load testing to handle peak loads, find bottlenecks before they affect users, validate resilience with simulated failures, and gather data for capacity planning.
An error budget is the amount of failure or downtime a service is allowed while still meeting its reliability target. The target is usually expressed as a service level objective (SLO), such as 99.99% availability, also called 4 nines. For a 99.99% target, downtime must not exceed 8.64 seconds in a 24-hour period, or no more than ten failures for every 100,000 requests received. Error budgets make the tradeoff between stability and innovation explicit. As long as budget remains, teams can release new features, while a depleted budget signals a need to pause releases and focus on reliability work. The choice of deployment location directly affects the available budget because spreading across more failure domains supports higher availability:
| Deployment location | Availability target | Estimated maximum downtime |
|---|---|---|
| Single zone | 99.9% | 43.2 minutes in a 30-day month |
| Multiple zones in a region | 99.99% | 4.3 minutes in a 30-day month |
| Multiple regions | 99.999% | 26 seconds in a 30-day month |
Specific service Service Level Agreements (SLAs) can differ from these targets based on configuration, such as database distribution and routing policy. The strategic rule remains the same: define the SLO first, calculate the error budget from it, and use the remaining budget to decide whether to move fast or stabilize.
Exam tip: 99.99% availability allows only 8.64 seconds of downtime per day, or 10 failed requests per 100,000 requests — the error budget is what is left before that limit is reached.
A blameless post-mortem is a review after an incident that focuses on fixing systems and processes rather than blaming individuals. Well-defined and regularly tested incident management protects performance, security, reliability, and customer trust, and it supports business continuity and system stability. Google Cloud supports this learning loop with the Service Health Dashboard for current status, a history of incidents with duration and affected zones and services, and Personalized Service Health for project-specific incidents and alerts. Effective incident response depends on clear roles and documented steps that every team member can follow under pressure. Established procedures define responsibilities for incident commanders, investigators, communicators, and technical experts, along with communication protocols and escalation paths so information is shared promptly. The documented runbook or playbook then guides the team through an ordered response:
Regular training and repeated testing keep these procedures working during a real failure. Load testing with realistic traffic, stress testing to system limits, chaos or simulated-failure testing, and automated checks in CI/CD pipelines all validate that applications recover gracefully and maintain high availability.
Reliability metrics describe how well a service performs and how much performance is promised. Three terms work together: a service-level indicator (SLI) measures performance, a service-level objective (SLO) sets a target for that measurement, and a service-level agreement (SLA) is the uptime commitment Google makes for a service. The gap between an SLO and perfect performance creates an error budget, which is the amount of failure allowed before the target is missed.
A service-level indicator (SLI) is a measurement of performance for one aspect of a service. Cloud Monitoring (opens in a new tab) collects performance metrics from service infrastructure, such as request count for HTTP requests per minute that return 2xx or 5xx responses and response latency for HTTP 2xx responses. These metrics become the basis of SLIs. Standard SLIs are built by turning a metric into a ratio. An availability SLI is the ratio of successful responses to all responses. A latency SLI is the ratio of calls below a latency threshold to all calls. For some service types — Cloud Service Mesh, Istio on Google Kubernetes Engine, and App Engine — useful SLIs are known and performance metrics are identified automatically, and you can also define your own service type and choose its metrics.
Custom SLIs generally fall into two categories:
Google Cloud groups services that expose metrics into request-response services such as Cloud Run, Cloud Run functions, App Engine, and Google Kubernetes Engine, data storage and retrieval services such as Cloud Storage, Bigtable, Spanner, and Datastore, and data-processing services such as Dataflow and Managed Service for Apache Spark. Availability and latency are the primary SLI focus for storage and retrieval services, though latency for a database query often depends on how much data is retrieved and varies with application workload.
A service-level objective (SLO) is a target value for an SLI, measured over a period of time, that defines what counts as good service. The service determines which SLIs are available, and you specify SLOs based on them. You can create up to 500 SLOs for each service in Cloud Monitoring. An SLO is built from three pieces of information:
For example, an SLO might require that latency exceed 300 ms in only 5 percent of requests over a rolling 30-day period, or that the system hold 99% availability measured over a calendar week. SLOs are usually described as a number of nines: 99% is two nines, 99.9% is three nines, and so on. The highest value you can set is 99.9%, and any lower value appropriate to the service is allowed. Alerting policies are typically used to monitor SLO compliance, since a change in compliance can signal the onset of a failure early enough to fix the problem before it cascades.
An error budget starts at 1 minus the SLO and declines as actual performance misses the SLO. The error budget represents the degree to which a service is allowed to fall short of its objective during the compliance period. A tighter SLO leaves a smaller error budget, and a looser SLO leaves a larger one. Because the error budget shrinks as failures accumulate, it acts as a risk signal. Alerting on the rate at which the budget is consumed, called the burn rate, warns teams before the budget is exhausted. This is why targeting less than 100% is useful — the remaining percentage is the room the service has to fail without breaking its promise.
A service-level agreement (SLA) is the uptime commitment Google makes for a specific Google Cloud service, and it can differ from general platform availability targets. For example, the uptime SLA for a Bigtable instance depends on the number of clusters, how those clusters are distributed across locations, and the routing policy configured for the instance. The Bigtable case shows how configuration changes the commitment. A Bigtable instance with clusters in three or more regions has a minimum uptime SLA of 99.999% when the multi-cluster routing policy is configured. If the single-cluster routing policy is configured instead, the minimum uptime SLA is 99.9% regardless of how many clusters exist or where they are located.
Comparing SLIs, SLOs, and SLAs shows how measurement, target, and commitment fit together. An SLI is a raw measurement, an SLO is an internal target you choose for that measurement, and an SLA is the contractual uptime commitment attached to a service.
A concept map showing that an SLI measures service performance, an SLO sets an internal target for that SLI over a compliance period, the gap between the SLO and 100% becomes the error budget, and an SLA is Google's separate contractual uptime commitment for a service.
| Metric | What it represents | Who sets it | Example |
|---|---|---|---|
| SLI | A measurement of performance | Derived from service metrics | Ratio of successful responses to all responses |
| SLO | A target value for an SLI over a compliance period | You specify it for your service | 99% availability over a calendar week |
| SLA | A contractual uptime commitment for a service | Google, per service | 99.999% minimum uptime for a multi-region Bigtable instance with multi-cluster routing |
The three metrics connect through the error budget: the SLO sets the target for the SLI, and the gap between that target and 100% becomes the allowed room for failure.
Exam tip: An SLI measures, an SLO targets, and an SLA commits. SLOs should target less than 100% because the gap between the SLO and 100% becomes the error budget.
Availability targets by deployment location rise as workloads spread across more failure domains. A failure domain is a resource or group of resources that can fail independently, such as a zone or a region. Distributing an application redundantly across failure domains produces a higher aggregate availability than any single failure domain provides.
| Deployment location | Availability (uptime) % | Estimated maximum downtime |
|---|---|---|
| Single zone | 3 nines: 99.9% | 43.2 minutes in a 30-day month |
| Multiple zones in a region | 4 nines: 99.99% | 4.3 minutes in a 30-day month |
| Multiple regions | 5 nines: 99.999% | 26 seconds in a 30-day month |
These percentages are targets, not guarantees, and the uptime SLA for a specific service can differ from them. The aggregate availability an application actually achieves depends on three factors: the Google Cloud SLAs of the resources in its stack, the redundancy built into those resources, and the depth of the stack — that is, how many dependent layers must all be working for the application to serve requests.
Observability is a comprehensive approach to collecting and analyzing telemetry data to understand the state of applications and their operating environment. In Google Cloud, the core observability pillars are metric data, log data, and trace data, and together these signals form telemetry data. Google Cloud Observability provides managed services for monitoring, logging, tracing, and profiling that collect, analyze, and correlate these signals.
| Pillar | What it records | Google Cloud service | Use it to |
|---|---|---|---|
| Metric data | Numeric health or performance measurements taken at regular intervals, such as CPU utilization and request latency | Cloud Monitoring | Detect unexpected changes, analyze patterns, anticipate resource needs, and alert |
| Log data | Timestamped entries that describe specific system or application events | Cloud Logging | Understand what happened in a specific component, search and analyze events, and create log-based metrics or alerts |
| Trace data | The path of a request across distributed application components, made of spans | Cloud Trace | Follow request flow, examine latency, and identify the root cause of an issue |
Logs describe events in one component, while traces connect a request across components. Use traces when you need to find where latency or failure originates across a distributed application.
Exam tip: Logs describe events in one component, while traces connect a request across components; use traces when you need to find where latency or failure originates across a distributed application.
Metric data is numeric data about health or performance that a system measures at regular intervals, such as CPU utilization and request latency. Unexpected changes in metric data can indicate an issue to investigate, and patterns over time can show usage trends and help anticipate resource needs. Cloud Monitoring (opens in a new tab) automatically collects and stores performance information for most Google Cloud services, and it can collect Prometheus metrics through Google Cloud Managed Service for Prometheus. Installing the Ops Agent on Compute Engine VMs lets Cloud Monitoring collect metrics and logs from applications and third-party applications. Cloud Monitoring answers questions about service load, website response correctness, service performance, and the health of an App Hub application. To be notified when a metric meets criteria you define, create an alerting policy that includes the people or groups to notify, using notification channels such as email, Cloud Mobile App, PagerDuty, and Slack. For example, an alerting policy can notify you when a VM's CPU utilization exceeds 80%.
Each notification includes information about the failure and a link to an incident. An incident is a persistent record used to troubleshoot the failure, and it typically lists the incident status, links to logs, a chart of recorded metric data, labels, and duration. Some Google Cloud services provide recommended alerts or a button on a chart that creates a preconfigured alerting policy, leaving you to specify the recipients. Alerting policies can be managed through the Google Cloud console, Cloud Monitoring API, Google Cloud CLI, or Terraform.
Cloud Monitoring also supports proactive monitoring and visualization. Synthetic monitors test availability, consistency, and performance of services, applications, web pages, and APIs, including uptime checks that probe HTTP, HTTPS, and TCP endpoints and a broken-link checker that crawls a web page for broken links. Google Cloud-managed dashboards are created automatically as resources are instantiated or applications are registered with App Hub, while custom dashboards can display metric charts and tables, log data and error groups, alerting policy charts, alert information, text, and events such as a reboot or crash. Metrics Explorer visualizes time-series data, compares current data to previous data, displays outliers and percentiles, shows multiple metrics, and lets you save charts to a custom dashboard.
Log data is a generated record of system or application activity over time. A log is a collection of timestamped log entries, and each entry describes a specific event. Log data often contains rich detail about what occurred in a specific part of an application, but it does not effectively show how a change in one component relates to activity in other components, and trace data bridges that gap. Cloud Logging is a real-time log-management system with storage, search, analysis, and monitoring support. Cloud Logging automatically collects log data from Google Cloud resources, and applications, on-premises resources, and resources from other clouds can send log data to Cloud Logging. For regulatory or security reasons, you can determine where log data is stored.
You can collect log data from applications you write by using a client library to instrument the application. It is not always necessary to instrument an application: for some configurations, the Ops Agent can send log data written to stdout or stderr to your Google Cloud project. You can also collect log data from third-party applications, such as nginx, by installing the Ops Agent and configuring it to write that application's log data to your project.
To view and analyze log data, use the Logs Explorer or Observability Analytics pages of the Google Cloud console. Logs Explorer lets you view individual log entries and find related entries, and it annotates entries when they are part of an error group or when trace data is available, and it is recommended for troubleshooting services and applications. Observability Analytics lets you query log data with SQL and find trends and patterns, such as computing the average latency for HTTP requests to a specific URL over time.
Cloud Logging can notify you when certain events occur in log data. A log-based alerting policy notifies you when a particular message appears in a log entry, which is useful for important but rare events such as an audit log entry or a deployment message. A log-based metric either counts the number of log entries that match a criterion or extracts information such as response times into histograms. You can monitor a log-based metric with an alerting policy to observe trends, receive notifications when values change unacceptably, and create charts.
Trace data represents the path of a request across the components of a distributed application. Each trace represents a single end-to-end operation, and traces are composed of spans, which are records for a single function or operation. Traces let you follow the flow of requests and examine latency data, which helps identify the root cause of an issue. For agentic applications, traces capture the actions that an agent performs, for example MCP calls. Cloud Trace finds performance bottlenecks in production, and distributed tracing identifies sources of latency issues in an implementation. Error Reporting identifies and understands application errors. Cloud Profiler continuously gathers performance information by using a low-impact CPU and heap profiling service.
Application Performance Monitoring (APM) monitors, diagnoses, and manages the performance, availability, and user experience of software applications, including agentic applications. An APM system typically provides dashboards that display telemetry data and services that monitor telemetry data, helping you identify failures. Application observability uses telemetry data to generate insights that help you understand application behavior. Application Monitoring creates dashboards that display telemetry data, AI resource metric data, and information such as open incidents. Agent observability refers to methods for understanding the internal state and behavior of software agents, especially AI-powered agents built using Large Language Models (LLMs). Because AI agents are non-deterministic and complex, observability is important for understanding, debugging, evaluating, and improving their performance, safety, and reliability.
Observability services collect, analyze, and correlate telemetry data such as log data, metric data, and trace data. Key observability services include:
For incident response, an alerting policy triggers a notification that links to an incident, and the incident record brings together status, logs, metric charts, labels, and duration for troubleshooting. Log-based alerting policies catch important but rare events, while log-based metrics with alerting policies catch trends, and synthetic monitors proactively test availability and performance. Well-defined incident management defines roles for incident commanders, investigators, communicators, and technical experts, communication protocols and escalation paths, runbook procedures for communication, triage, investigation, and resolution, and regular training. For infrastructure health, the Service Health Dashboard shows current status and incident history, while Personalized Service Health shows project-relevant incidents and lets you configure alerts.
A service-level indicator (SLI) is a measurement of performance such as the ratio of successful responses to all responses, a service-level objective (SLO) is a target value for that measurement over a compliance period such as 99 percent availability over a calendar week, and a service-level agreement (SLA) is the contractual uptime commitment Google makes for a service. The SLO sets the target for the SLI, and the gap between that target and 100 percent becomes the error budget allowed for failure.
An error budget is the amount of failure or downtime a service is allowed while still meeting its reliability target, starting at 1 minus the service-level objective (SLO) and declining as failures accumulate. As long as budget remains, teams can release new features, while a depleted budget signals a need to pause releases and focus on reliability work.
Use trace data from Cloud Trace when you need to find where latency or failure originates across a distributed application. Log data from Cloud Logging describes timestamped events in one specific component, while trace data records the path of a request across components as spans, so traces connect activity across components that logs alone do not show.
Spreading workloads across more failure domains raises the availability target from 99.9 percent for a single zone to 99.99 percent for multiple zones in a region to 99.999 percent for multiple regions. Estimated maximum downtime in a 30-day month drops from 43.2 minutes to 4.3 minutes to 26 seconds respectively, although specific service uptime commitments can differ based on configuration.
A retail company is modernizing its cloud operations by adopting Site Reliability Engineering (SRE) and DevOps principles. Currently, the engineering organization faces two major operational challenges:
Which SRE practices should the company implement to overcome these challenges?
Implement distributed tracing and maintain a centralized knowledge base
Establish standardized incident runbooks and perform fault injection testing
Design for graceful degradation and configure service-level indicators (SLIs)
Minimize toil through automation and conduct blameless postmortems
Last updated