Prepare and test your skills
Prepare and test your skills
Differentiating and aligning service level indicators (SLIs) (opens in a new tab), service level objectives (SLOs) (opens in a new tab), and service level agreements (SLAs) (opens in a new tab) means building a hierarchy from measurement to internal target to external promise. The SLI is the technical raw metric, the SLO is the operational internal target applied to that metric, and the SLA is the legal contractual commitment made to users. SLI data flows into SLO monitoring, and SLO compliance provides the buffer that ensures SLAs are met. The hierarchy only works when internal SLOs are deliberately stricter than external SLAs.
Service level indicators measure how well a service performs as a quantitative ratio of good events to total events over a time window. Common SLI promises include returning an HTTP 200 status code, responding to a gRPC request in under 100 ms, completing the Create Virtual Machine workflow successfully, serving data refreshed within the past 10 minutes, and starting a scheduled batch job within one minute of its scheduled time. In Google Cloud, this telemetry is collected with Cloud Monitoring, OpenTelemetry for metrics, traces, and logs, and Google Cloud Managed Service for Prometheus for scalable metrics. Sources include the service itself on Google Kubernetes Engine (GKE) or Cloud Run, load balancer logs, application server logs, black-box probes that send synthetic requests, and browser code that reports timing back to a collection service.
An SLI specification defines what to measure, such as the percentage of home page requests that load in under 100 ms, while SLI implementations define how to measure it. Each implementation trades off fidelity for how accurately it captures user experience, coverage for what proportion of interactions are measured, and cost in money and engineering time. Measurement closer to the user improves fidelity but can add noise from the public internet, so separate SLIs with labeled buckets are often needed when read requests must be fast but write requests must be successful. The gap between the SLI value and 100% defines the error budget available for change velocity.
Service level objectives control reliability by setting an internal performance target for an SLI over a defined compliance period. Examples include maintaining 99.9% availability or ensuring 99.5% of payment transactions complete within 400 milliseconds over a rolling 30-day window. The gap between actual SLI performance and the SLO goal defines the error budget, and Cloud Monitoring tracks whether the system remains within that budget. If the service misses its SLO, the response is operational and internal, such as freezing releases and shifting development resources to resilience fixes.
Effective SLOs are specific, measurable, achievable, relevant, and time-bound. Specific means the required level of service is clearly stated, measurable means it is quantifiable through SLIs, achievable means it is attainable with available capabilities, relevant means it aligns with business goals, and time-bound means it has a defined window such as 30 days or a calendar month. SLO definitions also state explicit exclusions, such as invalid requests and planned maintenance windows, so the target reflects only valid service responsibility. Because SLOs are internal, they can be tightened without legal renegotiation.
Service level agreements require a formal, legally binding commitment that guarantees a minimum level of availability, performance, and support to users. Unlike an SLI, which is technical, or an SLO, which is operational, an SLA creates legal obligations and financial consequences for both provider and customer. SLAs must document the specific services provided, the expected level of service such as minimum uptime or maximum resolution time, the responsibilities of both provider and customer, and the penalties or remedies for noncompliance such as service credits. Once signed, the SLA is the commitment that SLI monitoring and SLO discipline must protect.
Penalties make the legal distinction concrete, such as a 5% monthly fee credit for each 0.1% drop below minimum availability, or a commitment to resolve a critical outage affecting more than 1% of transactions within two hours of detection. Validation of the SLA requirement confirms that the promised level is realistic before it is offered to clients. Because breach triggers financial or legal remedy, SLAs are set lower than internal SLOs to leave room for transient issues. Clear communication of SLOs and SLAs for every critical workload ensures engineering teams and customers share the same expectations.
Safety margins protect contractual commitments through a defensive hierarchy where SLI measurement supports the SLO target and the SLO target protects the SLA guarantee. The safety margin is the numerical gap between the stricter internal SLO and the looser external SLA for the same business capability. That gap absorbs normal variation and short incidents without creating a contractual breach. If transaction success slips from 99.98% toward 99.5%, engineers consume error budget and slow releases long before the 99.0% SLA is threatened.
| Business capability | Internal SLO target | External SLA commitment |
|---|---|---|
| Payment transaction success | 99.98% success over rolling 30 days, excluding invalid requests and planned maintenance | 99.0% of client transactions successfully processed and confirmed within one second |
| Payment processing latency | 99.5% of transactions processed within 400 milliseconds over rolling 30 days | Critical payment issues resolved within two hours of report or detection |
| Platform availability | 99.995% availability per calendar month, excluding scheduled maintenance | Minimum 99.9% availability per calendar month, with 5% fee credit per 0.1% drop |
The table shows that the provider aims internally for near-perfect performance while contractually promising a lower threshold that is still acceptable to the business. Technical precision at the SLI layer, operational discipline at the SLO layer, and legal caution at the SLA layer work together to balance velocity with reliability.
Exam tip: Keep the SLO stricter than the SLA for the same capability, such as a 99.995% internal availability SLO protecting a 99.9% contractual SLA, so internal action triggers before legal penalties apply.
Formulating and measuring service level indicators is the practice of defining quantifiable ratios of good valid events to total valid events that show how well a service meets user expectations. An SLI produces a number from 0% to 100% over a time window, and that number provides the raw data for evaluating reliability and calculating error budgets. In Google Cloud, SLIs are monitored with Cloud Monitoring (opens in a new tab) using metrics from the service itself, from Cloud Load Balancing (opens in a new tab), and from open telemetry pipelines.
Promise-based SLI ratios count how many user-facing promises were kept out of all valid promises made during a window. The formula is (number of successful valid events ÷ total number of valid events) × 100, measured over a rolling window such as 5 minutes, 1 hour, or 30 days. Filtering out invalid events, such as client-caused 4xx errors or planned maintenance, keeps the SLI focused on service behavior. The remaining gap to 100% defines the error budget for the corresponding objective.
Promise-based SLIs apply across diverse workloads because any user expectation can be expressed as a promise:
200 response to a customer request.A payment platform shows how the same pattern covers availability, latency, and success. Transaction success is successful transactions divided by total valid transactions, processing latency is the share of transactions confirmed within 400 ms, and platform availability is (total operational time − downtime) ÷ total operational time × 100.
An SLI specification defines what to measure without stating how to measure it. A specification example is the percentage of home page requests that load in under 100 ms. An SLI implementation is the technical method used to collect that measurement, and one specification can have many implementations with different tradeoffs.
The same page-loading specification can be implemented in several ways:
Each implementation trades off three characteristics: fidelity, or how accurately it captures user experience; coverage, or what proportion of user interactions are measured; and cost, or the money and engineering time to build and maintain it. Fidelity usually improves when measurement moves closer to the user, so browser code reflects perceived latency better than server logs. The tradeoff is that browser data also includes the user connection over the public internet, which varies by geography and network conditions, so the browser signal is a good proxy for user happiness but may not isolate a fixable service problem.
Load balancer telemetry provides SLI data from the edge of the Google Cloud application without requiring custom code in every case. SLI data flows from sources such as Google Kubernetes Engine (GKE), Cloud Run, load balancer logs, OpenTelemetry metrics, traces, and logs, and Google Cloud Managed Service for Prometheus into Cloud Monitoring, where ratios and distributions are evaluated against objectives. Because Cloud Load Balancing is often the first entry point for Google Cloud applications, its traffic, availability, and latency metrics are an excellent SLI source.
For non-UDP request-driven services, availability is expressed as a request-based ratio of good requests to total requests using a TimeSeriesRatio. An External Application Load Balancer writes to Cloud Monitoring under the https_lb_rule monitored resource with metric type loadbalancing.googleapis.com/https/request_count, filtered by the response_code_class label. The total filter often excludes 400 class responses because 4xx errors indicate client errors rather than service failure, while the good filter counts only 200 class responses. For multi-backend traffic, the metric loadbalancing.googleapis.com/https/backend_request_count filtered by the backend_target_name resource label measures availability for one specific backend target behind the same load balancer.
Latency is expressed with a distributionCut that counts the share of requests whose latency falls inside a range such as 0 to 100 ms. The two primary External Application Load Balancer latency metrics differ by where measurement starts and ends:
| Metric | Measures from-to | Choose when |
|---|---|---|
| https/total_latencies | From the proxy receiving the request until the proxy receives ACK from the client for the last response byte | Overall user experience is the primary concern |
| https/backend_latencies | From the proxy sending the request to the backend until the proxy receives the last response byte from the backend | Latency of a specific backend behind the load balancer must be isolated |
Both latency metrics have value type DISTRIBUTION, meaning values arrive as histograms with bucket boundaries built from powers of 1.4. A single request-based latency objective cannot capture both typical performance and tail degradation, so teams pair a typical-performance objective with a tail objective such as 99.9% of requests under 1000 ms over a rolling 1-hour window.
Exam tip: Use https/total_latencies for end-to-end user experience and https/backend_latencies for backend-specific latency; only the backend metric isolates one backend target.
Windowed SLIs divide the compliance period into short measurement windows and compute the ratio of good windows to total windows. An example window-based SLI is that the 95th percentile latency is less than 100 ms for at least 99% of one-minute windows over a rolling 28-day period. A window-based SLI must be used when the raw metric is already bucketed into time periods and expressed as percentile groups such as p50, p90, p95, and p99, because each value then states that a percentage of requests in that minute completed within the stated time.
Bucketing uses metric labels to classify values into separate SLIs so different expectations do not hide each other. Services handling both reads and writes need separate buckets because reads must be fast while writes must be successful. Services with one task but different outcomes also need separate buckets because users tolerate fast errors that can be retried, tolerate slow successes less, and tolerate slow errors least, so the latency SLI must distinguish successful from unsuccessful requests.
Designing service level objectives and managing error budgets is the practice of setting an internal performance target for a service and using the allowed amount of failure to control how fast changes can ship. A service level indicator provides the raw measurement of performance, the SLO states the target value for that SLI over time, and the error budget is the gap between perfect performance and the SLO goal. When SLI data shows the budget is being consumed, the team must shift effort from new features to reliability work until compliance recovers.
Achievable SLO targets are internal goals that reflect what users expect while remaining attainable within organizational capabilities and resources. An SLO must define a specific level of service and performance, such as 99.9% of requests with latency less than 200 milliseconds, and include a defined timeframe such as a rolling 30-day window or a calendar month. Targets that are stricter than user expectations or platform capability create constant violations, while targets looser than a contractual service level agreement leave no margin for operational variation. Well-designed SLOs therefore sit between user tolerance and the formal SLA commitment.
SLOs must be specific, measurable, achievable, relevant, and time-bound (SMART):
An SLO such as 99.9% availability or average response time less than 200 ms meets these criteria because the SLI data can be collected continuously and evaluated against the stated window. Clearly defining and communicating these SLOs for all critical services ensures that engineering, operations, and business owners share the same definition of acceptable reliability.
Error budgets govern release velocity by defining how much failure or slowness an SLO permits before remedial action is required. Cloud Service Mesh (opens in a new tab) uses the Goal percentage entered for an SLO to calculate this budget, so a 99.9% goal leaves a 0.1% budget for failed or slow responses within the compliance period. SLI data is monitored against the SLO to determine whether the system remains within budget and whether the related SLA will continue to be met. As long as budget remains, the team can spend it on releases and experiments that carry risk.
When SLI data shows the service is not meeting its SLOs, the error budget policy requires internal remedial action that directly reduces release velocity. The documented response is to focus development resources on fixing resilience issues rather than shipping new functionality until the service returns to compliance. This link between budget depletion and engineering prioritization is what balances change velocity with reliability, because repeated misses force reliability work to take precedence. Validating the SLA requirement alongside the SLO ensures the internal target is tight enough that normal budget consumption does not breach the formal contract with users.
SLI collection is the flow of telemetry from where users interact with the service to the system that evaluates SLO compliance. SLI data flows from the service itself where applications run, such as Google Kubernetes Engine (GKE) or Cloud Run, and from infrastructure components such as the load balancer that provide request logs. Cloud Monitoring (opens in a new tab) provides the comprehensive monitoring and observability backend that collects and analyzes metrics related to availability, performance, and latency from those sources. This dual-source approach ensures that both application behavior and edge-observed behavior inform the same SLO evaluation.
Two open standards and managed backends support this flow at scale in Google Cloud. OpenTelemetry (opens in a new tab) provides an open-source standard and set of technologies to capture metrics, traces, and logs, while Google Cloud Managed Service for Prometheus provides a fully managed, highly scalable backend for Prometheus metrics. By using these tools with Cloud Monitoring, SLI measurements such as success ratio over a rolling 5-minute window or average confirmation time in milliseconds can be tracked continuously over longer SLO windows. Alerts on these SLIs notify relevant teams proactively so budget burn is detected before the SLA is affected.
Creating SLOs in Cloud Service Mesh is the ordered configuration of an SLI type, a performance goal, and a compliance period for a specific service. The configuration starts from the Health tab for the selected service in the Google Cloud project and defines how good events are counted and over what period compliance is judged. Availability SLIs measure the ratio of successful responses to all responses, excluding requests that fail before they reach the Envoy sidecar proxy for the service due to networking or DNS failures. Latency SLIs measure the ratio of calls below a specified Latency Threshold in milliseconds to all calls.
The creation workflow proceeds in order:
After these steps, the SLO continuously compares SLI data against the goal for the full compliance period. A shorter period reacts faster but can be noisy, while a longer rolling 30-day or monthly period smooths variation and aligns better with SLA reporting. Choosing the period therefore trades sensitivity to recent incidents against stability of the reliability signal.
Windowed SLIs detect short violations by dividing the compliance period into small windows that expose brief spikes an overall average would hide. A windowed SLI is useful when short bursts of requests increase latency or errors for a few minutes but the service still meets the aggregate goal over days. The service divides SLO performance into increments during the compliance period, and each increment is judged good or bad against a goodness criterion. Only after enough windows are judged good does the service count as compliant.
A windowed SLI requires a goodness criterion and a duration in addition to the overall goal and period. The goodness criterion sets the threshold for good performance that a given window must meet to count as good, and the duration sets the length of each window. For example, an Availability SLO with a rolling 7-day period and a 99% goal can add a 95% goodness criterion with a 5-minute duration, which means 95% of all 5-minute windows over the last 7 days must each be available at least 99% of the time. This structure forces attention to repeated short outages even when total uptime looks acceptable.
Exam tip: An SLO is the stricter internal target, such as 99.995% availability per month, while the SLA is the contractual minimum, such as 99.9% availability with a 5% service credit for each 0.1% drop.
loadbalancing.googleapis.com/https/request_count with response_code_class filtering, while latency uses https/total_latencies for end-to-end user experience and https/backend_latencies to isolate a specific backend.A Service Level Indicator (SLI) is the technical raw metric expressed as a ratio of good events to total events, a Service Level Objective (SLO) is the internal operational target applied to that metric over a defined compliance period, and a Service Level Agreement (SLA) is the formal legally binding commitment with financial consequences such as service credits. SLI data flows into SLO monitoring, and SLO compliance provides the buffer that ensures SLAs are met.
The numerical gap between the stricter internal SLO and the looser external SLA creates a safety margin that absorbs normal variation and short incidents without creating a contractual breach. For example, an internal target such as 99.995% availability per calendar month protects a contractual commitment such as 99.9% availability, so engineers consume error budget and slow releases long before the SLA is threatened.
A promise-based SLI is calculated as the number of successful valid events divided by the total number of valid events times 100, measured over a rolling window such as 5 minutes, 1 hour, or 30 days. Invalid events such as client-caused 4xx errors and planned maintenance are filtered out, and the remaining gap to 100 percent defines the error budget for the corresponding objective.
Use https/total_latencies when overall user experience is the primary concern because it measures from the proxy receiving the request until the proxy receives ACK from the client for the last response byte. Use https/backend_latencies when the latency of a specific backend behind the load balancer must be isolated because it measures from the proxy sending the request to the backend until the proxy receives the last response byte from the backend.
An SRE team is configuring Service Level Objectives (SLOs) in Google Cloud Monitoring for two distinct workloads:
You need to define quantifiable, request-based Service Level Indicators (SLIs) in Cloud Monitoring that accurately represent the ratio of good events to valid total events for both workloads.
Which configuration approach should the SRE team use?
Last updated