SLIs, SLOs, Error Budgets and Reliability Tradeoffs

Defining service level indicators (SLIs), service level objectives (SLOs), and service level agreements (SLAs) creates a hierarchy from measurement to internal target to external promise. The SLI is the technical measurement, the SLO is the internal operational target for that measurement, and the SLA is the legal commitment to users. SLI data flows into SLO monitoring, and SLO compliance provides the buffer that protects SLA commitments.

1.1The Hierarchy from Measurement to Target to Promise

The hierarchy from measurement to target to promise links raw telemetry to operational control to contractual obligation. Cloud Monitoring (opens in a new tab) collects SLI telemetry from services, load balancers, and probes, Cloud Monitoring evaluates that telemetry against SLO goals over a compliance period, and SLO discipline protects the SLA guarantee. The hierarchy only works when the internal SLO is deliberately stricter than the external SLA for the same capability. If transaction success slips from 99.98% toward 99.5%, engineers consume error budget and slow releases long before a 99.0% SLA is threatened.

1.2Formulating Promise-Based SLIs and Choosing Implementations

Promise-based SLIs count how many user-facing promises were kept out of all valid promises made during a window. The formula is successful valid events divided by total valid events multiplied by 100, measured over a rolling window such as 5 minutes, 1 hour, or 30 days. Filtering out invalid events such as client-caused 4xx errors and planned maintenance keeps the SLI focused on service behavior, and the gap between the SLI value and 100% defines the error budget.

An SLI specification defines what to measure, such as the percentage of home page requests that load in under 100 ms, while an SLI implementation defines how to measure it. One specification can have many implementations with different tradeoffs in fidelity, coverage, and cost:

• Latency field in the application server request log.
• Metrics exported directly by the application server.
• Metrics exported by a load balancer in front of the application servers.
• Black-box monitoring service that sends synthetic requests and times valid responses.
• Application code running in the customer browser that records timing and sends it to a collection service.

Fidelity improves when measurement moves closer to the user, so browser code reflects perceived latency better than server logs. The tradeoff is that browser data also includes variation from the public internet, so browser signals are good proxies for user happiness but may not isolate a fixable service problem. Separate SLIs with labeled buckets are needed when read requests must be fast but write requests must be successful, and when fast errors, slow successes, and slow errors have different user tolerance.

1.3Measuring Availability and Latency from Load Balancer Telemetry

Load balancer telemetry provides SLI data from the edge of a Google Cloud application without requiring custom code in every service. SLI data flows from Google Kubernetes Engine (GKE), Cloud Run, load balancer logs, OpenTelemetry pipelines, and Google Cloud Managed Service for Prometheus into Cloud Monitoring, where ratios and distributions are evaluated against objectives. Because Cloud Load Balancing (opens in a new tab) is often the first entry point for requests, its traffic, availability, and latency metrics are an excellent SLI source.

For non-UDP request-driven services, availability is a request-based ratio of good requests to total requests using a TimeSeriesRatio. An External Application Load Balancer writes to Cloud Monitoring under the https_lb_rule monitored resource with metric type loadbalancing.googleapis.com/https/request_count, filtered by the response_code_class label. The total filter often excludes 400 class responses because 4xx errors indicate client errors, while the good filter counts only 200 class responses. For multi-backend traffic, loadbalancing.googleapis.com/https/backend_request_count filtered by backend_target_name measures availability for one specific backend target.

Latency is expressed with a distributionCut that counts the share of requests whose latency falls inside a range such as 0 to 100 ms. Both latency metrics have value type DISTRIBUTION, meaning values arrive as histograms with bucket boundaries built from powers of 1.4:

A single latency objective cannot capture both typical performance and tail degradation, so teams pair a typical-performance objective with a tail objective such as 99.9% of requests under 1000 ms over a rolling 1-hour window. Windowed SLIs further expose short violations by dividing the period into small windows, such as requiring that 95th percentile latency stay under 100 ms in 99% of one-minute windows over a rolling 28-day period.

1.4Setting SLO Targets and SLA Commitments with Safety Margins

Service level objectives control reliability by setting an internal performance target for an SLI over a defined compliance period. Examples include maintaining 99.9% availability or ensuring 99.5% of payment transactions complete within 400 milliseconds over a rolling 30-day window. Effective SLOs are specific, measurable, achievable, relevant, and time-bound, state explicit exclusions such as invalid requests and planned maintenance, and can be tightened without legal renegotiation. If the service misses its SLO, the response is operational and internal, such as freezing releases and shifting development resources to resilience fixes.

Service level agreements require a formal, legally binding commitment that guarantees minimum availability, performance, and support to users. SLAs must document the specific services provided, the expected level of service such as minimum uptime or maximum resolution time, the responsibilities of both provider and customer, and the penalties for noncompliance such as service credits. Examples include a 5% monthly fee credit for each 0.1% drop below minimum availability and a commitment to resolve a critical outage affecting more than 1% of transactions within two hours of detection. The safety margin is the numerical gap between the stricter internal SLO and the looser external SLA, such as a 99.995% internal availability SLO protecting a 99.9% contractual SLA, which absorbs normal variation without creating a breach.

Error budgets connect remaining allowable unreliability to decisions about release velocity and operational response. Cloud Service Mesh (opens in a new tab) acts as the control plane that distributes health, routing, and backend information to sidecar proxies or proxyless gRPC clients, while Cloud Monitoring collects the resulting request counts and connectivity state for budget tracking. When failed or slow requests consume the allowed budget, operators visualize that consumption per service, per region, and per backend to decide whether to allow more releases or freeze changes.

2.1How Error Budgets Govern Release Velocity

The relationship between SLIs, SLOs, and error budgets defines how reliability is measured, targeted, and governed. An SLI is the measured performance over time, such as the fraction of successful requests, an SLO is the target value for that SLI over a compliance period such as 99.9% in 30 days, and the error budget is the allowed shortfall from that target. A 99.9% SLO over 1,000,000 requests allows 1,000 failed requests as the total budget. If monitoring measures 99.5% compliance, or 5,000 failures, the service has consumed five times its budget, while 99.95% compliance consumes half the budget and leaves 500 failures remaining.

When measured performance meets or exceeds the SLO, budget is preserved and teams can continue releases at normal velocity. When performance falls below the SLO, budget is consumed in proportion to the gap and the policy signal requires slowing releases, freezing non-essential changes, and prioritizing reliability work until the budget recovers. Cloud Monitoring visualizes this balance with an SLO chart that shows health, compliance, absolute error budget, or fractional error budget for a configured service. A custom dashboard can show the status of all SLOs for a set of services, with up to 100 widgets per dashboard, and creating or modifying dashboards requires the Monitoring Editor role.

2.2Defining Burn Rate Alerts with Lookback Duration and Threshold

Error budget consumption measures how much allowed unreliability a service has used within a time window. A burn rate alert forecasts budget exhaustion by comparing consumption percentage against time, where the Lookback Duration field defines the window of recent SLO performance to evaluate and the Threshold field defines the unacceptable consumption percentage within that window. A short lookback with a high threshold detects fast burns that need immediate response, while a longer lookback with a lower threshold detects slow burns that will exhaust the budget before the period ends. The alert re-evaluates on a Retest window with a minimum value of 30 minutes, which prevents brief spikes from causing flapping while still catching sustained loss.

Creating the burn rate condition from the Cloud Service Mesh dashboard follows an ordered workflow:

1. Name the condition: Click the Suggested title link to use the name based on the SLO, or enter a custom name.
2. Set the evaluation window: Enter the lookback period in the Lookback Duration field in the Target section, or use the default value.
3. Set the exhaustion signal: Enter the consumption percentage in the Threshold field in the Configuration section with position Above threshold.
4. Save the condition: Click Save to return to the Create new alerting policy window with the condition populated.

2.3Turning Burn Conditions into Alerting Policies That Drive Action

An alerting policy in Cloud Monitoring connects the burn rate condition to notifications and runbooks that govern development and operations. The policy carries the condition forward from Cloud Service Mesh, applies trigger logic, and delivers the alert to the team that must act on the budget status. After the policy is saved, the workflow returns to the Cloud Service Mesh dashboard under Anthos Services for ongoing monitoring.

Completing the policy that enforces the error budget follows an ordered workflow:

1. Name the policy: Enter a policy name that identifies the service, SLO, and burn severity.
2. Confirm conditions: Keep the automatically populated condition, and optionally add another condition for multi-window coverage.
3. Set trigger logic: When the policy has only one condition, leave Policy triggers at the default value of Any condition is met.
4. Route the response: Configure Notifications channels and Documentation so responders know whether to slow releases, roll back, or start incident response.
5. Activate the policy: Click Save to create the policy and open the Policy details page.
6. Return to operations view: Click the Navigation menu and go to Anthos Services to return to the dashboard.

2.4Integrating Error Budgets with Observability and Routing State

Tracking SLI compliance in Metrics Explorer starts from the resource type that matches the configuration APIs in use. For deployments that use service routing APIs, the mesh-level signal comes from the Gateway Scope or Mesh resource with the connected_clients metric, which shows whether Envoy proxies remain connected to the control plane. For request success and volume, the signal comes from the Compute Engine Backend Service resource with Request count or Request count by zone, which flows from each sidecar proxy to the backend service. A drop in connected_clients means the control plane can no longer reach proxies over xDS APIs, so routing updates stop flowing from the control plane to those clients, while an error spike in Request count means the data plane between proxies and backends is failing and directly consuming budget.

Cross-region consumption is visualized with Monitoring Query Language (MQL) by fetching the gce_backend_service resource and the trafficdirector.googleapis.com/xds/server/request_count metric, then filtering for records where source region does not equal destination region. Data flows from each sidecar proxy to Cloud Monitoring as xDS server request counts, and the MQL aggregation groups by source region, destination region, and backend service ID with a one-minute rate. When the Front End service in us-central1 sends more requests to the Shopping Cart service in asia-southeast1, that shift appears as growth in the cross-region series and attributes burn to failover or misrouting.

Grounding budgets in routing state links failures to the backend service, health check, and routing resources that direct traffic. Traffic flows from the user through an external global load balancer to the Front End microservice, then from the Front End proxy to the Shopping Cart microservice, then from the Shopping Cart proxy to the Payments microservice. If health checks report that Shopping Cart VMs in us-central1 are unhealthy, Cloud Service Mesh instructs Front End proxies to fail over traffic from us-central1 to backends in asia-southeast1 and notifies the managed instance group in asia-southeast1 to scale. Logs-based metrics complement this view by counting control-plane error messages, such as API resource conflicts, so misconfiguration is detected even when connected_clients looks healthy.

Opportunity cost of risk and reliability is the tradeoff between the business benefit of higher availability and the engineering cost of achieving it. AlloyDB Omni (opens in a new tab) illustrates the tradeoff through standard, enhanced, and premium availability architectures that survive progressively larger failure scopes at progressively higher cost. Every extra nine of availability reduces downtime but demands more redundancy, complexity, and delivery friction.

3.1Translating Availability Percentages into Yearly Downtime

Availability percentage is a direct promise about allowable downtime over a year. Yearly downtime equals 1 minus availability multiplied by the minutes in a year, which is about 525,600 minutes. A 99.9% target allows about 0.1% downtime, or about 526 minutes per year, which is about 8.8 hours, while a 99.99% target allows about 0.01% downtime, or about 52.6 minutes per year. That tenfold reduction in minutes translates into revenue loss, user impact, and reputation cost.

Revenue loss can be estimated as downtime minutes multiplied by revenue per minute for the affected flow, such as orders placed and payments received. User impact grows when downtime blocks critical journeys like checkout, deployment, or autoscaling during peak demand. Brand reputation cost grows when failures are visible, repeated, or require manual workarounds, even when direct revenue loss is small.

3.2Prioritizing Recovery Targets by Business Impact

Recovery targets prioritize components by how much their failure hurts developers and operations. Recovery time objective (RTO) sets how quickly service must be restored, and recovery point objective (RPO) sets how much data or configuration loss can be tolerated. Components with high developer impact and high operations impact have the greatest business impact and need the lowest RTO and RPO objectives, meaning stricter and more expensive targets. A lower RTO and RPO number therefore means less tolerable downtime and loss.

Artifact Registry and source control management (SCM) sit in the highest-impact quadrant because an outage blocks builds, deployments, and autoscaling. Artifact Registry therefore needs lower RTO and RPO targets than the infrastructure as code (IaC) pipeline, whose outage has smaller operational impact because infrastructure can sometimes be updated by other methods. Test data sources with moderate developer impact and low operations impact accept higher, cheaper objectives, while the deployment pipeline with moderate developer impact but high operations impact sits between those extremes. Cloud Key Management Service, the continuous integration pipeline, and the IaC pipeline sit in middle tiers with moderate impact on one or both dimensions.

3.3Comparing Standard, Enhanced, and Premium Availability Architectures

AlloyDB Omni offers standard, enhanced, and premium availability architectures to protect databases from planned and unplanned outages. Standard Availability protects data using backups and suits development environments, Enhanced Availability adds zonal replication in a region for high availability within one region, and Premium Availability adds zonal and regional replication for high availability plus disaster recovery across regions. Failure scopes drive the cost: a VM failure affects one database process, a node failure affects the host, a zone failure means the whole data center fails, and a region failure means the whole region with multiple zones is unavailable.

Moving from Standard to Enhanced buys survival of a full data-center loss through replicas in other zones of the same region. Moving from Enhanced to Premium buys survival of a full regional loss through replicas in a second region. Each step adds redundancy, cross-zone or cross-region replication, and operational complexity for failover, which is why mission-critical applications with several 9s and zero data loss justify Premium while development workloads accept restore-from-backup.

3.4Weighing Prevention Cost Against Unavailability Cost

A cost-benefit framework compares the cost of unavailability against the cost of prevention to set the right SLO for each tier. Cost of unavailability is lost deployments, failed autoscaling, prolonged outages, and developer downtime, while cost of prevention is duplicate infrastructure, larger machines, and continuous monitoring. For continuous integration and continuous delivery (CI/CD), the two high-level strategies are active/passive and backup/restore. In active/passive, the passive pipeline mirrors the active toolchain in a different region but stays scaled-down without handling workload, and recovery updates the repository trigger from the external Git provider to switch regions. Choose active/passive when only a small amount of downtime is tolerable for business-critical applications, and choose backup/restore when slower restoration and higher RTO and RPO are acceptable for lower cost.

The pipeline decision for mixed SLOs shows the same tradeoff. A single pipeline that receives all payments from a Pub/Sub or Kafka topic and writes to BigQuery with streaming inserts must meet the stricter less-than-10-minute SLO for all payments with shared resources and no prioritization. Separate pipelines isolate high-priority payments to a streaming pipeline and standard payments to a daily batch pipeline with BigQuery load jobs, assigning more Dataflow workers, larger machines, and autoscaling to the urgent path. Isolation shortens processing for urgent work without over-provisioning the entire system, while a shared Apache Beam transform library keeps development cost down.

Observability and pipeline hardening reduce business impact without buying full duplication. Metrics are numerical measurements at intervals such as request rate, error rate, CPU, and orders placed, logs are time-stamped records of discrete events such as failures and state changes including Cloud Storage access logs and VPC Flow Logs, and traces show the journey of a single transaction through microservices with bottlenecks and durations. Deployment pipelines must be treated like production because a faulty push can cause an outage and a broken pipeline can prolong one by blocking fixes. Protection includes emergency-access accounts with privileged Google Cloud access stored safely with audited procedures, plus cached input artifacts such as libraries and container base images for use when npm, Maven Central, NuGet Gallery, or the central repository is unavailable.

Key Facts

• A service level indicator is the technical measurement, a service level objective is the internal target for that measurement, and a service level agreement is the legal commitment with penalties such as service credits.
• Internal SLO targets must be stricter than external SLA commitments for the same capability to preserve a safety margin that triggers action before contractual breach.
• A 99.9% SLO leaves a 0.1% error budget, so over 1,000,000 requests the service allows 1,000 failures, and burn alerts fire only when consumption is Above threshold with a retest window of at least 30 minutes.
• External Application Load Balancer availability uses loadbalancing.googleapis.com/https/request_count with response class filtering, while latency uses https/total_latencies for end-to-end experience and https/backend_latencies to isolate a specific backend.
• A 99.9% availability target allows about 526 minutes of downtime per year while a 99.99% target allows about 52.6 minutes, and only AlloyDB Omni Premium Availability survives a regional failure.