Error Budgets and Burn Rate Alerts in Cloud Service Mesh

Error budget policies are rules that connect remaining error budget to decisions about release velocity and operational response. In Cloud Service Mesh (opens in a new tab), an error budget derives from a service level objective (SLO), and Cloud Monitoring turns budget consumption into an alerting signal. When consumption is high, the policy can slow releases, trigger incident response, or freeze changes until reliability recovers.

1.1Error Budget Consumption as a Policy Signal

Error budget consumption measures how much allowed unreliability a service has used within a time window. A policy defines a consumption percentage as the Threshold and fires only when consumption is in the Threshold position of Above threshold. Healthy services with low consumption therefore do not trigger action. The alert re-evaluates on a Retest window with a minimum acceptable value of 30 minutes, which prevents brief spikes from causing flapping while still catching sustained reliability loss.

1.2Defining Burn Rate Alerts with Lookback Duration and Threshold

A burn rate alert forecasts budget exhaustion by comparing consumption percentage against time. The Lookback Duration field in the Target section defines the window of recent SLO performance to evaluate. The Threshold field in the Configuration section defines the percentage of budget consumption that is unacceptable within that window. A short lookback with a high threshold detects fast burns that need immediate response, while a longer lookback with a lower threshold detects slow burns that will exhaust the budget before the SLO period ends.

To create the burn rate condition from the Cloud Service Mesh dashboard:

1. Name the condition: Click the Suggested title link to use the name based on your SLO, or enter a custom name.
2. Set the evaluation window: Enter the lookback period in the Lookback Duration field in the Target section, or use the default value.
3. Set the exhaustion signal: Enter the consumption percentage in the Threshold field in the Configuration section.
4. Save the condition: Click Save to return to the Create new alerting policy window with the condition populated.

1.3Turning Burn Conditions into Alerting Policies That Drive Action

An alerting policy in Cloud Monitoring connects the burn rate condition to notifications and runbooks that govern development and operations. The policy carries the condition forward from Cloud Service Mesh, applies trigger logic, and delivers the alert to the team that must act on the budget status. After the policy is saved, the workflow returns to the Cloud Service Mesh dashboard under Anthos > Services for ongoing service monitoring.

To complete the policy that enforces the error budget:

1. Name the policy: Enter a policy name that identifies the service, SLO, and burn severity.
2. Confirm conditions: Keep the automatically populated condition, and optionally add another condition for multi-window coverage.
3. Set trigger logic: When the alerting policy has only one condition, leave Policy triggers at the default value of Any condition is met.
4. Route the response: Configure the Notifications channels and Documentation section so responders know whether to slow releases, roll back, or start incident response.
5. Activate the policy: Click Save to create the policy and open the Policy details page.
6. Return to operations view: Click the Navigation menu and go to Anthos > Services to return to the Cloud Service Mesh dashboard.

Integration of error budgets with Cloud Service Mesh (opens in a new tab) turns mesh traffic and health signals into measurable service level indicator (SLI) compliance in Cloud Monitoring, Cloud Logging, and Cloud Trace workflows. Cloud Service Mesh acts as the control plane that distributes health, routing, and backend information to sidecar proxies or proxyless gRPC clients. Cloud Monitoring collects the resulting request counts and connectivity state for budget tracking. When failed or slow requests consume the allowed error budget, operators can visualize that consumption per service, per region, and per backend to decide whether to allow more releases or freeze changes.

2.1Tracking SLI Compliance in Metrics Explorer

Tracking SLI compliance in Metrics Explorer starts from the resource type that matches the configuration APIs in use, because the correct resource determines which mesh metrics are visible. For deployments that use the service routing APIs, the mesh-level signal comes from the Gateway Scope or Mesh resource with the connected_clients metric, which shows whether Envoy proxies remain connected to the control plane. For request success and volume, the signal comes from the Compute Engine Backend Service resource with Request count or Request count by zone, which flows from each sidecar proxy to the backend service that serves the microservice.

Viewing the core mesh metrics in Metrics Explorer follows an ordered workflow that moves from control-plane connectivity to data-plane traffic:

1. Open Metrics Explorer: Go to the Metrics Explorer page in the Google Cloud console.
2. Select mesh resource: In the Resource type list, select Gateway Scope or Mesh for service routing APIs, or Network for older APIs.
3. Select connectivity metric: In the Metric list, select connected_clients to verify proxy-to-control-plane attachment.
4. Select backend resource: Return to the Resource type list and select Compute Engine Backend Service.
5. Select traffic metric: In the Metric list, select Request count or Request count by zone to track request volume and failures by backend.

A drop in connected_clients means the Cloud Service Mesh control plane can no longer reach proxies over xDS APIs, so routing updates and failover instructions stop flowing from the control plane to those clients. A drop or error spike in Request count means the data plane between sidecar proxies and backends is failing, which directly consumes the error budget for that backend service.

2.2Visualizing Cross-Region Error Budget Consumption with MQL

Visualizing cross-region error budget consumption with Monitoring Query Language (MQL) isolates traffic that leaves its origin region, where latency and failover most affect distributed SLIs. The documented pattern fetches the gce_backend_service resource and the trafficdirector.googleapis.com/xds/server/request_count metric, then filters for records where metric.source_region does not equal metric.destination_region. The query aligns the rate to one minute, evaluates every minute, and groups by source region, destination region, and backend service ID to aggregate request counts per cross-region path.

Running the cross-region query in Metrics Explorer follows an explicit order:

1. Select MQL: Switch the query editor to MQL mode.
2. Enter cross-region query: Fetch gce_backend_service, filter on unequal source and destination regions, align as rate(1m), and group by regions and backend service.
3. Run query: Click Run query to chart cross-region request rates over time.

Data flows from each sidecar proxy to Cloud Monitoring as xDS server request counts, and the MQL aggregation transforms those raw counts into a per-path burn view from source region to destination region. When the Front End service in us-central1 begins sending a larger share of requests to the Shopping Cart service in asia-southeast1, that shift appears as growth in the cross-region series for that backend service ID. Operators use that growth to attribute budget burn to failover, overflow, or misrouting rather than to a global outage.

2.3Detecting Budget Threats with Logs-Based Metrics and Alerts

Detecting budget threats with Cloud Logging logs-based metrics converts unexpected control-plane messages into countable, alertable signals that complement request-count SLIs. A logs-based metric counts occurrences of a user-specified log message in included logs, so a configuration change that creates API resource conflicts becomes a rising metric rather than a hidden log line. Cloud Monitoring then evaluates that metric and notifies the operator when the conflict pattern appears, which allows intervention before failed routing consumes the remaining error budget.

The dependency runs from Cloud Service Mesh configuration to Cloud Logging to Cloud Monitoring: the mesh emits the error message, the logs-based metric counts messages from source log to metric, and the alerting policy applies to that metric to notify the operator. This path is especially important because a resource conflict can leave proxies connected yet misconfigured, so connected_clients may look healthy while requests fail. Choose logs-based alerts when the risk is control-plane misconfiguration, and choose request-count charts when the risk is data-plane errors or latency.

2.4Grounding Error Budgets in Load-Balancing Health and Routing State

Grounding error budgets in load-balancing state means linking SLI failures to the backend service, health check, and routing resources that actually direct traffic. During setup, Cloud Service Mesh depends on several configured components that together determine where a proxy sends each request:

• Backend service, which holds configuration values including capacity and backend membership for a microservice.
• Health check, which provides health results for virtual machine (VM) instances and Google Kubernetes Engine (GKE) Pods in the deployment.
• Mesh or Gateway resource and Route resource, which define routing when using the service routing APIs.
• Global forwarding rule with VIP address, target proxy, and URL map, which defines routing when using the load balancing APIs.

An xDS API-compatible sidecar proxy such as Envoy runs on each client VM instance or in each Kubernetes Pod, and Cloud Service Mesh serves as the control plane that communicates directly with each proxy over xDS APIs. Traffic flows from the user through an external global load balancer to the Front End microservice, then from the Front End proxy to the Shopping Cart microservice, then from the Shopping Cart proxy to the Payments microservice. Each hop depends on the health check result and backend service capacity that the control plane last pushed to that proxy.

Failover shows how health state moves error budget consumption between regions. If health checks report that Shopping Cart VMs in us-central1 are unhealthy, Cloud Service Mesh instructs the Front End proxies to fail over traffic from us-central1 to Shopping Cart backends in asia-southeast1. Because autoscaling is integrated with traffic management, Cloud Service Mesh notifies the managed instance group (MIG) in asia-southeast1 of the additional load and that MIG increases in size. If all Payments backends remain healthy, the Shopping Cart proxies send traffic up to the configured capacity in the local region and overflow the rest to the remote region, which spreads request volume and budget burn across both regions in a predictable, capacity-bound way.

An error budget for SLO-based reliability is the amount of unreliability a service is allowed within a compliance period while still meeting its Service Level Objective (SLO). The budget is the complement of the SLO target, and tracking how much of that budget remains lets teams balance development velocity against reliability. In Google Cloud, Cloud Monitoring (opens in a new tab) visualizes this balance with an SLO chart that summarizes service performance relative to its configured SLO.

3.1How SLIs, SLOs, and Error Budgets Relate

The relationship between Service Level Indicators (SLIs), SLOs, and error budgets defines how reliability is measured, targeted, and governed. An SLI is the measured performance of a service over time, such as the fraction of successful requests or the fraction of requests completed under a latency threshold. An SLO is the target value for that SLI over a compliance period, such as 99.9% successful requests in 30 days. The error budget is the allowed shortfall from that target, so a 99.9% SLO leaves a 0.1% error budget for failures, slow responses, and planned changes.

When measured SLI performance meets or exceeds the SLO, error budget is preserved and teams can continue to release changes at normal velocity. When measured performance falls below the SLO, error budget is consumed in proportion to the gap between actual compliance and the target. An exhausted budget is therefore a policy signal to slow releases, freeze non-essential changes, and prioritize reliability work until the budget recovers in the next period.

3.2Calculating Remaining Error Budget

Remaining error budget is calculated from the SLO target and the measured SLI compliance over the same period. The total budget equals 100% minus the SLO target, and the consumed budget equals the SLO target minus the measured compliance when compliance is below target. Remaining budget is total budget minus consumed budget, expressed either as absolute bad events or as a fraction of the starting budget.

For example, a 99.9% SLO over 1,000,000 requests allows 1,000 failed requests as the total error budget. If monitoring measures 99.5% compliance, or 5,000 failures, the service has consumed 5 times its budget and the remaining fractional budget is negative. If monitoring measures 99.95% compliance, or 500 failures, the service has consumed half its budget and 500 absolute failures, or 0.5 fractional budget, remain for the rest of the period.

3.3Monitoring Error Budgets on Custom Dashboards

An SLO chart in Cloud Monitoring is the supported way to view the health, compliance, or error budget of an SLO for a configured service. Before an SLO chart can be used, services must already be configured for SLO monitoring, and the dashboard feature is supported only for Google Cloud projects. For App Hub configurations, the App Hub host project or management project must be selected.

The chart type menu controls which view of the same SLO relationship is displayed:

A custom dashboard can show the status of all SLOs for a set of services, with up to 100 widgets per dashboard. Creating or modifying dashboards from the Google Cloud console requires the Monitoring Editor (roles/monitoring.editor) Identity and Access Management role on the project.

Adding an SLO chart to a dashboard follows an ordered workflow:

1. Open Dashboards: Go to the Cloud Monitoring Dashboards page and select the Google Cloud project.
2. Select dashboard: Select an existing dashboard or click Create dashboard.
3. Add widget: Click Add widget and select SLO.
4. Select service and SLO: Choose the service, then choose an existing SLO from the Service-level Objectives menu, creating one first if none exists.
5. Select metric and apply: Choose health, compliance, or absolute or fractional error budget from the Service Level Objective Chart Type menu, optionally update the title, then click Apply.

Key Facts

• A 99.9% SLO leaves a 0.1% error budget, so over 1,000,000 requests the service allows 1,000 failures before the budget is exhausted.
• A burn rate alert in Cloud Service Mesh uses Lookback Duration in the Target section to set the evaluation window and Threshold in the Configuration section to set the unacceptable consumption percentage.
• The burn alert retest window cannot be set below 30 minutes and fires only when consumption is Above threshold.
• A drop in connected_clients means proxies lost connection to the Cloud Service Mesh control plane over xDS APIs, while a spike in Request count errors means the data plane between sidecar proxies and backends is consuming budget.
• An SLO chart in Cloud Monitoring can display health, compliance, absolute error budget, or fractional error budget, with up to 100 widgets per custom dashboard.