Prepare and test your skills
Prepare and test your skills
SLO-based alerting in Cloud Monitoring (opens in a new tab) warns when a service is consuming its error budget too fast and is on track to violate a Service Level Objective (SLO) (opens in a new tab). The alert monitors the burn rate for a chosen SLO and fires when the rate stays above a threshold for an allowed duration. This approach detects both sudden outages and slow drifts before the compliance period ends.
A Service Level Indicator (SLI) is a measurement of service performance derived from metrics such as request counts and response latencies. Common examples are an availability SLI, defined as the ratio of successful responses to all responses, and a latency SLI, defined as the ratio of calls below a latency threshold to all calls. SLIs can be request-based, which count atomic units like successful HTTP requests, or windows-based, which count time windows where a goodness criterion was met.
A Service Level Objective (SLO) is a target value for an SLI measured over a compliance period. Each SLO combines an SLI, a performance goal such as 95% or 99%, and a compliance period such as a rolling 30 days or a calendar week. You can create up to 500 SLOs for each service in Cloud Monitoring, and a useful SLO targets less than 100% because the gap from perfect performance creates the error budget.
The error budget for a compliance period is (1 − SLO goal) × eligible events in the compliance period. If the SLO goal is 95%, then 5% of measured SLI events may fail before the goal is missed. The budget starts full and declines as actual performance misses the SLO, so tracking its consumption shows service health and remaining risk tolerance.
The burn rate tells how fast the error budget is being consumed for a compliance period, based on the number of eligible events and error events received. If no error events occur, no budget is consumed and the burn rate is zero. The metric is normalized so a burn rate greater than 1 means the service will be out of SLO for a future compliance period if the measured error rate is sustained.
The burn-rate metric is retrieved by the time-series selector select_slo_burn_rate. A burn-rate alerting policy notifies when budget consumption exceeds a defined threshold measured over the alert's compliance period. Other time-series selectors for SLO data exist, but policies that use them must be created through the Cloud Monitoring API.
Creating a burn-rate alerting policy follows the same pattern as creating a metric-based alerting policy, with the SLO as the monitored resource. The condition expresses a threshold violation that is permitted only for a limited duration before the policy triggers.
select_slo_burn_rate, plus the threshold, comparison, and duration that define a violation.The condition therefore controls both sensitivity and persistence: the threshold defines how far over the desired rate consumption is, and the duration defines for how long it must persist before an alert fires.
The lookback period for an SLO alert determines how far back in time SLO data is retrieved. Critically, the lookback period is also used as the compliance period for calculating SLO performance and error budget inside that alert. As a result, a short alert window evaluates budget consumption as if the long-term SLO had that short compliance period.
It is not currently possible to base alerts on error-budget consumption rate with a compliance period greater than 24 hours. Approximating a long-term 28- or 30-day compliance period with a period under 24 hours is sufficient for detecting outages and driving short-term operational response. Shorter compliance periods detect problems faster, but large daily swings in traffic and error rates can make alerts overly sensitive during low-traffic periods, so using a burn-rate threshold significantly larger than 1 reduces that sensitivity.
A complete error-budget alerting strategy uses two related policies with different rates and windows to balance speed against false positives. The fast-burn policy catches sudden, large consumption that would exhaust the budget very soon, while the slow-burn policy catches sustained, slightly high consumption that would exhaust the budget before the end of the compliance period.
| Alert type | Purpose | Lookback period | Starting threshold |
|---|---|---|---|
| Fast-burn alert | Detect a disastrous spike that persists even briefly and needs immediate response | Short 1- or 2-hour lookback for fast notification | 10x the baseline burn rate, much higher than ideal for the window |
| Slow-burn alert | Detect a less-urgent drift that stays modestly too high and smooths short-term variation | Longer 24-hour lookback to average out fluctuations | 2x the baseline burn rate, only moderately above ideal |
The fast-burn alert uses a high threshold because only a large deviation justifies paging on a short window. The slow-burn alert uses a lower threshold over a longer window because even a small sustained excess will eventually consume the whole budget. Together, the pair detects rapid depletion without generating excessive alerts when short-term consumption levels out.
Exam tip: Use 10x baseline with a 1- or 2-hour lookback for fast-burn alerts and 2x baseline with a 24-hour lookback for slow-burn alerts; alert compliance periods greater than 24 hours are not supported.
Advanced alerting in Cloud Monitoring (opens in a new tab) is the practice of defining precise conditions over time-series data or logs that create an incident and route notifications to the right responders. Data flows from the monitored resource to the alerting policy condition, then from the condition to an incident record, and finally from the incident through notification channels to people or systems. This chain lets teams detect high latency, missing data, future capacity problems, and cost overruns with one consistent model.
Cloud Monitoring alerting connects three parts into one response path. The alerting policy describes what to watch and who to notify, the incident is the persistent record of what was watched and when the condition was met, and the notification channel defines how the notice is delivered. When metric data stored by Monitoring or log data stored by Cloud Logging (opens in a new tab) meets the condition, Monitoring creates the incident and sends notifications through every channel attached to the policy. Each notification carries troubleshooting context such as incident status, links to logs, a chart of recorded metric data, labels, and duration.
An alerting policy continuously evaluates after it is enabled and cannot be scheduled to run only during certain hours. If an incident is open and Monitoring later determines that a metric-based condition is no longer met, Monitoring moves the incident to closed and sends a closure notification when configured. To pause evaluation temporarily without disabling the policy, operators create a snooze for a defined time period. Policies can be created in the Google Cloud console, the Cloud Monitoring API, the Google Cloud CLI, Terraform, or by enabling recommended policies from integration packages for services such as Google Kubernetes Engine, Compute Engine, Cloud SQL, MongoDB, Kafka, and Elasticsearch.
Metric-based alerting policies evaluate time-series data stored by Monitoring using threshold, absence, query-language, and forecast logic. Each condition combines a data source with filters that select time series, aggregation settings that combine them, and a static or dynamic threshold with a retest window that controls how long the violation must persist. Conditions can monitor a single metric, multiple metrics, or a ratio of metrics, and multi-condition policies use a combiner such as OR to decide when the policy as a whole is met. Changing the retest window and the alert trigger, such as Any time series violates, directly changes how often incidents fire.
The four time-series condition types differ by what they detect and how far ahead they look:
| Condition type | When condition is met | Example use |
|---|---|---|
| Metric-threshold | Metric value stays above or below a threshold for a retest window | Response latency at 500ms or higher for five uptime checks over 10 minutes, or HTTP latency above two seconds for five minutes |
| Metric-absence | Monitored time series produces no data for a retest window up to 24 hours | Resource sends no HTTP responses over five minutes |
| Prometheus Query Language (PromQL) | PromQL expression evaluates to true in real time, including ratios and dynamic thresholds | Ratio of two metrics, reuse of a public repository expression, or analysis over more than 25 hours of data |
| Forecasted metric-value | Forecast predicts a threshold violation within a forecast window of 1 hour to 7 days | Disk usage likely to reach 80% within the next 24 hours |
Log data uses two related paths that complement metric alerts. Log-based policies fire when an individual log entry matches a phrase, while SQL-based policies periodically analyze aggregations over many entries, such as more than 50 entries with severity WARNING in the last 60 minutes. For cost governance, billable and forecasted charges are handled through budgets rather than metric thresholds, with default budget notifications at 50%, 90%, and 100% and product selections for Cloud Logging, Stackdriver Monitoring, and Stackdriver Trace.
Aggregation and alignment settings in Cloud Monitoring determine which time series enter a condition and how their points are combined before the threshold is tested. Filters remove unwanted time series from query results, the rolling window and rolling window function align points within each series over time, and the across-time-series aggregation and group-by combine series by labels such as zone or priority. The console maps these concepts to chart settings, where rolling window corresponds to minimum interval and group-by and aggregation correspond to chart aggregation menus. A BigQuery example groups by priority, applies 99th percentile across series, and uses a 5 m rolling window with sum to test whether 99th-percentile query execution time stays above a chosen value such as 60 seconds.
Reliable selection depends on using labels defined by monitored-resource types rather than system- or user-defined resource metadata labels. Resource metadata labels are added asynchronously and do not follow the same latency expectations as metric or monitored-resource labels, so a sensitive policy that filters on metadata can behave erratically when label population is delayed. When a resource label and a metric label carry similar information, such as cluster and kubernetes_cluster_name, filtering on the resource label produces more stable selection. Together with threshold position, retest window, combiner logic, incident autoclose duration, and snoozes, these settings are the main levers for reducing alert fatigue without missing real failures.
Exam tip: Do not use resource metadata labels to select time series for sensitive alerting policies; use labels defined by monitored-resource types because metadata labels populate asynchronously and can cause erratic triggering.
Query-based conditions in Cloud Monitoring handle logic that simple thresholds cannot express, including ratios, dynamic thresholds, long lookbacks, and multi-entry log patterns. A Prometheus Query Language (PromQL) condition accepts any valid PromQL expression over time-series data, which supports metric combinations, ratios, conditional logic, and evaluations over more than 25 hours. A SQL-based condition runs a SQL query in Observability Analytics against log data through a linked BigQuery dataset and fires on one of two evaluations: row-count threshold, where the number of result rows is greater than, equal to, or less than a value, or boolean, where any row in a named boolean column is true. A SQL-based policy must contain only one condition.
SQL-based execution has explicit timing, cost, and failure behavior that the policy depends on. Observability Analytics delays each scheduled query by five minutes so Cloud Logging can index data from the lookback window, then the resulting incident can take up to two additional minutes to propagate. Queries run on the BigQuery engine in the project where the policy is defined, using on-demand slots by default or dedicated reservation slots when configured, and both paths follow BigQuery pricing. A query fails when the Monitoring Service Account is missing or lacks permission, when execution exceeds five minutes, or on internal error, and each failure writes a log entry with the policy ID that can itself be watched by a log-based alert.
Notification routing in Cloud Monitoring sends each incident from one alerting policy to one or more notification channels owned by different teams or systems. A policy can attach multiple channels for redundancy, and configuring multiple channel types ensures that an email outage or chat outage alone does not block response. Supported destinations named in the documentation provide the routing vocabulary for most designs.
An alerting policy can route incidents to channels including:
Routing behavior is controlled in a repeatable creation flow that links selection, triggering, delivery, and documentation. First, the operator selects the resource type, metric category, metric, and filters and applies aggregation and alignment. Then the operator configures the condition type, threshold position, threshold value, alert trigger, and retest window. Then the operator attaches notification channels and chooses whether to notify on closure. Then the operator sets incident autoclose duration and adds documentation such as playbook links, dashboard links, and a custom subject line. Only after these dependencies are complete does the operator name and create the policy, after which monitoring starts immediately for recommended policies or newly saved custom policies.
Custom content and limits shape what responders actually receive and how often delivery repeats. Log-alert policies can set labelExtractors for header properties and content with markdown variable substitution for email bodies, while metric policies use documentation variables for subject and body customization. Personalized Service Health builds on the same log-alert path, with each service-health notification linking to its incident page and the Service Health dashboard and with Monitoring limits such as 20 alerts per policy per day per project bounding repeated delivery. Incident autoclose duration closes incidents when metric data stops arriving, and the missing-data handling choice determines whether silent data loss violates the condition or is treated as closure, which matters because treating missing data as a violation can create false positives when an instance is deleted.
Cloud Billing budgets (opens in a new tab) are alerts-only monitors that track actual and forecasted Google Cloud charges against a planned spend amount. A budget evaluates spend for its defined scope and time period, compares it to threshold rules, and then sends notifications to email recipients and to Pub/Sub when a threshold is crossed. Notifications do not stop usage by themselves, so cost control depends on people responding to emails or on automation that subscribes to Pub/Sub and takes action.
Budget scope, amount, and time period define exactly what spend a Cloud Billing budget watches and what limit it is measured against. Scope controls which costs are included, amount sets the target number, and time period controls whether the budget resets monthly, quarterly, yearly, or over a custom range. A single billing account is commonly used across projects, with narrower budgets delegated to workload owners for day-to-day monitoring.
The documented scope levels for a budget are:
billingcode label used for cost centersThe budget amount is either a fixed total that you specify or an amount based on the previous calendar period spend. The cost trend chart shows the prior 12 months of costs to help choose a realistic amount. Because there is a delay between resource usage and billing reporting, plus several hours before the first notification arrives, budgets should be set below available funds.
Alert threshold rules are percentage-of-budget conditions that fire on actual costs, forecasted costs, or both. By default, budgets notify at 50%, 90%, and 100% of the budget, and additional thresholds can be added or removed during budget creation or editing. For example, an enterprise foundation may alert when forecasted spend is on track to reach 120% of budget so a central team can investigate overspending that could also indicate a security incident.
Actual-cost rules react to spend already incurred, while forecasted-cost rules react to spend projected by the end of the calendar period. A $100 budget with a 110% forecasted rule sends a notification when the projection exceeds $110 for that period. Budgets configured for a custom time range cannot trigger alerts on forecasted costs. If all threshold rules are removed, email settings are disabled and overridden, leaving only programmatic notifications.
Notification delivery determines who learns about a threshold breach and which system can act on it. Cloud Billing budgets support role-based email, Cloud Monitoring notification channels, and Pub/Sub programmatic delivery, and at least one threshold rule plus one email option is required when threshold-based email or detailed Pub/Sub threshold fields are needed. Project-level permissions allow managing budgets for one selected project at a time, while billing-account permissions allow viewing and managing all budgets on the account.
| Delivery option | Delivers budget alert to | Choose when |
|---|---|---|
| Role-based email | Billing Account Administrators and Billing Account Users by default, plus Project Owners for single-project budgets | The response owner holds a billing or project-owner role on the account or project |
| Cloud Monitoring notification channels | Custom email addresses linked to the budget, such as project managers | Recipients outside billing roles need direct email without broadening IAM access |
| Pub/Sub programmatic notification | JSON message to a connected Pub/Sub topic, including alertThresholdExceeded and forecastThresholdExceeded when thresholds are configured | The alert must trigger automation or forward to another medium such as Slack |
Role-based email is controlled by selecting or clearing Email alerts to billing admins and users and, for single-project budgets, Email alerts to project owners. Cloud Monitoring channels are added by linking Monitoring email notification channels to the budget. Budget alerts are consumption alerts for budgets you define, which is different from Essential Contacts billing notifications about pricing updates, errors, and credits.
Exam tip: An alerts-only budget never caps usage or spending automatically; it only sends email and Pub/Sub notifications when thresholds are met.
Programmatic cost control is the path where a budget alert flows from Cloud Billing to a Pub/Sub topic and then to automation that enforces a response. Data flows from the budget evaluation to the configured Pub/Sub topic, then from the topic to each subscriber, and then from the subscriber to the target project or service where the control is applied. The Cloud Billing Budget API supports creating and managing these budgets programmatically at scale across an organization.
Common automated responses built on budget Pub/Sub messages include:
To include threshold detail in the Pub/Sub JSON object, the budget must retain at least one alert threshold rule and at least one email notification option. Deleted budgets cannot be recovered, so automation should treat budget deletion as a governance event. For services that support it, consider a spend cap budget in preview on a project when a hard limit is required instead of notification-only control.
Billing anomaly detection, cost allocation, and budget governance provide the surrounding controls that make budgets reliable for financial and security operations. Anomaly detection in Cloud Billing automatically flags spikes or deviations from expected spend, and a sudden unexplained spike is treated as a possible compromise because attackers may consume resources for unauthorized activity. Allocation and governance ensure the right team sees the right cost and can act before the period closes.
Allocation relies on consistent labeling and export: each project carries a billingcode metadata label, billing data is exported to a BigQuery dataset for chargeback by cost center, and reports group charges and credits by that label and service description. Governance relies on setting environment- or folder-level budgets for central oversight while workload owners set more granular thresholds. Together, budgets for planned limits, anomalies for unplanned deviations, and BigQuery export for attribution give both cost control and an early warning system for suspicious activity.
The burn rate measures how fast the error budget is being consumed for a compliance period, where the error budget is (1 minus SLO goal) times eligible events in the compliance period. A burn rate greater than 1 means the service will miss its Service Level Objective if the current error rate continues, and a burn-rate alerting policy notifies when consumption exceeds a defined threshold measured over the alert's compliance period.
Use a fast-burn alert with a 10x baseline threshold and a short 1- or 2-hour lookback to detect a disastrous spike that persists even briefly and needs immediate response. Use a slow-burn alert with a 2x baseline threshold and a longer 24-hour lookback to detect less-urgent sustained drift while smoothing short-term variation, since alert compliance periods greater than 24 hours are not supported.
No, Cloud Billing budgets are alerts-only monitors that never stop usage automatically. They compare actual and forecasted charges to the budget amount and send notifications to email recipients and to Pub/Sub when threshold rules are crossed, with default notifications at 50 percent, 90 percent, and 100 percent, so cost control depends on people responding or automation subscribed to Pub/Sub.
When metric data stored by Monitoring or log data stored by Cloud Logging meets the policy condition, Monitoring creates an incident and sends notifications through every notification channel attached to the policy. If an open incident's metric-based condition is later no longer met, Monitoring moves the incident to closed and sends a closure notification when configured.
Your DevOps team is implementing a Service Level Objective (SLO) error-rate alerting policy in Google Cloud Monitoring for a microservice deployed across 200 Compute Engine virtual machine (VM) instances. The initial configuration computed a ratio across two independent custom metrics: custom.googleapis.com/rpc/total_count and custom.googleapis.com/rpc/error_count.
During low-traffic intervals, on-call engineers receive frequent false-positive alerts caused by anomalous ratio spikes (such as 1/0 or 2/1) evaluated on individual instances. You need to redesign the custom metrics, alignment, and alerting policy condition to eliminate these ratio calculation anomalies, reduce alert fatigue, and minimize monitoring costs.
What should you do?
Last updated