Prepare and test your skills
Prepare and test your skills
Metric and log-based alerting policies in Cloud Monitoring (opens in a new tab) detect unhealthy behavior and trigger a response before users are affected. A metric-based policy evaluates numerical time-series data stored by Monitoring, while a log-based policy evaluates log entries stored by Cloud Logging. When the monitored data meets the policy condition, Monitoring creates an incident and sends notifications through the configured notification channels.
Cloud Monitoring alerting connects a policy, an incident, and a notification channel into one response flow. The alerting policy defines what data to watch, what condition means trouble, and how to notify responders. When data from Monitoring or Cloud Logging meets that condition, Monitoring creates an incident that records what was monitored and when the condition was met, and then sends notifications to each channel in the policy.
A left-to-right process flow showing an alerting policy evaluating metric, log, or SQL data, creating an incident when the condition is met, and sending notifications to multiple channels such as email and Slack, with automatic incident closure when the condition clears.
An alerting policy can evaluate time-series metric data, individual log entries, or the results of a SQL query against logs in Observability Analytics. A single policy can list one or more notification channels, so the same incident can email my-support-team@example.com and post to #my-support-team at the same time. After a policy is enabled, Monitoring evaluates its conditions continuously, and when the condition for a metric-based policy is no longer met, Monitoring automatically closes the incident and sends a closure notification if configured.
Metric-based alerting policies monitor numerical time-series data for threshold violations, missing data, complex expressions, and forecasted problems. Each policy condition selects a resource type and metric, narrows the time series with filters and group-by aggregation, and then compares the aligned values against a static or dynamic threshold for a specific duration. For example, a policy can watch HTTP response latency and create an incident only when latency stays above two seconds for at least five minutes, which prevents a brief spike from paging the team.
The four time-series condition types differ by what pattern they detect and how far they look back:
| Condition Type | What triggers it | Example use |
|---|---|---|
| Metric-threshold condition | Metric value is above or below a threshold for a specific retest window | Notify when response latency is 500ms or higher for five consecutive uptime checks over 10 minutes |
| Metric-absence condition | Monitored time series has no data for a specific retest window, up to 24 hours | Open an incident when a resource receives no HTTP requests for five minutes |
| Prometheus Query Language (PromQL) condition | PromQL expression using combinations, ratios, or dynamic thresholds evaluates to true | Alert on a ratio of two metrics or evaluate more than 25 hours of data |
| Forecasted metric-value condition | Forecast predicts a threshold violation within a forecast window of 1 hour to 7 days | Warn when disk usage is likely to reach 80% within the next 24 hours |
Choose a metric-threshold condition for simple high-or-low checks, a metric-absence condition for silent or dead resources, a PromQL condition for ratios and multi-metric logic, and a forecasted condition for capacity problems that have not happened yet. Use resource labels from the monitored-resource type to select time series, because resource metadata labels are added asynchronously and can make sensitive policies behave erratically if their population is delayed.
Log-based alerting policies notify responders when a specific message appears in included logs stored by Cloud Logging. The condition is met when a log entry matches the policy filter, such as a message field containing specific product IDs. This approach fits immediate events like an authentication failure, a failed integration execution, a specific severity log, or an execution time that exceeds a value.
To monitor recurring events in logs over time, log-based metrics first convert logging data into numerical time-series data that a metric policy can evaluate and chart. Log-based metrics are suitable when you want to:
Exam tip: A log-based alerting policy fires on an individual matching log entry, while a log-based metric counts or trends recurring log events over time before a metric threshold fires.
Accurate alerting policies combine a precise filter, an appropriate threshold duration, and clear response information. The condition defines the data source, threshold, and aggregation through filters and group-by, and it can monitor a single metric, multiple metrics, or a ratio of metrics. Longer durations and retest windows, such as requiring five minutes above threshold instead of firing on one point, reduce false positives while still catching sustained latency, error-rate, or utilization problems.
Clear operations practices keep alerts actionable after they are created:
A policy cannot be scheduled to monitor only during certain hours, so create a snooze to silence a policy for a defined period, and rely on automatic incident closure to confirm when the system has returned to normal.
SRE principles and SLO-based alerting define reliability from the user's perspective and alert only when that reliability is at risk. Holistic observability in Google Cloud Observability (opens in a new tab) requires adopting SRE principles, defining clear reliability goals, tracking metrics across system layers, and using insights for continuous improvement. This approach keeps teams focused on critical user-impact events instead of noisy infrastructure signals.
The SRE approach turns reliability into a measurable engineering property by defining Service Level Objectives (SLOs) that quantify acceptable levels of reliability and performance from the user's perspective. Clear SLOs align engineering work directly with business needs, so reliability efforts focus on the system behavior that most affects users and the business. Google Cloud Monitoring and Cloud Logging together provide the metrics, dashboards, and alerting mechanisms used to track those goals and support triage in advance.
Service Level Indicators (SLIs) and SLOs connect business outcomes to technical reliability metrics across complex cloud architectures. For each business key performance indicator (KPI), such as revenue influenced by AI recommendations, customer churn, or user engagement, teams define the corresponding technical metrics that affect it, such as request success rate, latency, harmful-response rate, and successful task completion. Those technical metrics are then quantified with specific SLO targets, for example 99.9% of API calls must return a successful response, 95th percentile inference latency must be below 300 ms, time to first token must be below 500 ms for 99% of requests, and rate of harmful output must be below 0.1%.
Google Cloud Observability monitors infrastructure, application performance, and request flows using complementary telemetry signals and managed services. The four golden signals provide a standard way to track application health:
Infrastructure metrics such as processor usage, memory usage, network throughput and latency, and disk I/O are collected across managed environments and self-managed resources like GKE nodes and Cloud Run instances. OpenTelemetry provides an open-source standard to capture metrics, traces, and logs, while Google Cloud Managed Service for Prometheus provides a fully managed backend for Prometheus metrics at scale. Cloud Trace pinpoints latency and request flows across distributed microservices, Cloud Profiler identifies bottlenecks in code, and Error Reporting automatically captures and groups errors for analysis.
SLO-based alerting strategies use Cloud Monitoring alerting policies to notify teams when user-facing performance fails to meet acceptable values and to manage error budgets effectively. Each strategy contains three connected parts: an alerting policy describes the condition to watch and how to notify, an incident records what data met the condition and when, and a notification channel defines where the notification goes, such as email, Slack, PagerDuty, Pub/Sub, SMS, or webhook. Data flows from the monitored time series or logs to the alerting policy condition, then to a created incident, then to each configured notification channel.
Alerting policies in Cloud Monitoring evaluate different kinds of data:
| Policy type | Data evaluated | Choose when |
|---|---|---|
| Metric-based alerting policy | Time-series metric data stored by Monitoring | Response latency stays above 2 seconds for 5 minutes or CPU, error rate, or quota usage crosses a threshold |
| Log-based alerting policy | Individual log entries or log-based metrics stored by Cloud Logging | A specific error message appears, failed executions spike, or latency values extracted from logs change unacceptably |
| SQL-based alerting policy | Results of a SQL query in Observability Analytics against logs | The answer requires aggregating and analyzing log data with SQL before deciding to alert |
Effective SLO-based alerts prioritize symptoms that burn the error budget and affect users, with clear, actionable information and appropriate severity levels. Custom alerts for business-specific thresholds extend the same pattern to model quality, training-serving skew, prediction drift, and quota limits, and they can integrate with incident tools like PagerDuty or Slack for proactive response.
Exam tip: Alert on SLO-relevant symptoms such as sustained high latency or elevated error rate that affect users, not on every short infrastructure fluctuation, to reduce noise and protect the error budget.
Notification channels and incident management integration is how an alerting policy gets actionable information to the right people during an incident. In Cloud Monitoring (opens in a new tab), an alerting policy watches data, creates an incident when conditions are met, and sends notifications through one or more notification channels. A strong design combines multiple channels for different audiences with clear documentation and playbooks inside the alert so responders can act quickly.
The Cloud Monitoring alerting process connects three parts in order. First, an alerting policy describes the condition to watch and how to notify, and it can monitor time-series metric data, individual log entries, or SQL query results. When the data meets the condition, Monitoring creates an incident as a record of what was monitored and when the condition was met. The incident then flows from the alerting policy to each configured notification channel, which defines how responders receive the message, such as email to my-support-team@example.com and a Slack post to #my-support-team.
An alerting policy can contain one or more notification channels, so one incident can notify several stakeholders at once. By default, notifications are sent only when incidents are opened, and responders are notified about closure only if Notify on incident closure is selected. If an open metric-based incident recovers because the condition is no longer met, Monitoring automatically closes the incident and can send a closure notification.
A multi-channel notification strategy routes the same incident to different tools based on urgency, role, and location of the responder. Log alerts and metric alerts both use notification channels to notify you or your on-call team when a policy is triggered, and supported integrations include:
For third-party incident tools, use webhooks to send a notification to any public endpoint such as ServiceNow, Datadog, or OpsGenie, and use Pub/Sub to send a notification to any private endpoint. You control which events reach those tools with logging filters written in the Logging query language, and you can add labels from log fields to help the receiving tool parse the alert. Alerts are also sent to the essential contacts configured in the Google Cloud project, so project contacts and policy channels work together to cover both owners and responders.
Exam tip: By default, notifications are sent only when incidents open; select Notify on incident closure when you also want a message when the incident closes.
Alert documentation is user-defined text stored with the alerting policy and included in the notification message. The Documentation field lets you add context such as what the alert means, likely causes, triage steps, and links to runbooks or playbooks. Because this text travels with the incident notification, a responder who receives email, Slack, or PagerDuty sees the same instructions without searching for them.
Well-written documentation makes alerts clear and actionable by stating the issue and suggesting appropriate actions. Effective incident response procedures document roles for incident commanders, investigators, communicators, and technical experts, plus communication protocols and escalation paths. The runbook or playbook referenced in the alert should outline the steps for communication, triage, investigation, and resolution so any on-call responder follows the same process.
Severity and routing decide who must act now and who can act later. A best practice is to categorize alerts by impact and urgency, document the alert rules, notification channels, and escalation procedures, and test alerts regularly to verify triggering and delivery. In Google Distributed Cloud, alerts can generate pages and tickets for critical errors, where pages require immediate attention from an operator while tickets are less urgent.
Alertmanager is the component that receives alerts from defined rules and handles notification routing, grouping, silencing, and inhibiting to reduce noise. Grouping combines related alerts, silencing pauses notifications for a time, and inhibiting suppresses lower-priority alerts when a higher-priority condition explains them. This flow moves from data collection with tools like Prometheus and Fluent Bit, to storage and visualization in Grafana dashboards, to evaluation by alerting rules, to Alertmanager for routing decisions, and finally to destinations such as email, messages, or webhooks.
A log-based alerting policy fires on an individual matching log entry, while a log-based metric converts recurring log events into numerical time-series data for a metric-threshold policy to evaluate. A log-based alerting policy notifies when a specific message appears in included logs stored by Cloud Logging, such as an authentication failure or specific severity log, while a log-based metric counts occurrences or observes trends over time and can be charted before a threshold fires.
Choose a metric-threshold condition for simple high-or-low checks, a metric-absence condition for silent or dead resources, a Prometheus Query Language (PromQL) condition for ratios and multi-metric logic, and a forecasted metric-value condition for capacity problems that have not happened yet. A metric-absence condition detects no data for a retest window up to 24 hours, a PromQL condition can evaluate more than 25 hours of data, and a forecasted condition predicts a threshold violation within a forecast window of 1 hour to 7 days. For example, a threshold policy can create an incident only when latency stays above two seconds for at least five minutes to prevent a brief spike from paging the team.
In Cloud Monitoring an alerting policy defines what data to watch, what condition means trouble, and how to notify, and when monitored data meets that condition Monitoring creates an incident that records what was monitored and when and then sends notifications to each configured notification channel. A single policy can list one or more channels such as email, Slack, PagerDuty, Pub/Sub, SMS and webhooks, so the same incident can notify several stakeholders at once. By default notifications are sent only when incidents open, metric-based incidents close automatically when the condition is no longer met, and a closure message is sent only if Notify on incident closure is selected.
Service Level Objectives (SLOs) based alerting in Cloud Monitoring notifies teams when user-facing performance fails to meet acceptable values and manages error budgets effectively. It prioritizes symptoms that burn the error budget and affect users, such as sustained high latency or elevated error rate, instead of noisy infrastructure signals or short fluctuations. Each strategy connects an alerting policy that describes the condition and notification method, an incident that records what met the condition and when, and a notification channel that defines where the notification goes.
An e-commerce company runs an order-processing backend on Google Compute Engine instances. The CPU utilization metric is sampled every 60 seconds. During normal operation, background memory garbage collection creates transient CPU utilization spikes of up to 90% that resolve within 45 seconds.
The operations team currently suffers from alert fatigue due to frequent false alarms triggered whenever a single measurement exceeds 85% CPU utilization. You need to configure a Cloud Monitoring metric-threshold alerting policy that ignores these transient spikes and only alerts when CPU utilization remains continuously above 85% for at least 5 minutes.
How should you configure the condition settings in Cloud Monitoring?
Set the alignment period (rolling window) to 5 minutes, select the max aligner, and configure the retest window (duration) to 0 minutes (no retest).
Set the alignment period (rolling window) to 1 minute, select the mean aligner, and configure the retest window (duration) to 5 minutes.
Set the alignment period (rolling window) to 1 minute, configure the retest window (duration) to 0 minutes, and set the condition combiner to OR.
Set the alignment period (rolling window) to 5 minutes, configure the retest window (duration) to 0 minutes, and set the missing data evaluation policy to treat missing data as violating the condition.
Last updated