Prepare and test your skills

Prepare and test your skills

Notification channels and incident documentation are the parts of a Cloud Monitoring alert policy that decide who is told about a resource problem and what instructions they receive. Cloud Monitoring watches resource utilization metrics in real time, and the alert policy sends a notification to its configured contacts when a predefined threshold is met. Resource Manager (opens in a new tab) provides the hierarchy that controls where those contact settings apply, while Essential Contacts carries critical notifications through that hierarchy to the responsible teams.
Essential Contacts is the service for designating custom points of contact for critical notifications across a resource hierarchy. The service depends on Resource Manager for its scope, and notification settings configured on a parent container cascade to child resources. Essential Contacts covers distinct notification areas:
Because the contacts are tied to the hierarchy rather than to one project, vital messages continue to reach the right team even as projects are added or moved.
The resource hierarchy is the organization system that controls where notification and policy settings are inherited from. Resources lower in the hierarchy inherit settings from their parent containers, so a contact set at the organization or folder level automatically applies to everything below it. The hierarchy consists of the following levels:
If a project moves into a different folder or organization, its inherited notification settings change to match the new parent. This parent-to-child inheritance is why contacts should be set high in the hierarchy when the same team must be notified for many projects.
Cloud Monitoring alerts based on resource metrics provide real-time visibility into utilization for right-sizing and performance management. The service tracks standard utilization patterns and also supports custom metrics that follow application-specific utilization relevant to a workload. When a metric reaches a predefined threshold, the alert policy notifies administrators, and the administrators then adjust resource allocation to restore performance and control cost. This flow runs from metric collection to threshold evaluation to administrator notification to corrective action, which connects monitoring directly to managing service disruptions.
Reliable delivery practices ensure that critical notifications reach a durable team address instead of a single person. Reliable delivery depends on configuring group email aliases at the organization or folder level for Essential Contacts. A group alias reaches the whole responsible team and survives individual staffing changes, while the hierarchy cascades that alias to all child folders, projects, and resources. Teams should not rely on default Identity and Access Management (IAM) fallback roles for critical messages, because fallback delivery only occurs when no custom contact is defined.
Exam tip: Configure a group email alias at the organization or folder level for Essential Contacts instead of relying on default IAM fallback roles.
Metric selection and threshold configuration is the process of defining when a Cloud Monitoring (opens in a new tab) alerting policy should create an alert. The policy watches time-series data stored by Monitoring, compares the selected resource metric against a threshold for a set duration, and then sends notifications through a notification channel. This combination of the right metric, a clear threshold, and a duration window keeps alerts accurate and actionable.
Metric-based alerting connects an alerting policy, an alert, and a notification channel into one flow. First, an alerting policy describes what to watch and who to notify, including the data source, threshold, and aggregation settings. When the monitored data meets the condition, Monitoring creates an alert as a record of what was monitored and when the condition was met. The alert then flows to a notification channel, which delivers the message by email, Slack, or another channel to a destination such as my-support-team@example.com.
A five-step process flow showing how an alerting policy watches a resource metric, evaluates it against a threshold and retest window, creates an alert, delivers a notification through a channel such as my-support-team@example.com, and how administrators correct the issue before the alert auto-closes.
Metric selection starts by choosing a resource type, a metric category, and a specific metric that reflects health or performance. For example, a VM instance resource with an instance category can expose CPU Utilization, while a Cloud Bigtable cluster (opens in a new tab) can expose storage utilization and a Spanner instance (opens in a new tab) can expose CPU Utilization by priority. The condition also uses filters and grouping to narrow the time series, such as filtering to one cluster ID or instance ID and grouping across time series by location or dataset ID. Good filtering focuses the policy on the exact resources that require a response, such as high CPU, high memory use, heavy disk I/O, slow HTTP response latency, or rising storage use.
When filtering, prefer labels defined on monitored-resource types instead of labels defined on metric types, especially for high-cardinality metrics. If a monitored resource includes a cluster label and the metric includes a similar kubernetes_cluster_name label, filter on the resource cluster label. Do not use system- or user-defined resource metadata labels to select time series for sensitive policies, because those labels are added asynchronously and can cause erratic triggering if their population is delayed.
Thresholds and duration windows together define how long a bad state must persist before Monitoring opens an alert. The threshold has a position, such as above threshold or below threshold, and a value, such as HTTP response latency higher than two seconds, response latency at 500ms or higher, Bigtable storage utilization above 70, or Spanner high-priority CPU above 45% for multi-region instances and above 65% for regional instances. The duration is set with a retest window and a rolling window, such as a 10-minute retest window with a 10-minute rolling mean, so brief spikes do not trigger a notification. The trigger is commonly set to any time series violates, which means Monitoring opens an alert when any filtered time series stays beyond the threshold for the full window.
Aggregation controls how raw points are combined before the threshold comparison. Across time series, an aggregation such as sum or 99th percentile combines multiple streams, while a rolling window function such as mean, max, or sum aligns points over time. For example, a Spanner CPU policy can group by location, aggregate with sum, and average over a 10-minute rolling window before checking the 45% or 65% threshold. After the policy is enabled, Monitoring evaluates it continuously and automatically closes the alert when the condition is no longer met.
Time-series condition types define different ways a metric pattern can trigger an alert. Each type pairs a metric selection with its own rule for threshold and time, so the choice depends on whether a simple limit, missing data, custom logic, or a prediction is needed. The four options compare as follows:
| Condition type | When the condition is met | Example use |
|---|---|---|
| Metric-threshold condition | Metric value is more than or less than a static threshold for a specific retest window | Response latency is 500ms or higher over 10 minutes |
| Metric-absence condition | Monitored time series has no data for a retest window, up to 24 hours | Resource receives no HTTP requests over five minutes |
| Prometheus Query Language (PromQL) condition | Custom PromQL expression is true, including ratios, combinations, and dynamic thresholds | Alert based on a ratio of two metrics or on more than 25 hours of data |
| Forecasted metric-value condition | Forecast predicts a threshold violation within a forecast window of 1 hour to 7 days | Disk space is likely to reach 80% usage within the next 24 hours |
A metric-threshold condition fits most resource metrics such as CPU utilization, memory usage, disk I/O, and storage utilization. Choose metric-absence when silence means failure, PromQL when the rule needs a ratio or dynamic limit across multiple metrics, and forecasted values when early action can prevent exhaustion such as a full disk.
Exam tip: Filter on monitored-resource labels for reliable selection, and do not depend on resource metadata labels for sensitive alerting policies because metadata arrives asynchronously.
Data alignment and aggregation techniques in Cloud Monitoring (opens in a new tab) are the methods that turn raw time-series points into a reliable signal for an alerting policy. Raw points arrive at different times and from many resources, so a condition first aligns points into regular time intervals and then aggregates them before comparing the result to a threshold. This process keeps normal fluctuation from triggering alerts and helps ensure that alerts fire only when performance truly fails to meet acceptable values.
Alert conditions that use aggregated data define what data to watch, how to combine it, and when to respond. Each condition includes a data source, a static or dynamic threshold, and aggregation methods such as filters and group-by. A condition can monitor a single metric, multiple metrics, or a ratio of metrics, and it can use Prometheus Query Language (PromQL) (opens in a new tab) for complex expressions such as dynamic thresholds and conditional logic. After an alerting policy is enabled, Monitoring continuously evaluates the aggregated result, and when the result meets the condition, Monitoring creates an alert and sends notifications through the configured notification channels.
Filtering before aggregation selects which time series enter the aggregation, and Cloud Monitoring recommends filtering on labels defined on monitored-resource types instead of labels defined on metric types when possible. Filtering on resource labels is especially important for high-cardinality metrics, where many unique label combinations can return many points. For example, if a monitored resource includes a cluster label and the metric includes a kubernetes_cluster_name label with similar information, filter on the resource cluster label. Do not rely on system- or user-defined resource metadata labels for sensitive alerting policies, because metadata labels are added asynchronously and are not subject to the same latency expectations, so a policy that depends on them can behave erratically if label population is delayed.
Common aggregation functions combine aligned points into one value per interval or per group, which smooths short spikes and makes thresholds meaningful across distributed resources. Common approaches supported for alerting policies include value comparison, rate-of-change comparison, ratio comparison, and forecasted-value comparison:
These functions let one policy handle many distributed resources without reacting to every brief fluctuation.
Grouping and retest windows control how many streams are evaluated and how long a violation must last before an alert opens. A point is returned for each time series monitored by the policy, so aggregating 100 virtual machines (VMs) that belong to five services to the service level returns 5 points per evaluation, while aggregating to the VM level returns 100 points per evaluation. Choose the grouping that matches the operational question: aggregate to the VM and CPU level when CPU utilization per machine matters, and aggregate to the service level when latency by service matters. Consolidate coverage into one alerting policy with one condition that monitors many resources, rather than creating 100 separate policies for 100 VMs, because each metric-threshold condition adds a metric reference cost (opens in a new tab) even when both designs return the same number of points.
A retest window requires the aggregated value to violate the threshold for a sustained period before an alert opens, which minimizes false positives from brief spikes. For example, a metric-threshold condition can require latency at or above 500ms for five consecutive checks over 10 minutes, or latency above two seconds for at least five minutes. A metric-absence condition uses a different kind of window and fires when a monitored time series has no data for a specific retest window, up to a maximum of 24 hours, such as when a resource does not respond to HTTP requests over five minutes.
Exam tip: Filter on monitored-resource labels for stable selection, and do not use asynchronously added resource metadata labels to select time series for sensitive alerts.
Use a metric-threshold condition in Cloud Monitoring for most resource metrics such as CPU utilization, memory usage, disk I/O and storage utilization, use a metric-absence condition when silence means failure such as a resource receiving no HTTP requests over five minutes, use a Prometheus Query Language (PromQL) condition when the rule needs a ratio of two metrics or a dynamic limit, and use a forecasted metric-value condition when early action can prevent exhaustion such as disk space likely to reach 80 percent usage within 24 hours. A metric-absence condition fires when a monitored time series has no data for a retest window up to 24 hours, while a forecasted metric-value condition predicts a threshold violation within a forecast window of 1 hour to 7 days.
Filter Cloud Monitoring alerting policies on labels defined on monitored-resource types because that selection is more stable and reliable, especially for high-cardinality metrics where many unique label combinations can return many points. For example, if a monitored resource includes a cluster label and the metric includes a similar kubernetes_cluster_name label, filter on the resource cluster label. Do not rely on system-defined or user-defined resource metadata labels to select time series for sensitive alerting policies because those labels are added asynchronously and can cause erratic triggering if their population is delayed.
Essential Contacts designates custom points of contact for critical billing, security and legal notifications, and it depends on Resource Manager for scope so notification settings configured on a parent container cascade to child resources. Resource Manager organizes resources as Organization as the root node to Folders as grouping mechanisms to Projects as base-level containers to Resources that each belong to exactly one project, so a contact set at the organization or folder level automatically applies to everything below it. Reliable delivery depends on configuring a group email alias at the organization or folder level instead of relying on default Identity and Access Management (IAM) fallback roles, because fallback delivery only occurs when no custom contact is defined and a group alias survives individual staffing changes.
Your operations team is configuring a Cloud Monitoring alerting policy to detect when virtual machine (VM) instances experience sustained high CPU utilization.
You must satisfy the following operational requirements:
How should you configure the alerting policy?
Last updated