Metrics, Dashboards and Alerts in Cloud Monitoring

Correlating metrics in Cloud Monitoring (opens in a new tab) is the practice of querying related time series together to trace visible degradation back to its origin. No single chart explains an outage, so analysis compares frontend signals such as latency and errors against backend signals such as CPU utilization, memory pressure, and storage errors over the same time window. Confidence comes from aligning timing, scope, and distribution across those signals rather than from any single spike.

1.1Metric Sources, Scope, and Retention That Feed Correlation

The correlated data set in Cloud Monitoring comes from several collection paths that are stored as numeric time series in a Google Cloud project. System metrics from Google Cloud services are collected automatically, while deeper guest and application signals require agents or instrumentation. All sources share the same time-series model and metrics scope, which makes them comparable in Metrics Explorer.

• Google Cloud service metrics: automatically collected system metrics, such as more than 25 Compute Engine metrics reported per virtual machine (VM) instance.
• Ops Agent metrics: CPU, disk, memory, and swap from Linux and Windows Compute Engine VMs, plus third-party telemetry from plugins such as Apache, Nginx, MongoDB, and PostgreSQL.
• Prometheus metrics: application and Kubernetes signals collected by Google Cloud Managed Service for Prometheus, or by the Ops Agent through the Prometheus receiver or OTLP receiver.
• User-defined metrics: application-specific measurements instrumented with OpenTelemetry or written through the Cloud Monitoring API.
• Log-based metrics: numeric counts derived from Cloud Logging, including Google-defined error counts and custom counts such as 404 responses from App Engine.
• External metrics: signals defined by open-source libraries or third-party providers.

The metrics scope of a project controls which time series can be charted together. By default the scope limits a project to time series stored by that project, and the scope must be expanded to include time series stored by other projects for multi-project correlation. Cloud Monitoring retains metrics for several weeks for real-time health checks and recent investigations, while longer-term storage supports comparison across outages, peaks, and architectural changes.

1.2Time-Series Labels and Kinds That Enable Slicing

A Cloud Monitoring time series is identified by what is measured, where it was measured, and how adjacent points relate. The metric.type field names the service and measurement, such as compute.googleapis.com/instance/cpu/utilization, and the labels field carries key-value pairs such as instance_name or instance_id. Cloud Monitoring writes one time series for each combination of resource and metric label values, so filtering by one label isolates a single VM while grouping by another label aggregates across machine types.

• GAUGE: stores the value at a moment in time, such as current CPU utilization.
• CUMULATIVE: stores the accumulated total at a moment in time, such as a vehicle odometer.
• DELTA: stores the change over a period, such as gains or losses since the prior summary.

Interpreting kind correctly prevents false causation because a rising CUMULATIVE line may reflect normal accumulation while a rising GAUGE line reflects growing pressure. The valueType field defines the measurement type as INT64, DOUBLE, BOOL, STRING, or DISTRIBUTION, and DISTRIBUTION values preserve the full spread needed for percentile analysis. Grouping and aggregation then turn labeled series into comparable signals, such as average CPU utilization grouped by machine_type plotted beside request latency for the same interval.

1.3Building and Refining Queries in Metrics Explorer

Metrics Explorer is the chart service for quickly visualizing time-series data without building a full dashboard first. The explorer selects a monitored resource type and metric, then refines the view with filtering, grouping, and aggregation before overlaying additional metrics on the same timeline. Charts can be saved to a custom dashboard once a useful correlation view is found.

1. Select resource and metric: choose the Configuration tab, select a resource type such as Cloud Run service or Compute Engine VM, and select a metric.
2. Refine scope: use Filter to isolate one instance or service, use Group By to split by resource or metric labels, and use Aggregator to align statistics across series.
3. Overlay and persist: add related metrics to the same chart, compare against previous periods, show outliers and percentiles, and save the chart to a custom dashboard.

Google Cloud-managed dashboards are created automatically for instantiated resources and for App Hub applications, services, and workloads. Custom dashboards are required when managed views do not expose the needed trend, outlier, or multi-metric comparison, and custom dashboards can combine metric charts and tables with log data, error groups, alert charts, alert status, text, and events such as reboots or crashes.

1.4Matching Metric Kind to Visualization Goal

Differentiating metric kinds means matching the visualization to whether the goal is state, throughput, or latency. Config Sync (opens in a new tab) custom metrics provide the working example, using the form custom.googleapis.com/opencensus/config_sync/METRIC where METRIC is a name such as reconciler_errors or reconcile_duration_seconds. Metrics without data do not appear in Metrics Explorer, so an absent metric means no samples were written for the selected time and filters, not necessarily zero.

Filtering turns a fleet-wide metric into a diagnostic signal. Almost all Config Sync metrics carry a reconciler label, so filtering reconciler = root-reconciler isolates one RootSync loop while ns-reconciler-NAMESPACE isolates one RepoSync loop. The reconciler_errors metric adds a component label that identifies where in the import, source, and sync pipeline the error occurred. A healthy pattern shows last_sync_timestamp advancing, ready counts tracking declared counts, and error counters flat; a break in that relationship points to the specific object and pipeline stage.

1.5Connecting Infrastructure Signals to User Health

Root-cause analysis connects low-level saturation to user-facing service level objectives (SLOs) through latency, traffic, errors, and saturation. No single metric reliably indicates an unhealthy service, so Cloud Monitoring encourages combining signals and creating custom metrics that capture the joint condition. Distribution matters because averages can hide wide imbalances across instances, zones, or operations.

Correlation paths follow the dependency from the user-facing request layer to the storage and processing layers that complete the work. Traffic flows from the request-response service to the data service it depends on, and degradation propagates in the same direction.

Metric querying with Prometheus Query Language (PromQL) and Monitoring Query Language (MQL) is how Cloud Monitoring turns stored time series into dashboard charts and alerting policies. Every chart requires a query that selects a metric type, a monitored-resource type, and filters, and Metrics Explorer and custom dashboards provide menu-driven, PromQL, and Monitoring-filter interfaces for that query. Google Cloud Managed Service for Prometheus stores Prometheus metrics in Monarch, the same global backend as Cloud Monitoring, so both native and Prometheus metrics are queryable with PromQL.

2.1Query Paths for Dashboard Charts

Query paths for dashboard charts are the three ways Cloud Monitoring selects time-series data for a widget. Menu-driven queries are the default and preserve selections when switching interfaces, while PromQL queries embed selection, grouping, and alignment in one expression and Monitoring-filter queries select time series without grouping or alignment. Switching to or from PromQL or a Monitoring filter generally discards unsaved queries because conversion is not guaranteed.

PromQL and Monitoring-filter queries both require numeric time series for console charts. The Google Cloud console displays only numeric data, so metrics with String type must be retrieved through the API, such as with timeSeries.list.

2.2Visualizing with PromQL in Cloud Monitoring

Visualizing with PromQL in Cloud Monitoring uses a text query editor that provides syntax checks and suggestions. Data flows from Prometheus exporters to collectors that scrape local targets and forward samples to Monarch, then Monarch executes the PromQL query and unions results across all Google Cloud regions and across up to 3,500 Google Cloud projects in a metrics scope. Because evaluation happens in Monarch, the same PromQL works in the Cloud Monitoring UI, in Grafana, and in any tool that can call the Prometheus query API.

PromQL can query Prometheus metrics and native Cloud Monitoring data without separate ingestion. PromQL can query over 6,500 free Cloud Monitoring metrics, including free Google Kubernetes Engine (GKE) system metrics, plus custom metrics and log-based metrics. For example, avg(compute_googleapis_com:instance_cpu_utilization) charts average VM CPU utilization, while up{job="postgresql", cluster="CLUSTER_NAME", namespace="NAMESPACE_NAME"} filters the Up gauge to a specific target. Monarch retains Prometheus data for 24 months, and existing Grafana dashboards continue to work because metric names and PromQL semantics remain compatible.

Adding a PromQL widget to a custom dashboard follows an ordered flow:

1. Open Dashboards: go to Dashboards in Cloud Monitoring, select the project, then create a new dashboard or open an existing dashboard.
2. Add Metric widget: click Add widget and select Metric for a line chart or another visualization.
3. Switch to PromQL: in the query-builder toolbar select the code PromQL button, and optionally disable Auto-run while editing.
4. Run query: enter the PromQL expression and click Run query to render the chart.

2.3Prometheus Data Model and Resource Filtering

The Prometheus data model in Cloud Monitoring centers on the prometheus_target monitored-resource type, which is the resource under which Prometheus time-series data is written. Monitoring queries for Prometheus metrics that are not written in PromQL must specify this resource type, while PromQL queries address the same data by metric name and label selectors. Filtering on monitored-resource labels rather than metric labels is recommended for high-cardinality metrics, such as using the resource cluster label instead of a metric kubernetes_cluster_name label.

The prometheus_target resource exposes labels set during collection that support filtering and grouping:

• project_id: identifier of the Google Cloud project associated with the resource
• location: Google Cloud region where the data is stored
• cluster: GKE cluster or related concept, which might be empty
• namespace: GKE namespace or related concept, which might be empty
• job: job label of the Prometheus target, which might be empty for rule-evaluation results
• instance: instance label of the Prometheus target, which might be empty for rule-evaluation results

2.4Managing Dashboards as Code and Reusable Templates

Managing dashboards as code is the practice of defining, versioning, and deploying custom dashboards as JSON through the Cloud Monitoring API and Google Cloud command-line interface (CLI). The API and CLI support bulk operations through scripts, which makes it practical to manage many dashboards at once, install curated definitions from GitHub, import Grafana dashboards, or install dashboards from templates. The API is also required for capabilities the console does not expose, such as adding the blank placeholder widget and embedding MQL or PromQL queries directly in widget definitions.

Reusable definitions come from the template library in the console and JSON files in the monitoring-dashboard-samples repository on GitHub. Installing from the library uses Dashboard Templates and Add dashboard to your list, while installing from GitHub uses Create Dashboard, Settings, JSON, JSON Editor, and File Upload. Some GitHub templates depend on user-defined metrics existing first, so installation fails until the required metrics and their data are present. A standard reuse workflow preserves order:

1. Select scope: select the Google Cloud project whose data will be shown, or for App Hub select the host or management project.
2. Install base: install a template from the library or upload a JSON file into the JSON Editor.
3. Organize: set the display name and labels so the dashboard can be found by list filtering.
4. Adapt: customize widgets and pin event choices if events must persist, then save.

2.5Dynamic Filtering and Secure Dashboard Sharing

Dynamic filtering for multi-resource analysis limits the time-series data shown by every chart on a dashboard without duplicating the dashboard for each resource. The filter is stored in the dashboard JSON definition in the dashboardFilters array, and when a viewer selects values for zone, instance, cluster, or other resource labels, Cloud Monitoring re-issues queries for all charts with that constraint. Dashboard filters typically filter both time-series data and event annotations, so log-derived events and Service Health events stay consistent with the metrics in view.

Viewing and managing dashboards depends on project-level roles. Monitoring Viewer (roles/monitoring.viewer) or basic Viewer (roles/viewer) views dashboards and metrics, while Monitoring Editor (roles/monitoring.editor) views and creates dashboards and alerting policies. Logs Viewer (roles/logging.viewer) views events from non-data access logs, Private Logs Viewer (roles/logging.privateLogViewer) views events from any logs including private logs, and Personalized Service Health viewer (roles/servicehealth.viewer) views Service Health events. In Google SecOps (opens in a new tab), each action depends on its own chronicle.nativeDashboards permission for list, create, update, delete, get, and duplicate.

A predefined dashboard is automatically configured, managed by Monitoring, and cannot be modified directly, but it can be copied into a custom dashboard that the team owns, edits, and shares.

2.6Playbook Dashboards with Logs, Errors, and Trends

Incident playbook dashboards place critical metric charts together with log, error, and trend widgets so responders see a symptom and its evidence without switching tools. A logs panel widget shows one log entry per row with severity, timestamp, and summary, and supports the same Logging query language as Logs Explorer. The widget requires explicit Scope by Project or Scope by Storage selection of at least one project or log view, supports up to 100 widgets per dashboard, and displays only entries the viewer has permission to see.

Error-reporting panel widgets display recent error groups alongside metrics and logs, answering which distinct failures are active rather than showing every raw line. Observability Analytics chart widgets display SQL query results against logs in a log bucket to reveal trends such as counts by severity or service. Together the three widgets form a diagnostic sequence where data flows from trend signal to error grouping to individual log evidence, all scoped to the same incident window. The predefined Logs Dashboard starts triage with severity and error pairs for GKE, Compute Engine, App Engine, Cloud Load Balancing, Cloud SQL, and BigQuery, and copying it creates a custom starting point for a production playbook.

SLO-based alerting in Cloud Monitoring (opens in a new tab) warns when a service is consuming its error budget too fast and is on track to violate a Service Level Objective (SLO). The alert monitors the burn rate for a chosen SLO and fires when the rate stays above a threshold for an allowed duration. This approach detects both sudden outages and slow drifts before the compliance period ends.

3.1Service Level Indicators, Objectives, and Error Budgets

A Service Level Indicator (SLI) is a measurement of service performance derived from metrics such as request counts and response latencies. Common examples are an availability SLI, defined as the ratio of successful responses to all responses, and a latency SLI, defined as the ratio of calls below a latency threshold to all calls. SLIs can be request-based, which count atomic units like successful HTTP requests, or windows-based, which count time windows where a goodness criterion was met.

A Service Level Objective (SLO) is a target value for an SLI measured over a compliance period. Each SLO combines an SLI, a performance goal such as 95% or 99%, and a compliance period such as a rolling 30 days or a calendar week. You can create up to 500 SLOs for each service in Cloud Monitoring, and a useful SLO targets less than 100% because the gap from perfect performance creates the error budget. The error budget is (1 − SLO goal) × eligible events, so a 95% SLO allows 5% of events to fail before the goal is missed.

3.2Burn-Rate Calculation and Fast-Burn Versus Slow-Burn Alerts

The burn rate tells how fast the error budget is being consumed, based on eligible events and error events received. If no errors occur, no budget is consumed and the burn rate is zero. The metric is normalized so a burn rate greater than 1 means the service will be out of SLO for a future compliance period if the measured error rate is sustained. The burn-rate metric is retrieved by the time-series selector select_slo_burn_rate.

A complete error-budget strategy uses two related policies to balance speed against false positives. The lookback period is also used as the compliance period for calculating SLO performance inside that alert, and alert compliance periods greater than 24 hours are not supported.

The fast-burn alert uses a high threshold because only a large deviation justifies paging on a short window. The slow-burn alert uses a lower threshold over a longer window because even a small sustained excess will eventually consume the whole budget. Shorter windows detect problems faster, but large daily swings can make alerts overly sensitive during low-traffic periods, so using a threshold significantly larger than 1 reduces that sensitivity.

Creating a burn-rate policy follows an ordered pattern: identify the SLO identifier, construct the condition with selector, threshold, comparison, and duration, select a notification channel, and add documentation that tells responders what triggered the policy and what to do.

3.3Metric, Log, Forecast, and Query-Based Conditions

Advanced alerting defines precise conditions over time-series or log data that create an incident and route notifications. Data flows from the monitored resource to the alerting policy condition, then from the condition to an incident record, and finally from the incident through notification channels to people or systems. An alerting policy continuously evaluates after it is enabled and cannot be scheduled to run only during certain hours.

Log data uses two complementary paths. Log-based policies fire when an individual log entry matches a phrase, while SQL-based policies periodically analyze aggregations over many entries, such as more than 50 entries with severity WARNING in 60 minutes. A SQL-based policy must contain only one condition, runs with a five-minute indexing delay plus up to two minutes propagation, executes on BigQuery on-demand or reservation slots, and fails when the Monitoring Service Account is missing, execution exceeds five minutes, or an internal error occurs.

3.4Aggregation, Alignment, and Fatigue Reduction

Aggregation and alignment settings determine which time series enter a condition and how their points are combined before the threshold is tested. Filters remove unwanted time series, the rolling window and rolling window function align points within each series over time, and the across-time-series aggregation and group-by combine series by labels such as zone or priority. A BigQuery example groups by priority, applies 99th percentile across series, and uses a 5 m rolling window with sum to test whether execution time stays above 60 seconds.

Reliable selection depends on using labels defined by monitored-resource types rather than system- or user-defined resource metadata labels. Resource metadata labels populate asynchronously and do not follow the same latency expectations, so a sensitive policy that filters on metadata can behave erratically. When a resource label and a metric label carry similar information, such as cluster and kubernetes_cluster_name, filtering on the resource label produces more stable selection. Threshold position, retest window, combiner logic such as OR, incident autoclose duration, and snoozes are the main levers for reducing fatigue without missing failures.

3.5Notification Routing and Policy Lifecycle

Notification routing sends each incident from one alerting policy to one or more notification channels owned by different teams or systems. A policy can attach multiple channels for redundancy, and configuring multiple channel types ensures that an email or chat outage alone does not block response. Supported destinations include email from alerting-noreply@google.com, SMS, Pub/Sub for programmatic fan-out, webhooks for custom destinations such as ServiceNow, Slack, PagerDuty, and Cloud Mobile App push.

When metric data meets the condition, Monitoring creates the incident and sends notifications carrying status, links to logs, a chart of recorded data, labels, and duration. If an incident is open and Monitoring later determines that a metric-based condition is no longer met, Monitoring moves the incident to closed and sends a closure notification when configured. To pause evaluation temporarily, operators create a snooze for a defined period. The creation flow links selection, triggering, delivery, and documentation: select resource, metric, filters, aggregation and alignment; configure condition, threshold, trigger, and retest window; attach channels and closure behavior; set autoclose and add documentation with playbook and dashboard links; then name and create the policy.

3.6Billing Budgets and Programmatic Cost Control

Cloud Billing budgets are alerts-only monitors that track actual and forecasted Google Cloud charges against a planned spend amount. A budget evaluates spend for its defined scope and time period, compares it to threshold rules, and sends notifications to email recipients and to Pub/Sub when a threshold is crossed. Notifications do not stop usage by themselves, so cost control depends on people responding or on automation that subscribes to Pub/Sub.

Budget scope, amount, and time period define exactly what spend is watched. Scope can cover the entire billing account, one or more organizations, folders, or projects, one or more services such as Compute Engine or BigQuery, or resources carrying a specific label such as billingcode. The amount is either a fixed total or based on previous period spend, aided by a 12-month cost trend chart. Because billing reporting lags usage by several hours, budgets should be set below available funds.

Alert threshold rules are percentage-of-budget conditions that fire on actual costs, forecasted costs, or both. By default budgets notify at 50%, 90%, and 100%, and additional thresholds can be added or removed. A $100 budget with a 110% forecasted rule notifies when the projection exceeds $110. Budgets with a custom time range cannot alert on forecasted costs. Delivery uses role-based email to billing admins and project owners, Cloud Monitoring notification channels for custom emails, and Pub/Sub JSON messages with alertThresholdExceeded and forecastThresholdExceeded for automation such as scaling down non-production, capping APIs, forwarding to Slack, or disabling billing on an over-budget project via the Cloud Billing Budget API.

Alert data flows from the monitored system to the third-party incident platform through an ordered chain where each step depends on the previous step. Cloud Monitoring (opens in a new tab) evaluates log or metric data from Cloud Logging and Cloud Monitoring, then sends a payload through a notification channel to email, Slack, PagerDuty, SMS, Pub/Sub, or a webhook. When delivery fails, diagnosis follows the same path in reverse, from the receiver back through the channel, the alert policy, and the originating log.

4.1Alert Flow from Source to External Endpoint

Alert data moves in a fixed order from source event to downstream automation. A threshold deviation writes a structured log such as networkmanagement.googleapis.com/insights_alarm, while a structural change such as a Monitoring Point going offline and online or a route change writes insights_event. The log-based alert policy evaluates the entry and opens an incident when conditions match. The policy then sends the payload through the configured channel, where data flows from the channel to PagerDuty, Slack, webhook, SMS, email, or Pub/Sub. A Pub/Sub topic can then fan out to a Cloud Run function that runs remediation or enrichment logic.

Enriching the payload means ensuring the notification carries the facts needed to triage without opening multiple consoles: affected services and projects, issue description, detection time, unique incident identifier, and links to dashboards or support resources. Predefined alert policy templates simplify configuration for common scenarios such as a Critical Network Alarm, and each policy connects to existing notification channels to deliver the enriched payload.

4.2Troubleshooting Delivery Failures and Network Path Issues

Troubleshooting delivery failures means isolating whether the break is in log generation, policy evaluation, channel configuration, or the network path to the receiver. Cloud Logging audit logs provide the diagnostic record for what was written, what was evaluated, and what access change may have blocked delivery. Diagnosis must check each dependency in order because Cloud Monitoring depends on Cloud Logging for the trigger and the third-party platform depends on the channel for the payload.

• Audit log analysis shows whether the expected insights_alarm or insights_event entry reached Cloud Logging and whether calls such as SetIamPolicy changed access.
• Channel and contact checks verify Essential Contacts group email, webhook URL or PagerDuty service, and Advisory Notifications Viewer role (roles/advisorynotifications.viewer) for console visibility.
• Network path checks use Connectivity Tests to test connectivity across VPC network, peering, and VPN tunnels, including firewall blocks.
• Agent and flow evidence confirms source health by checking whether Monitoring Points report Active or Offline, reviewing VPC Flow Logs, and enabling Firewall Rules Logging for TCP and UDP connections.

4.3Native PagerDuty Integration and Channel Setup

Integrating native incident response platforms automates alert response by sending Monitoring incidents directly to an external on-call service. Cloud Monitoring moves data from an alerting policy to a PagerDuty notification channel to a PagerDuty service, where the event creates a PagerDuty alert that follows that service's escalation policy. This design keeps detection in Google Cloud while paging, escalation, and resolution tracking live in the external platform.

The native PagerDuty channel connects through the PagerDuty Events API v1 integration. The operator first creates the endpoint in PagerDuty and copies the integration key, then creates the channel in Google Cloud, and only after the channel is saved can policies select PagerDuty in Notifications.

1. Create the PagerDuty endpoint: add an Events API v1 integration to a PagerDuty service and copy the integration key.
2. Create the Monitoring channel: in Monitoring Alerting open Edit notification channels, add a new PagerDuty channel, set Display Name to match the PagerDuty service, and paste the key into Service Key.
3. Attach the channel to policy: in each alerting policy select PagerDuty in Notifications and choose the saved configuration.

The integration key is the trust link because possession of the key authorizes Cloud Monitoring to post events into that specific PagerDuty service.

4.4Incident Payload and Severity Mapping to PagerDuty

The PagerDuty payload carries the full Monitoring incident context in a JSON packet with a description string and a details object using schema version 1.2. Data flows from the Monitoring backend to the PagerDuty endpoint, where an opened Monitoring incident triggers a PagerDuty alert and a later resolved or closed incident updates that external alert with end time and state. The details.incident object identifies identity and navigation with incident_id, scoping_project_id, scoping_project_number, and url; lifecycle timing with started_at, ended_at, state, and renotify; affected target with resource, metric, and labels; and policy cause with policy_name, condition, severity, threshold_value, and observed_value.

By default PagerDuty notifications show all Monitoring severity levels as critical, so a low-severity warning and a critical outage escalate the same way without extra configuration. To preserve urgency, configure event orchestration rules in PagerDuty that read event.custom_details.incident.severity and set the corresponding PagerDuty severity before escalation runs:

1. Open orchestration: in PagerDuty go to Services, select the service, open Settings, then Event Management and View Orchestration Rules.
2. Create a conditional rule: click New Rule and choose If events match certain conditions.
3. Match Monitoring severity: enter event.custom_details.incident.severity matches part 'VALUE' for one Monitoring value.
4. Set PagerDuty severity: under actions select Alert Data and set the matching severity, then save.
5. Repeat and chain: create one rule per severity value, plus an Else rule for no severity.

4.5Webhook Channels and Private Delivery Options

A custom webhook notification channel posts alert data as an HTTP request to any public endpoint you control. An alerting policy evaluates a condition, opens an incident, and then Cloud Monitoring sends a notification through the channel to ServiceNow, DataDog, OpsGenie, Rootly, or another custom tool. Use webhooks when no built-in integration exists and you need automated response outside Google Cloud.

Configuration requires a channel first and a policy second. The principal must hold Monitoring Editor (roles/monitoring.editor), and only after the webhook channel exists can it be selected during policy creation. A metric-based workflow selects resource and metric, configures threshold and trigger, attaches channels including the webhook, sets Notify on incident closure only if needed, and adds Documentation runbook text before naming and creating the policy. A log-based path defines an inclusion filter in Logging query language, extracts labels for routing, and sets Time between notifications and Incident autoclose duration. Alerts are subject to limits such as 20 alerts per policy per day per project, and securing delivery depends on narrow filters, documentation, and IAM control of who can create and attach channels.

Formulating effective prompts for Gemini Cloud Assist (opens in a new tab) is the practice of writing precise natural language queries that direct the assistant to correlate metrics, surface anomalies, and propose causes. An effective prompt names the resource scope, the symptom or metric, and the time or location window. That precision lets the assistant pull data from Cloud Monitoring, Cloud SQL insights, and Personalized Service Health instead of returning a generic answer.

5.1Principles of Precise Metric Prompts

Precise metric prompts specify what to measure, where to measure it, and how to rank or slice the result. A prompt such as Which of my VMs have the highest utilization? ranks Compute Engine utilization, while Total VM network bytes sent per second for each instance? requests a per-instance time series breakdown. A prompt such as What are the top 5 dataflow jobs with the highest estimated backlog processing time? adds both a ranking limit and a specific performance signal. Because resource, metric, and grouping are explicit, Gemini Cloud Assist can correlate disparate series and highlight which resource deviates.

Context-aware prompting improves precision when issued from a specific console page. Prompts sent while viewing a Cloud Storage page can reference buckets and objects, prompts from a BigQuery page can reference datasets and jobs, and prompts from a Cloud SQL page can reference that instance's CPU, latency, and error logs. The flow moves from page context to the Cloud Assist panel to underlying telemetry, so the same short question returns a scoped answer. For advanced analysis, always include project, region or zone, and time window when page context does not provide them.

5.2Prompts for Performance, Database, and Incident Hypotheses

Prompts for correlating performance signals ask Gemini Cloud Assist to compare utilization, throughput, and saturation across resources to form a hypothesis. Focused prompts should be issued first and then compared across the same time range:

• Which of my VMs have the highest utilization? to rank compute saturation.
• Total VM network bytes sent per second for each instance? to compare network throughput.
• What are the top 5 dataflow jobs with the highest estimated backlog processing time? to isolate pipeline backlog.
• What's the QPS on my SQL databases? to correlate query rate with latency.
• What are all the quotas for my project that are almost hitting the limit? to test throttling.
• Which of my VMs are idle? and Give me recommendations to reduce my resource costs to separate waste from contention.

Prompts for database investigation link query telemetry with instance load in Cloud SQL. Common patterns include What is the current CPU utilization of this database?, Which of my queries have high latency for this database instance?, What is the slowest query for this database in this instance?, How many error log entries are there for this database instance?, and What issues are affecting my databases?. Prompts for service disruption check Personalized Service Health, such as Is there something going on with Google Cloud?, What's the latest update on the Compute Engine outage in us-central1?, and How is my project impacted by the Cloud Storage incident?.

5.3Validating AI-Generated Insights Against Telemetry

Evaluating AI-generated insights is the discipline of treating AI summaries as hypotheses and validating them against independent telemetry. Metric data flows from Cloud Monitoring, Personalized Service Health, logs, and telemetry views into the Cloud Assist panel and investigations, which then produce natural-language findings and next steps. Because Gemini Cloud Assist is an early-stage technology, its output can seem plausible but be factually incorrect, so every summary must be confirmed in underlying telemetry.

Validation means comparing the AI trend description to what normal looks like. Cloud Hub Health and troubleshooting charts show health metrics for the selected period, by default the last hour, with annotations for recommended events. The same view surfaces Open alerts from metric-based policies and Google Cloud incidents from Personalized Service Health alongside workload charts. A reliable conclusion requires alignment of chart shape, sustained threshold crossings, and historical comparison, not narrative alone.

Anomaly explanations often list events that occurred at the same time, which establishes correlation rather than causation. Causation requires a dependency or mechanism, such as a query that consumed slot-milliseconds and spilled bytes and then increased queue time, a reservation limit that throttled capacity, or a caller identity in Cloud Audit Logs that changed configuration before latency rose. BigQuery INFORMATION_SCHEMA views such as JOBS, JOBS_TIMELINE, RESERVATIONS, CAPACITY_COMMITMENTS, STREAMING_TIMELINE, and TABLE_STORAGE provide that mechanism check.

5.4Investigations That Turn Metric Shifts into Actions

Gemini Cloud Assist investigations is the root-cause analysis tool that turns a troubleshooting chat into a structured, runnable investigation. A prompt becomes an investigation through an ordered workflow: ask Gemini Cloud Assist to investigate specific resources, select Create investigation when the model offers it, refine the pre-populated project, resources, symptoms, and time window, and click Create to review results on the Investigation details page. As of April 10, 2026, creating, running, and editing investigations requires a Premium Support contract or account-team access plus the Investigation Creator or Investigation Owner role.

Scope and telemetry depth limit trustworthiness. Cloud SQL query anomaly detection and AI troubleshooting require Enterprise Plus edition with the new network architecture and query insights enabled, and exclude Enterprise edition, old architecture, VPC Service Controls perimeters, Access Transparency, and read replicas. For Bigtable, the investigation compares baseline aggregated data against current behavior across client latencies, server latencies, CPU utilization, disk utilization, and request count. The Investigation Overview pane then organizes results into Issue for what is investigated with start and stop time, Findings for observations such as incorrect channel pool size, and Hypotheses for recommended actions.

Prioritization orders Hypotheses by severity first and business impact second. Findings describe observations about what changed, while Hypotheses contain the remediation to implement. Every recommendation must be validated against dashboards, logs, and configuration before changing production.

Key Facts

• A metrics scope shows only its own project's time series by default and must be expanded to include time series stored by other projects for multi-project correlation.
• GAUGE stores a point-in-time value, CUMULATIVE stores an accumulated total, and DELTA stores change over a period, while Distribution metrics require heatmaps or percentile lines for latency analysis.
• Google Cloud Managed Service for Prometheus stores Prometheus metrics in Monarch with 24 months retention and supports PromQL across the Cloud Monitoring UI, Grafana, and the Prometheus query API.
• Fast-burn alerts use a 10x baseline threshold with a 1- or 2-hour lookback, while slow-burn alerts use a 2x baseline threshold with a 24-hour lookback, and alert compliance periods greater than 24 hours are not supported.
• Cloud Billing budgets are alerts-only and never stop usage automatically; they send email and Pub/Sub notifications at default thresholds of 50%, 90%, and 100%.