Cloud Monitoring Dashboards and Incident Playbooks

Metric querying and visualizing with Prometheus Query Language (PromQL) and Monitoring Query Language (MQL) is how Cloud Monitoring (opens in a new tab) turns stored time series into dashboard charts and alerting policies. Every chart requires a query that selects a metric type, a monitored-resource type, and filters, and Metrics Explorer and custom dashboards provide menu-driven, PromQL, and Monitoring-filter interfaces for that query. Google Cloud Managed Service for Prometheus (opens in a new tab) stores Prometheus metrics in Monarch, the same global backend as Cloud Monitoring, so both native Cloud Monitoring metrics and Prometheus metrics are queryable with PromQL.

1.1Query Paths for Dashboard Charts

Query paths for dashboard charts are the three ways Cloud Monitoring selects time-series data for a widget: menus, PromQL, and Monitoring filters. Menu-driven queries are the default and preserve selections when switching interfaces, while PromQL queries embed selection, grouping, and alignment in one expression and Monitoring-filter queries select time series without grouping or alignment. Switching to or from PromQL or a Monitoring filter generally discards unsaved queries because conversion between forms is not guaranteed.

The three paths differ in how grouping is configured and what happens on conversion:

PromQL and Monitoring-filter queries both require numeric time series for console charts. The Google Cloud console displays only numeric data, so metrics with String type must be retrieved through the API, such as with timeSeries.list.

1.2Visualizing with PromQL in Cloud Monitoring

Visualizing with PromQL in Cloud Monitoring uses a text query editor that provides syntax checks and suggestions. Data flows from Prometheus exporters to collectors that scrape local targets and forward samples to Monarch, then Monarch executes the PromQL query and unions results across all Google Cloud regions and across up to 3,500 Google Cloud projects in a metrics scope. Because evaluation happens in Monarch, the same PromQL works in the Cloud Monitoring UI, in Grafana, and in any tool that can call the Prometheus query API.

PromQL can query Prometheus metrics and native Cloud Monitoring data without separate ingestion. PromQL can query over 6,500 free Cloud Monitoring metrics, including free Google Kubernetes Engine (GKE) system metrics, plus custom metrics and log-based metrics. For example, avg(compute_googleapis_com:instance_cpu_utilization) charts the average CPU utilization of virtual machine (VM) instances, while up{job="postgresql", cluster="CLUSTER_NAME", namespace="NAMESPACE_NAME"} filters the Up gauge to a specific PostgreSQL target. Existing Grafana dashboards continue to work against Managed Service for Prometheus because metric names and PromQL semantics remain compatible with upstream Prometheus, and Monarch retains Prometheus data for 24 months.

Adding a PromQL widget to a custom dashboard follows an ordered flow from dashboard selection to query execution:

1. Open Dashboards: Go to Dashboards in Cloud Monitoring and select the project, then create a new dashboard or open an existing dashboard.
2. Add Metric widget: Click Add widget and select Metric for a line chart or another visualization from the widget library.
3. Switch to PromQL: In the query-builder toolbar select the code PromQL button, and optionally disable Auto-run while editing.
4. Run query: Enter the PromQL expression in the editor and click Run query to render the chart.

1.3Prometheus Data Model in Cloud Monitoring

The Prometheus data model in Cloud Monitoring centers on the prometheus_target monitored-resource type, which is the resource under which Prometheus time-series data is written. Monitoring queries for Prometheus metrics that are not written in PromQL must specify this resource type, while PromQL queries address the same data by metric name and label selectors. Filtering on monitored-resource labels rather than metric labels is recommended for high-cardinality metrics, such as using the resource cluster label instead of a metric kubernetes_cluster_name label when both capture similar information.

The prometheus_target resource exposes labels set during collection that support filtering and grouping:

• project_id: identifier of the Google Cloud project associated with the resource
• location: Google Cloud region where the data is stored, typically the region of the GKE cluster or Compute Engine instance
• cluster: GKE cluster or related concept, which might be empty
• namespace: GKE namespace or related concept, which might be empty
• job: job label of the Prometheus target, which might be empty for rule-evaluation results
• instance: instance label of the Prometheus target, which might be empty for rule-evaluation results

1.4Managing Dashboards as Code

Managing dashboards as code is the practice of defining, versioning, and deploying custom dashboards as JSON through the Cloud Monitoring API and Google Cloud command-line interface (CLI) instead of only clicking in the console. The API and CLI support bulk operations through scripts, which makes it practical to manage many dashboards at once, install curated dashboard definitions from GitHub, import Grafana dashboards, or install dashboards from templates. The API is also required for capabilities the console does not expose, such as adding the blank placeholder widget, and for embedding MQL or PromQL queries directly in widget definitions.

Custom dashboards complement Google Cloud-managed dashboards and Metrics Explorer in the visualization lifecycle. Google Cloud-managed dashboards are created automatically for instantiated resources and App Hub applications and show curated logs, metric data, and open alerts, while Metrics Explorer supports rapid exploration with comparisons to previous data, outlier and percentile display, and multi-metric charts. Charts built in Metrics Explorer can be saved to a custom dashboard, and custom dashboards can display metric charts and tables, log data and error groups, alerting-policy charts, alert information, text, and operational events such as reboots or crashes.

Dynamic filtering and secure dashboard sharing in Cloud Monitoring (opens in a new tab) is the practice of using one dashboard definition to analyze many resources while controlling who can view, edit, and reuse it. Filters and labels narrow what data and which dashboards are shown, and Identity and Access Management (IAM) roles plus sharing settings enforce the trust boundary around that data.

2.1Dynamic Filtering for Multi-Resource Analysis

Dynamic filtering for multi-resource analysis limits the time-series data shown by every chart on a dashboard without duplicating the dashboard for each resource. The filter is stored in the dashboard JSON definition in the dashboardFilters array, and when a viewer selects values for zone, instance, cluster, or other resource labels, Cloud Monitoring re-issues queries for all charts with that constraint. Dashboard filters typically filter both time-series data and event annotations, so log-derived events and Service Health events stay consistent with the metrics in view. This dependency means one maintained dashboard can serve fleet-wide triage, and changing the filter value changes the source-to-chart flow from the monitored resources to each widget in order.

The dashboard list itself can also be filtered to streamline navigation across many dashboards. On the Dashboards page, list filters surface relevant dashboards through searching and pinning, and deleting those list filters restores the full inventory. List filtering only changes which dashboard names are visible and does not change the data inside a selected dashboard.

2.2Labels and Reusable Dashboard Definitions

Labels and reusable dashboard definitions in Cloud Monitoring provide organization and reuse for multi-resource analysis. Every custom dashboard definition includes a labels object alongside displayName, mosaicLayout, and dashboardFilters, and the install dialog for a template lets the owner set the name and add labels at creation time. User-defined labels can later be added or removed, which allows teams to group dashboards by service, environment, or owner such as Compute Engine, Cloud Storage, or BigQuery. Because labels travel with the JSON definition, exporting and importing that JSON preserves the organizational context when a dashboard moves between projects.

Reusable definitions come from two sources: the template library in the console and JSON files in the monitoring-dashboard-samples repository on GitHub. Installing from the library uses Dashboard Templates (opens in a new tab) and Add dashboard to your list, while installing from GitHub uses Create Dashboard, Settings, JSON, JSON Editor, and File Upload to replace the empty definition. Some GitHub templates display custom metric data and depend on user-defined metrics existing first, so the dashboard installation fails until the required metrics and their data are present. After upload, the owner can customize the title, add a text widget that records the source, delete unwanted charts, or add charts before selecting Save.

A standard reuse workflow preserves order and ownership:

1. Select scope: Select the Google Cloud project whose data will be shown, or for App Hub configurations select the App Hub host project or management project.
2. Install base: Install a template from the library or upload a JSON file into the JSON Editor.
3. Organize: Set the display name and labels so the dashboard can be found by list filtering.
4. Adapt: Customize widgets and pin event annotation choices if events must persist across reloads, then save the dashboard.

2.3Secure Collaboration with IAM Roles and Sharing Controls

Secure collaboration with IAM roles and sharing controls rests on IAM, which decides who can act, and sharing controls, which decide who can see. The viewer must first select the correct Google Cloud project from the console menu bar, because dashboards only show data from the selected project and an incorrect selection appears as missing resources. The selected time range also bounds the query, so a range that is too narrow shows no data even when permissions and collection are correct. The cluster and node service accounts form a separate write boundary and depend on Logs Writer (roles/logging.logWriter) and Monitoring write permissions to deliver data; high error rates on the Enabled APIs and services page signal a missing write role rather than a viewer problem.

Viewing and managing Monitoring dashboards depends on project-level roles:

• View dashboards and metrics: Monitoring Viewer (roles/monitoring.viewer) or basic Viewer (roles/viewer)
• View and create dashboards and alerting policies: Monitoring Editor (roles/monitoring.editor)
• View events from non-data access logs: Logs Viewer (roles/logging.viewer)
• View events from any logs including private logs: Private Logs Viewer (roles/logging.privateLogViewer)
• View Service Health events: Personalized Service Health viewer (roles/servicehealth.viewer)

Visualization workspaces such as Google SecOps enforce the same separation with finer dashboard permissions. Confirm IAM permissions before managing dashboards in that environment, where each action depends on its own permission:

• View dashboards depends on chronicle.nativeDashboards.list
• Create dashboards depends on chronicle.nativeDashboards.create
• Edit and share dashboards depends on chronicle.nativeDashboards.update
• Delete dashboards depends on chronicle.nativeDashboards.delete
• Export, download, or get dashboard data depends on chronicle.nativeDashboards.get
• Duplicate dashboards depends on chronicle.nativeDashboards.duplicate

Sharing controls then set visibility inside the authorized boundary. A dashboard owner can keep a dashboard private during development and change access to shared when the team needs a standard view, which prevents redundant dashboard creation and standardizes reporting.

Predefined or curated dashboards and custom dashboards compose with these controls. A predefined dashboard is automatically configured, managed by Monitoring, and cannot be modified directly, but it can be copied into a custom dashboard that the team owns, edits, and shares. Choosing Customize Dashboard copies the predefined content into a new custom view where widgets can be added, removed, edited, or moved, and that copy then inherits the normal IAM and sharing lifecycle for secure collaboration.

Incident mitigation integration and diagnostic playbooks is the practice of placing critical metric charts together with log, error, and trend widgets on one custom dashboard (opens in a new tab) so responders can see a symptom and its diagnostic evidence without switching tools. The dashboard queries Cloud Logging storage directly, so the same context that shows a latency spike or error-rate increase also shows the related log entries, grouped errors, and SQL-derived trends. This layout turns the dashboard from a display into an actionable playbook where each visual panel maps to the next troubleshooting step.

3.1Logs Panel Widgets for Aligned Metrics and Logs

Logs panel widgets display individual log entries alongside metric data on a custom dashboard. A logs panel widget shows one log entry per row with columns for severity, timestamp, and a summary, and clicking View log details opens the JSON-formatted representation of that entry. The widget supports the same Logging query language as Logs Explorer, and both surfaces display only the log entries that the viewer has permission to view based on Identity and Access Management (IAM) roles on the resources that store the logs.

The logs panel widget does not replace Logs Explorer. Logs Explorer remains the standard investigation tool, while the widget brings a filtered view of those same entries into the metric context so responders can correlate a chart anomaly with the events that occurred at the same time. From the widget, an operator can pivot to Logs Explorer for full investigation, which preserves a direct link from the dashboard playbook step to Cloud Logging.

Adding a logs panel widget to a dashboard follows an ordered flow from dashboard selection to scope configuration:

1. Select project and dashboard: Select the Google Cloud project, then select an existing dashboard or click Create dashboard.
2. Add Logs widget: Click Add widget and select Logs, with an optional Widget title.
3. Configure search scope: Set Scope by to Project and explicitly select at least one project, or set Scope by to Storage and explicitly select at least one log view.
4. Define query and filters: Enter a Logging query, add variables and filters, and apply the configuration to render matching entries.

No projects or log views are automatically selected, so a widget with no explicit selection returns no data. A dashboard supports up to 100 widgets, and the feature is supported only for Google Cloud projects, with App Hub configurations using the App Hub host project or management project.

3.2Error and Trend Widgets for Diagnostic Steps

Error and trend widgets for diagnostic steps add grouped failures and aggregated log trends next to metrics and raw logs. Error-reporting panel widgets display recent error groups on the dashboard alongside metrics and logs. An error group collects related errors as defined by Error Reporting, so the widget answers which distinct failures are active rather than showing every raw log line. When a metric chart shows elevated failures, the adjacent error panel identifies the dominant group to investigate first, and the adjacent logs panel then shows the detailed entries for that group.

Observability Analytics chart widgets display the results of a SQL query against logs stored in a log bucket to reveal trends in log data. The SQL chart aggregates events over time, such as counts by severity or service, while the logs panel shows the individual entries behind the trend. Together the three widgets form a diagnostic sequence where data flows from trend signal to error grouping to individual log evidence, all scoped to the same incident window.

3.3Permissions and Search Scope as Playbook Guardrails

Permissions and search scope as playbook guardrails control dashboard playbook visibility through IAM roles and log scopes that define which projects and storage the widgets can query. The logs panel widget queries the projects and log views listed in the default log scope, and if that scope is not accessible it searches the log entries that originate in the selected project, including entries routed to the project by a sink in another project and stored in a log bucket. IAM on the storage resources acts as the trust boundary, so two responders viewing the same dashboard can see different entries.

The three roles that gate playbook configuration and viewing are:

• Monitoring Editor (roles/monitoring.editor) on the dashboard project to add and configure the widget.
• Logging Viewer (roles/logging.viewer) on each project whose log entries must be viewed, which still does not grant access to every log entry.
• Logs View Accessor (roles/logging.viewAccessor) on each project that stores the log bucket for a log view that must be accessed, which grants access to all log views stored by that project.

Scope configuration determines which side of the storage boundary the widget reads from. Scope by Project searches selected projects in the current metrics scope and warns when the viewer lacks project-level permission to view logs in a project. Scope by Storage searches selected log views on log buckets, and views in other projects can be added only by importing a project that is already included in the current metrics scope.

3.4Predefined and Custom Logs Dashboards for Triage Playbooks

Predefined and custom logs dashboards for triage playbooks start from the Logs Dashboard page, which provides a predefined triage view that detects trends in system behavior by resource type. The predefined dashboard is automatically configured and managed by Cloud Monitoring, and it shows data only for resources in the selected Google Cloud project that are producing log data. When no project is selected, the dashboard shows a loading-data spinner instead of charts.

The predefined dashboard is organized as a pair of charts for each of six resource types: Google Kubernetes Engine, Compute Engine, App Engine, Cloud Load Balancing, Cloud SQL, and BigQuery. One chart in each pair displays the number of logs and their severity types such as WARNING, and the other chart displays logs that contain errors. Holding the pointer over a chart reveals more information about severity types or error logs.

The two dashboard types differ in management and playbook use:

A custom dashboard begins as a copy of the predefined dashboard, so the severity and error pairs are preserved as the starting diagnostic steps. The operator then adds or removes widgets, edits existing widgets, or moves widgets to align the critical metric displays with the specific troubleshooting sequence for that service or environment.

Key Facts

• Google Cloud Managed Service for Prometheus stores Prometheus metrics in Monarch and supports PromQL across the Cloud Monitoring UI, Grafana, and the Prometheus query API with 24 months retention.
• Non-PromQL queries for Prometheus metrics must specify the prometheus_target monitored-resource type, while PromQL queries select the same data by metric name and labels.
• A dashboardFilters entry inside dashboard JSON filters the data and event annotations shown by all charts, while dashboard list filtering only changes which dashboard names are visible.
• A logs panel widget requires explicit selection of at least one project or one log view and displays only entries the viewer has permission to see under Logging Viewer or Logs View Accessor.
• A predefined dashboard is managed by Monitoring and cannot be modified directly, but choosing Customize Dashboard copies it into an editable custom dashboard.