Prepare and test your skills
Prepare and test your skills
Custom metric pipelines in Cloud Monitoring (opens in a new tab) move data from writers to readers through an ordered chain. Data flows from an instrumented application, the Ops Agent, Cloud Logging for log-based metrics, or Google Cloud Managed Service for Prometheus into time series stored against a monitored resource. A query in Metrics Explorer, a custom dashboard, or an alerting policy then reads those time series, and an alerting policy sends notifications to channels when its threshold condition is met. Troubleshooting the pipeline means checking each link in order: whether writes were accepted, whether the query names the metric correctly, and whether permissions and latency allow the data to appear.
Querying custom metrics means selecting time-series data to chart or to evaluate in an alert, using a menu-driven builder, Prometheus Query Language (PromQL), or a Monitoring filter expression. The Google Cloud console provides menu selections plus a PromQL query editor with syntax checks and suggestions, while the Cloud Monitoring API (opens in a new tab) supports PromQL and Monitoring filter expressions. Monitoring Query Language (MQL) uses a piped-query model that is more expressive than Monitoring filters, but customer support for MQL ended on July 22, 2025, and MQL is no longer available in the console for new charts, dashboards, and alerting policies. Existing MQL charts, dashboards, and alerting policies continue to work, and MQL objects can still be created through the Cloud Monitoring API.
| Query option | Where new queries can be created | Behavior in Cloud Monitoring |
|---|---|---|
| Menu-driven builder | Console Metrics Explorer and dashboards | Selects resource type, metric type, filters, aggregation, and display without code |
| PromQL | Console editor and Cloud Monitoring API | Queries Prometheus and mapped Monitoring metrics; supports PromQL alerts and alert rules |
| Monitoring filter | Console and Cloud Monitoring API | Text expression for selecting time series used by charts and API calls |
| MQL | Cloud Monitoring API only for new objects | Piped queries for time-series data; existing console objects still function |
PromQL queries for Monitoring metrics must use mapped metric names because Monitoring metric types contain slashes and dots that PromQL does not allow. The mapping replaces the first / with : and replaces all other special characters, including . and remaining / characters, with _, so workload.googleapis.com/otlp.test.gauge becomes workload_googleapis_com:otlp_test_gauge. When a custom metric can be written against more than one monitored-resource type, the PromQL query must also filter for the resource, for example {monitored_resource="gce_instance"} for a gauge written against gce_instance. Non-PromQL queries for Prometheus metrics must specify the prometheus_target monitored-resource type, whose labels define the source and destination context for filtering and grouping.
The prometheus_target resource carries labels that identify where the Prometheus data was collected and stored:
project_id identifies the Google Cloud project associated with the resource.location identifies the region where the data is stored, typically the GKE or Compute Engine region.cluster identifies the GKE cluster or related concept and may be empty.namespace identifies the GKE namespace or related concept and may be empty.job identifies the Prometheus job label and may be empty for rule-evaluation results.instance identifies the Prometheus instance label and may be empty for rule-evaluation results.Exam tip: Add {monitored_resource="gce_instance"} when a custom metric supports multiple resource types; omit the resource filter only when the metric can be written against a single monitored-resource type.
An alerting policy in Cloud Monitoring is a configuration resource that defines the criteria for generating an alert or incident and how to notify responders when that criteria is met. A metric-threshold policy sends notifications when values are more than or less than a specific threshold, such as CPU utilization exceeding 80%, while the API can also define a policy based on the ratio of two metrics, which cannot currently be created in the console. Cloud Monitoring also supports alerting policies based on PromQL queries and Prometheus alert rules, so the same mapped metric names and resource filters used in charts apply to alerts. Policies can be created and managed through the console, the Cloud Monitoring API, the Google Cloud CLI, or Terraform, and integrations may offer pre-configured recommended alerts where only notification recipients need to be added.
Each notification from an alerting policy links to a persistent alert or incident record used for troubleshooting the failure. The alerting policy depends on two other objects: the query that selects the custom or log-based metric time series, and the notification channel that delivers the notification to email, the Cloud Mobile App, PagerDuty, Slack, or another supported endpoint.
The incident record preserves the failure context after the notification is sent:
Diagnosing a custom metric pipeline starts with Metrics Management, which reports ingestion, cardinality, reads, policy use, and write errors for billable metrics. The Metrics Management page shows data for the previous one day by default and helps control cost without losing observability by identifying high-volume or high-cardinality metrics. Metric-write errors, ingestion volumes for byte- and sample-based billing, and use in alerting policies and custom dashboards all point to whether the problem is on the write path, the query path, or the cost-control path. Unneeded metrics can be excluded from ingestion to eliminate their cost.
The Metrics Management page reports diagnostic signals that separate ingestion, query, and dependency failures:
Permission constraints commonly break the write path because the identity writing time series must hold a writer role for that signal. The Compute Engine service account used by the Ops Agent by default receives roles/logging.logWriter and roles/monitoring.metricWriter but not trace writer roles, so missing traces require roles/cloudtrace.agent or roles/cloudtrace.admin, checked with gcloud projects get-iam-policy. The same principle applies to user-defined metrics (opens in a new tab): without roles/monitoring.metricWriter, timeSeries.create calls fail, while metricDescriptors.create and metricDescriptors.delete succeed only for user-defined metrics and never for built-in metrics. Latency and retention complete the diagnosis because newly written points are not immediately queryable and data is kept only for the retention period in quotas and limits; data that must survive longer has to be copied manually to Cloud Storage or BigQuery, and multi-project visibility additionally depends on the metrics scope of the scoping project that stores alerts, dashboards, and synthetic monitors.
Instrumenting an application for custom metric collection means adding code to measure application-specific behavior and sending those measurements to Cloud Monitoring (opens in a new tab) as user-defined metrics. User-defined metrics cover any metric not defined by Google Cloud, including metrics defined by third-party applications, and they capture data that built-in metrics cannot, such as how many background routines an application spawned or how often an internal function is called. Once stored in Cloud Monitoring, user-defined metrics work like built-in metrics for charting, alerting, grouping, and filtering.
The recommended path for custom metric collection is the vendor-neutral OpenTelemetry standard rather than vendor-specific client libraries. An OpenTelemetry-instrumented application sends metrics through an OpenTelemetry Protocol (OTLP) pipeline to Cloud Monitoring, which keeps the application portable across backends. Direct use of the Cloud Monitoring API remains supported for creating metric descriptors and writing time-series points.
Common collection paths from an instrumented application to Cloud Monitoring include:
timeSeries.create calls that write points directly from application codeData flows from the instrumented application to the collector or agent, and then from the collector or agent to the Cloud Monitoring backend where time series are stored per Google Cloud project. The collection overview (opens in a new tab) groups these paths by environment, so Compute Engine uses the Ops Agent, Kubernetes uses Managed Service for Prometheus, and Cloud Run uses a sidecar collector.
A MetricDescriptor is the Cloud Monitoring data structure that defines how metric data is organized, including the metric type name, description, unit, labels, metric kind, and value type. Cloud Monitoring can create the descriptor implicitly from the first written data points, or the application owner can create it explicitly with metricDescriptors.create before writing data. Explicit creation gives tighter control over labels, kind, value type, and documentation.
The metric type name must be unique among user-defined metrics in the project and must use a user-defined prefix followed by a descriptive path, such as custom.googleapis.com/stores/daily_sales. Supported prefixes are custom.googleapis.com/, workload.googleapis.com/, external.googleapis.com/user, and external.googleapis.com/prometheus. A type that begins with custom or external signals a custom or third-party metric, while a type such as compute.googleapis.com/instance/cpu/utilization signals a Google Cloud system metric. The custom metric creation guide (opens in a new tab) shows the same naming, kind, and value-type fields used for explicit descriptor creation.
Metric kind defines the relationship between adjacent measurements in a time series, and value type defines the data type of each measurement:
| Metric kind | What it stores | Example from documentation |
|---|---|---|
| GAUGE | Value at a moment in time | Hourly temperature record |
| CUMULATIVE | Accumulated total at a moment in time | Vehicle odometer reading |
| DELTA | Change over a specified period | Stock gains or losses for a period |
Choose GAUGE for point-in-time state such as queue depth or daily sales, CUMULATIVE for monotonically growing counters such as total calls, and DELTA for per-interval change. Value type must be one of INT64, DOUBLE, BOOL, STRING, or DISTRIBUTION, with DOUBLE common for measurements such as sales in {USD} and DISTRIBUTION used for latency or size distributions that support percentile analysis. The sample sales descriptor, for example, combines type custom.googleapis.com/stores/daily_sales, kind GAUGE, value type DOUBLE, unit {USD}, and a store_id STRING label.
A Cloud Monitoring time series is one unique combination of monitored-resource values plus metric-label values, and Cloud Monitoring writes one time series for each combination it receives. Labels therefore control both query power and resource consumption, because each new label value creates additional active time series that must be stored, charted, and paid for as a chargeable feature. User-defined metrics (opens in a new tab) are subject to quotas and retention limits, and data that must live beyond retention must be manually copied to another location such as Cloud Storage or BigQuery.
The design tradeoff appears in a program that calls auxiliary programs A and B and needs separate alerts when A exceeds 10 calls per minute and B exceeds 5 calls per minute:
A and one counting calls to B, produce one time series each and allow simple alert conditions without label filters.A and one with label value B, and requires alert filters on the label value but makes ratio calculations simple.A and B cannot be separated.Prefer separate metric types when the activities are unrelated and need independent alerting, and prefer a shared metric with a low-cardinality label when comparison, grouping, or ratio math matters. Avoid unbounded label values such as user IDs, request IDs, or full URLs, because high cardinality multiplies time-series count, increases cost and query load, and can hit quota limits.
Every custom-metric point in Cloud Monitoring must attach to a monitored resource that identifies where the code is running. Writing against the same resource that produced related system metrics gives better performance than funneling everything through one resource, avoids out-of-order writes caused by multiple processes updating the same time series, and allows grouping custom data with other data from that resource. The write path depends on the Cloud Monitoring API, where timeSeries.create, metricDescriptors.create, and metricDescriptors.delete apply only to user-defined metrics, while timeSeries.list and descriptor get and list operations apply to both built-in and user-defined metrics.
The global resource has only a project_id label, so many writers sharing one global object in the same project can collide and overwrite each other's data. The generic_task and generic_node resources are safer fallbacks when no specific resource fits, with generic_task suited to task-like applications and generic_node suited to node-like VMs, and both provide multiple common labels for building unique resource objects that work well in filters and aggregations. Choose a specific resource such as a VM, container, or Cloud Run service when available, fall back to generic_task or generic_node for custom workloads, and reserve global for truly project-wide single-writer signals.
Exam tip: Use generic_task or generic_node with distinct label values for multi-source custom metrics; shared use of the global resource with only project_id risks time-series collisions and overwrites.
Log-based metrics in Cloud Logging (opens in a new tab) turn log entries into numeric time series stored in Cloud Monitoring. The Log Router evaluates each incoming log entry against a Logging filter expression, and when an entry matches, Logging extracts a count or numeric value and writes it as metric data. This pipeline lets teams observe error rates, request counts, and latencies without changing application code, then chart the results and alert on them in Monitoring.
The Log Router is the component that decides whether a log entry contributes to a log-based metric. Data flows from the log entry to the Log Router to the log-based metric to Cloud Monitoring, and a match is counted only when all routing conditions are met.
Project-scoped metrics count logs routed through that project, while bucket-scoped metrics are calculated from all logs destined for that bucket regardless of where they originated. Data for a user-defined log-based metric comes only from log entries received after the metric is created, so a metric is never backfilled from older logs. System log-based metrics are calculated from included logs only, while user-defined log-based metrics are calculated from both included and excluded logs.
Log-based metrics support three data types, but user-defined log-based metrics can be only counter or distribution types. Most system-defined log-based metrics are counters, while some are boolean type, which captures whether a log entry matches a filter. The choice between counter and distribution controls whether Monitoring stores a simple event count or a statistical summary of numeric values.
| Metric type | What it stores | Choose when |
|---|---|---|
| Counter metrics | Number of log entries matching a filter in a period | Tracking how often a string or event appears, such as a specific error message or 404 Not Found errors |
| Distribution metrics | Count, mean, sum of squared deviations, and histogram bucket counts for extracted numbers | Measuring values such as request latencies and analyzing percentiles and outliers |
| Boolean metrics | Whether a log entry matches a filter | Using system-defined checks for presence or absence of a condition |
Counter metrics answer questions such as how many times user sally called the update operation by combining a filter with labels for username and operation. Distribution metrics accumulate each extracted numeric value into a distribution object at regular intervals, then write that object to Monitoring for percentile visualization. A common distribution use is extracting a latency field from each matching entry and using the default bucket layout or a custom layout to study tail latency.
Distribution objects in log-based metrics are time-series values that summarize many log entries in each interval. Each distribution object contains statistical fields that support mean, variance, and percentile analysis without storing every raw value.
As each log entry is received, the extracted numeric value is added to the current distribution. At regular intervals, the accumulated distribution is written to Cloud Monitoring, where it can be charted with heatmaps and percentile overlays. Because bucket boundaries shape percentile accuracy, teams tracking strict latency objectives often define custom buckets around their service level thresholds.
Labels in log-based metrics allow multiple time series to be collected for one metric. Values for the labels are extracted from fields in the matching log entries, and Logging records a separate time series for each combination of label values. Most log-based metrics provide a default log label that holds the LOG_ID portion of the logName field, such as cloudresourcemanager.googleapis.com%2Factivity. System log-based metrics have predefined labels that cannot be changed, while user-defined metrics can define custom labels such as username, operation, or originating folder or organization.
Resource type scoping controls how a log-based metric appears in Monitoring. A log-based metric should specify a single resource type in its filter, for example severity>="ERROR" combined with resource.type="gce_instance". When the filter names one resource type, Monitoring recognizes the data as belonging to Compute Engine VM instances and lists the metric when charting that resource. If no resource type is specified, Monitoring lists the metric as an option for multiple resource types even though entries were not counted from those resources, which creates ambiguity in charts and alerting policies.
Log-based metric naming in Cloud Monitoring (opens in a new tab) uses fixed patterns that distinguish system from user-defined data. System metrics use logging.googleapis.com/SYSTEM_METRIC_NAME, while user-defined metrics use logging.googleapis.com/user/USER_METRIC_NAME and always include the string user. Both system and user-defined log-based metrics can be used in Monitoring to create charts and alerting policies, where labels can filter or combine time series to show only selected services, operations, or logs.
Visibility of log-based metric data is determined by a Monitoring metrics scope. A metrics scope is a list of projects monitored by the hosting project, called the scoping project, and by default each project hosts a scope that includes only itself. A scoping project with a multi-project metrics scope can see metrics from all projects in that scope, and a single project can appear in multiple scopes so its metrics are visible to each scoping project. Log-based metrics cannot count entries by error group and cannot extract an error group ID into a label.
Exam tip: User-defined log-based metrics start counting only after creation and are named logging.googleapis.com/user/NAME, so they never include historical logs and are easy to distinguish from system metrics.
Direct application instrumentation uses the Cloud Monitoring API or a vendor-neutral framework such as OpenTelemetry to send custom metrics from code, agents, or Prometheus collectors. Log-based metrics derive numeric signals from logs already written to Cloud Logging, so they require no code deployment, agent change, or new collection pipeline. This makes log-based metrics attractive for counting application errors, auditing operations per user, or extracting latencies from existing structured logs. The collection overview (opens in a new tab) places log-based metrics alongside these direct paths as complementary ways to get data into Monitoring.
The trade-off is dependence on the logging pipeline and log volume. Log-based metrics only observe what is logged, add load to log ingestion and storage, and provide no data before the metric exists, while direct instrumentation can emit high-frequency gauges, fine-grained histograms, and business metrics even when no log is written. Choose log-based metrics when the signal already exists in logs and speed of setup matters, and choose direct instrumentation when precise control over sampling, cardinality, timeliness, or metric semantics is required.
/ with : and all other . and / characters with _, and must filter by monitored resource when a metric supports multiple resource types.global monitored resource risks time-series collisions; generic_task or generic_node with distinct labels are safer fallbacks for multi-source custom metrics.logging.googleapis.com/user/USER_METRIC_NAME.Use separate metric types when the activities are unrelated and need independent alerting, and use a shared metric with a low-cardinality label when comparison, grouping, or ratio math matters. Two separate types produce one time series each and allow simple alert conditions without label filters, while one type with a program identifier produces two time series and requires alert filters on the label value. Avoid unbounded label values such as user IDs, request IDs, or full URLs because high cardinality multiplies time-series count, increases cost and query load, and can hit quota limits.
You must use the mapped metric name because PromQL does not allow slashes and dots, replacing the first slash with a colon and replacing all other dots and slashes with underscores, so workload.googleapis.com/otlp.test.gauge becomes workload_googleapis_com:otlp_test_gauge. When the custom metric can be written against more than one monitored-resource type, you must also filter for the resource, for example by monitored resource gce_instance, and you may omit the resource filter only when the metric can be written against a single monitored-resource type.
Project-scoped metrics count only logs routed through the project where the metric is defined, while bucket-scoped metrics count all logs destined for that bucket regardless of where they originated. Both types count only log entries received after the metric is created and are never backfilled from older logs, and billing must be enabled on the project where the metric is defined.
No, customer support for Monitoring Query Language ended on July 22, 2025 and MQL is no longer available in the console for new charts, dashboards, and alerting policies. Existing MQL charts, dashboards, and alerting policies continue to work, and MQL objects can still be created through the Cloud Monitoring API.
Your DevOps team deploys a critical microservice on Compute Engine instances with the Google Cloud Ops Agent. The Ops Agent is configured to ingest OpenTelemetry Protocol (OTLP) metrics using the Cloud Monitoring API mode (metrics_mode: googlecloudmonitoring). The application emits two cumulative counter metrics:
workload.googleapis.com/custom/http_requests_totalworkload.googleapis.com/custom/http_errors_totalYou need to construct a PromQL-based alerting policy condition in Cloud Monitoring that triggers an incident whenever the 5-minute HTTP error rate exceeds 5% of total requests across the fleet.
Which PromQL query correctly references these metrics according to Cloud Monitoring syntax conventions?
Last updated