Prepare and test your skills

Prepare and test your skills

User-defined metrics in Cloud Monitoring (opens in a new tab) capture application-level telemetry that built-in metrics cannot measure, such as background routines spawned, widgets sold, or daily sales per store. An application instruments its code with a client library and sends measurements through the Cloud Monitoring API (opens in a new tab) to a backend project, where each measurement becomes part of a time series. Once stored, a user-defined metric works like a built-in metric and can be charted, read with the API, and used in alerting policies.
User-defined metrics fill gaps left by built-in system metrics for backend latency, disk usage, and CPU utilization. A team defines a custom measurement when it needs client-side or business logic, such as counting calls to auxiliary programs or tracking sales by store. Data flows from instrumented application code to the Cloud Monitoring API to stored time series in a Google Cloud project, and then to charts, dashboards, and alerting policies. Log-based metrics are the exception to this flow because that class of user-defined metric must be created from Cloud Logging, not from the Monitoring API.
A left-to-right process flow showing instrumented app code writing through the Monitoring API to stored time series in a GCP project, then to charts and alerts, with a separate exception path from log entries via Cloud Logging.
A metric descriptor (opens in a new tab) is the definition of a metric type that tells Cloud Monitoring how to organize and interpret incoming data. The metric descriptor for a custom metric specifies its type string, description, metric kind, value type, unit, and labels. A developer can explicitly create the metric descriptor first and then write metric data, or Cloud Monitoring can create the descriptor automatically from the first written data points. In either path, the developer must still decide how to organize the data so separate behaviors stay separable for alerting and analysis.
A custom metric name is a unique type string in the project that starts with a required prefix followed by a descriptive path. The allowed prefixes for user-defined metrics are:
custom.googleapis.com/ for metrics written directly through the APIworkload.googleapis.com/ for workload-specific metricsexternal.googleapis.com/user for user-defined external metricsexternal.googleapis.com/prometheus for Prometheus-format metricsThe full resource name of the descriptor adds project scope in the form projects/PROJECT_ID/metricDescriptors/METRIC_TYPE, such as projects/my-project-id/metricDescriptors/custom.googleapis.com/stores/daily_sales. In the descriptor, the type field holds the METRIC_TYPE string while the name field holds the full resource name. Value settings give precise categorization: metricKind describes whether values are instantaneous like GAUGE or changes over time like DELTA, valueType describes the data format like DOUBLE or 64-bit integer, and labels like store_id split one metric into one time series per unique label value.
A monitored resource identifies the source of measurements, while the metric type describes what is measured and the time series (opens in a new tab) binds one resource to timestamped values. The best practice is to write custom data against the same monitored-resource type where the application code runs, which improves performance, avoids out-of-order writes to the same time series, and groups custom data with other data from that resource. Choosing between generic and global resources controls collision risk and filtering power.
| Resource type | Best use | Labels and behavior |
|---|---|---|
generic_node | Node-like sources such as virtual machines | Has common labels for unique resource objects that support metric filters, aggregations, and reductions |
generic_task | Task-like sources such as applications | Has common labels for unique resource objects that support metric filters, aggregations, and reductions |
global | Fallback when no specific resource fits | Has only the project_id label, so many writers in one project can collide and overwrite the same time series |
The Cloud Monitoring API uses a MetricServiceClient in C#, Go, Java, Node.js, PHP, Python, and Ruby, plus the APIs Explorer, to manage descriptors and time series data. Read operations work for both built-in and user-defined metrics, while write and delete operations work only for user-defined metrics. A typical implementation follows this collection overview (opens in a new tab) order:
metricDescriptors.create with the project name and the descriptor containing type, GAUGE kind, DOUBLE value type, unit like {USD}, and labels like store_id.timeSeries.create to send timestamped points that carry both the metric labels and the monitored-resource labels to the project.timeSeries.list or metricDescriptors.list to retrieve data, then chart the time series, aggregate by labels, and alert on thresholds.User-defined metrics are chargeable and retained only for the standard retention period, so data that must be kept longer must be manually copied to another location such as Cloud Storage or BigQuery.
Exam tip: timeSeries.create, metricDescriptors.create, and metricDescriptors.delete apply to user-defined metrics only; built-in metrics can be listed and read but cannot be written or deleted.
Agent-based and open-source ingestion in Cloud Monitoring (opens in a new tab) extends monitoring beyond automatically collected Google Cloud system metrics. The Ops Agent runs on Compute Engine virtual machines (VMs) to collect host and application telemetry, while open-source standards like Prometheus and OpenTelemetry let applications expose custom metrics that flow from the workload to Cloud Monitoring for charting and alerting. Together they cover third-party software such as Apache, MySQL, Redis, Postgres, and MongoDB on Compute Engine, Google Kubernetes Engine (GKE), and Cloud Run.
By default, a Google Cloud project stores its own time-series data and its metrics scope limits charts and alerts to that stored data. Additional metrics collected by an agent or open-source framework are written into the project and then become available in Metrics Explorer, dashboards, and alerting policies. This makes the agent and the application endpoint the source, and Cloud Monitoring the destination for long-term storage and analysis.
The Ops Agent is deterministic telemetry software installed on Linux and Windows Compute Engine VMs. Compute Engine automatically collects only basic system metrics, so installing the Ops Agent adds deeper host data for troubleshooting, performance monitoring, and alerting. Data flows from the VM to Cloud Monitoring in the host project, where it can be charted alongside default Compute Engine metrics.
The Ops Agent automatically collects host metric data such as CPU, GPU, memory, disk, swap, and process metrics. The Ops Agent also automatically collects system log data such as syslog from Linux VMs and Windows Event Log from Windows VMs. This automatic collection fills gaps left by the 25-plus default Compute Engine metrics without requiring application changes.
Application observability with the Ops Agent is configured through receivers and pipelines:
The normal Ops Agent configuration links receivers, which select what to collect, and processors, which modify the data, together into pipelines within a service. Configuration for Prometheus metrics is simpler because it uses receivers and pipelines with no processors involved.
Third-party application integrations are prebuilt Ops Agent configurations for specific software on Compute Engine. An integration defines which metrics to collect, writes them as workload.googleapis.com metrics, and provides a dashboard for visualizing the metrics. Data flows from the third-party application to the Ops Agent receiver on the same VM, then to Cloud Monitoring.
Use an existing integration when one exists for the software being monitored. The metric set is curated and known in advance, which makes cost and dashboard planning easier. For example, the Apache Web Server integration collects five scalar metrics at the default 60-second sampling frequency, with each data point counted as 8 bytes for byte-based pricing for agent-collected metrics.
The Prometheus receiver is an Ops Agent receiver that scrapes Prometheus metrics on Compute Engine. The receiver gives Compute Engine users the same Prometheus capability that GKE users get through Google Cloud Managed Service for Prometheus. Metrics flow from the application Prometheus endpoint to the Ops Agent on the same VM, then to Cloud Monitoring as prometheus.googleapis.com metrics.
To use the Prometheus receiver (opens in a new tab), two prerequisites must be met. First, the Ops Agent version 2.25.0 or higher must be installed on the VM. Second, the application must expose an endpoint that emits Prometheus metrics, either directly or through a Prometheus client library or exporter such as Spring Metrics for Spring, statsd_exporter for StatsD, or graphite_exporter for Graphite.
Prometheus receiver metrics behave differently from integration metrics:
| Option | Metric namespace | Dashboard and curation | Pricing basis |
|---|---|---|---|
| Existing integration | workload.googleapis.com | Curated metric set plus prebuilt dashboard | Byte-based, 8 bytes per data point |
| Prometheus receiver | prometheus.googleapis.com | No curated set, user designs dashboards | Sample-based, based on samples ingested |
Prometheus receiver metrics are classified as custom metrics in Cloud Monitoring and are subject to custom metrics quotas and limits. Pricing depends on the number of scalar metrics where each value is one sample, the number of distribution metrics where each histogram counts as 2 plus the number of buckets, the sampling frequency, and the number of hosts. Because the application controls metric count and cardinality, scrape interval and label design directly affect ingestion volume and cost.
Exam tip: Choose workload.googleapis.com when using a prebuilt Ops Agent integration, and choose prometheus.googleapis.com when using the Ops Agent Prometheus receiver for custom Prometheus endpoints.
Google Cloud Managed Service for Prometheus (opens in a new tab) is the Google Cloud service that collects Prometheus and user-defined metrics for Kubernetes and serverless workloads. By default, GKE clusters send system logs and system metrics to Logging and Monitoring, while Managed Service for Prometheus handles third-party and user-defined metric data such as Postgres, MongoDB, and Redis. Cloud Run services that write Prometheus metrics use a Prometheus sidecar, and services that write OTLP metrics use an OpenTelemetry sidecar, to send data from the Cloud Run revision to Cloud Monitoring.
Managed Service for Prometheus supports four collection paths that differ by environment and operational control. Managed collection uses a Kubernetes operator to handle scrape configuration, scaling, and ingestion and is the recommended approach for GKE and other Kubernetes environments. Self-deployed collection uses a drop-in replacement for the upstream Prometheus binary, so existing Prometheus configurations can be reused with manual scaling and sharding. The OpenTelemetry Collector can scrape Prometheus exporters and send metrics, logs, and traces from any compute or Kubernetes environment, including stateless environments like Cloud Run. The Ops Agent can scrape Prometheus metrics on any Compute Engine instance, which simplifies VM discovery without running separate Prometheus servers.
For application instrumentation, the vendor-neutral observability framework (opens in a new tab) OpenTelemetry is recommended over vendor-specific APIs or client libraries. An instrumented application can emit logs, metrics, and traces to any backend, including Google Cloud, and OTLP provides a common protocol for moving that telemetry. Choose OpenTelemetry instrumentation when custom application insight is needed for performance understanding and failure troubleshooting, and choose Prometheus scraping when the application or exporter already exposes a Prometheus endpoint.
Log-based metrics turn text in Cloud Logging (opens in a new tab) into numeric time-series data in Cloud Monitoring (opens in a new tab). Data flows from log entries received by Cloud Logging through a filter defined in the metric to time-series data stored by Cloud Monitoring. Cloud Monitoring then makes that data available for charts, dashboards, and metric-based alerting policies. Log-based metrics are used to count recurring events over time, observe trends such as latency, and display extracted numbers visually.
Log-based metrics are useful when you need quantitative insight from logs:
Once the metric exists in Cloud Monitoring, charts and alerting policies treat it like other metrics and can filter or combine its time series by labels.
How log-based metrics collect data is governed by creation time and scope. Log-based metrics collect data only from log entries received after the metric is created. A user-defined log-based metric is not retroactively populated with log entries that are already stored in Cloud Logging. System-defined log-based metrics are calculated from included logs only, while user-defined log-based metrics are calculated from both included and excluded logs.
User-defined log-based metrics can apply to a Google Cloud project or to a specific log bucket in a project. Project-scoped metrics count a log entry when billing is enabled on the defining project and the entry passes through that project’s log sinks. Bucket-scoped metrics count a log entry when billing is enabled and the entry is stored in the bucket where the metric is defined.
Exam tip: A user-defined log-based metric never backfills history; only log entries received after creation create new points in Cloud Monitoring.
Counter metrics count the number of log entries that match a specified filter within each time interval. A counter metric is the right choice when the question is how often something happened, such as how many errors occurred or how many times a string appeared in logs. Each matching entry increases the count, and Cloud Monitoring stores the per-interval totals as a time series that can be charted and alerted on.
Counter metrics can use labels to split one metric into many time series. For example, a filter can match log messages in the form ... user USERNAME called OPERATION ... and extract USERNAME and OPERATION as label values. Logging then records a separate time series for each combination of label values, so you can later ask how many times sally called update, how many people called read, or how many operations george called.
Distribution metrics accumulate numeric values extracted from matching log entries and group them into histogram buckets. A distribution metric is the right choice when the question is how values are spread out, such as request latency, response size, or other measurements recorded in logs. As each log entry arrives, the extracted value is added to the current distribution, and at regular intervals the accumulated distribution is written to Cloud Monitoring.
Each distribution point written to Cloud Monitoring contains related statistics for that interval:
Because distribution metrics preserve count, mean, variation, and bucket counts, they support analysis of averages, spread, and percentiles in charts. A common pattern is to extract a latency field from each request log and track how latency shifts over time.
Choosing the metric type depends on whether logs provide events to count or numbers to analyze. Counter metrics track event frequency from matching entries, while distribution metrics analyze the range and shape of extracted numeric values. Both user-defined types rely on a log filter, but only distribution metrics also require a field that contains the numeric value to extract.
| Metric type | What it records | Choose when |
|---|---|---|
| Counter metric | Number of log entries matching a filter in each interval | You need frequency of errors, warnings, or operations by label values |
| Distribution metric | Count, mean, sum of squared deviations, and histogram bucket counts for extracted numbers | You need spread, average, or trends for latencies or other value ranges |
Use a counter metric when counting is enough, even if labels divide the counts by user, operation, or log name. Use a distribution metric when a single count would hide important changes, such as when most requests are fast but a growing share is slow.
Labels and resource types in log-based metrics control how time series appear in Cloud Monitoring. Labels allow multiple time series to be collected for one metric, with values extracted from fields in each matching log entry. Most log-based metrics provide a default log label that holds the LOG_ID portion of the entry’s logName field, while user-defined metrics can add custom labels such as user name, operation, folder, or organization.
Resource types connect a log-based metric to the data model expected by Cloud Monitoring. A log-based metric should specify a single resource type, such as gce_instance with a filter like severity>="ERROR" and resource.type="gce_instance". When the resource type is specified, Cloud Monitoring lists the metric as an option for charts and alerting policies for that resource, while a metric without a resource type may appear under multiple resources even though it does not count entries from them.
In Cloud Monitoring, log-based metrics use names that begin with logging.googleapis.com/. System metrics use the pattern logging.googleapis.com/SYSTEM_METRIC_NAME, and user-defined metrics use the pattern logging.googleapis.com/user/USER_METRIC_NAME, where the string user marks the metric as user-defined. You can chart either type and create alerting policies that filter or combine time series by labels.
custom.googleapis.com/ and resource names in the form projects/PROJECT_ID/metricDescriptors/METRIC_TYPE.timeSeries.create, metricDescriptors.create, and metricDescriptors.delete apply to user-defined metrics only; built-in metrics can be listed and read but cannot be written or deleted.workload.googleapis.com metrics with curated dashboards and byte-based pricing, while the Ops Agent Prometheus receiver writes prometheus.googleapis.com metrics with sample-based pricing.Counter log-based metrics record the number of log entries matching a filter in each interval, while distribution log-based metrics record count, mean, sum of squared deviations, and histogram bucket counts for extracted numeric values. Use a counter metric when you need frequency of errors, warnings, or operations by label values, and use a distribution metric when you need spread, average, or trends for latencies or other value ranges.
Use workload.googleapis.com metrics when using a prebuilt Ops Agent third-party integration, and use prometheus.googleapis.com metrics when using the Ops Agent Prometheus receiver for a custom Prometheus endpoint. Existing integrations write workload.googleapis.com metrics with a curated metric set plus prebuilt dashboard and byte-based pricing, while the Prometheus receiver writes prometheus.googleapis.com metrics with no curated set and sample-based pricing.
The methods timeSeries.create, metricDescriptors.create, and metricDescriptors.delete apply to user-defined metrics only. Built-in metrics can be listed and read with timeSeries.list or metricDescriptors.list but cannot be written or deleted, and user-defined metrics are chargeable and retained only for the standard retention period.
Your team is developing an application running on Google Cloud and needs to record custom application-level telemetry directly into Cloud Monitoring using the Google Cloud client library.
You must define the schema for a new metric named custom.googleapis.com/retail/store_sales that tracks monetary sale amounts as 64-bit floating-point numbers and allows filtering and grouping by a store identifier.
Which configuration should you define on the MetricDescriptor object before creating it via the MetricServiceClient?
Last updated