Professional Cloud DevOps Engineer

Instrument Applications for Custom Metric Collection

Custom Telemetry Sources

Developers use user-defined metrics, often called custom metrics, to capture specialized application data that Google Cloud's built-in system metrics miss. You can collect these metrics by writing directly to the Cloud Monitoring API or by implementing the vendor-neutral OpenTelemetry standard. Standardizing on OpenTelemetry is highly recommended because it builds an adaptable observability pipeline and avoids vendor lock-in. These custom metrics are ingested into Google Cloud Monitoring under different modes, which ultimately determine how they are billed and which resource domains they occupy.

Metric Descriptors and Schemas

Every custom metric requires a metric descriptor that defines its schema, identity, and behavior. You must specify a metricKind, such as GAUGE for instant values, CUMULATIVE for values that continually grow, or DELTA for changes over a specific window of time. The descriptor also requires a valueType to establish data formats like INT64, DOUBLE, BOOL, STRING, or DISTRIBUTION. Selecting the correct matching types is critical because incorrect pairings will cause failures in downstream queries, dashboards, and aggregations.

Collector Ingestion Modes

When running an OpenTelemetry Collector as a sidecar, your chose ingestion configuration reshapes how metric paths and resources are labeled. Choosing Prometheus API ingestion mode automatically transforms metric names by adding a prometheus.googleapis.com prefix and links them to a generic prometheus_target resource. Alternatively, choosing the Monitoring API mode preserves the original OpenTelemetry metric name under the workload.googleapis.com namespace and maps the data to a gce_instance resource. This structural choice directly alters how you write queries, target alerting rules, and manage your monthly bill.

Managing Ingestion and Costs

For workloads where you cannot modify the application source code, you can build log-based metrics inside Cloud Logging to extract time-series data from log text. To prevent high costs and performance bottlenecks from these and other custom metrics, administrators use the Metrics Management dashboard. This page helps teams inspect ingestion rates, detect metric-write errors, identify unused metrics, and discard high-cardinality labels before they drive up costs.

Query, Alert, and Troubleshoot Custom Metric Pipelines

Querying Telemetry Data

After metric data is successfully ingested, you must query it to analyze trends, set up alerts, and resolve system issues. Cloud Monitoring supports two key languages: Monitoring Query Language (MQL) and Prometheus Query Language (PromQL). When writing PromQL queries, you must translate standard Cloud Monitoring metric names into Prometheus conventions by replacing the first slash with a colon and swapping other special characters for underscores. To make your queries run efficiently and return accurate data, you must explicitly declare the target resource using the monitored_resource label.

Incident Alerts and Thresholds

You can verify query results inside Metrics Explorer and then use those same queries to build alerting policies. A PromQL-based alerting policy continuously checks your system conditions against custom thresholds, generating an incident and sending notifications when rules are violated. These alerting policies have strict limitations, such as a maximum combined retest and alignment window of 25 hours. For simpler monitoring setups, you can also apply pre-built recommended alerts provided directly by integrated Google Cloud services.

Identifying Pipeline Failures

When telemetry data fails to appear in your dashboards, you must systemically isolate where the ingestion pipeline is broken. Common pipeline issues include configuration errors in the Ops Agent or OpenTelemetry Collector, or IAM authorization failures where a service account lacks the Monitoring Metric Writer role. If you observe latency anomalies where metrics are delayed, the issue is typically caused by long agent collection intervals or replication delays in the global Monarch database. Using the Metrics Management tool allows you to inspect write-error rates and identify whether cardinality limits are being exceeded.

Scalability and Multi-Project Scopes

Ensuring your observability pipeline scales with your infrastructure involves continuous optimization and validation. You can run a quick up query in PromQL to check if your Prometheus exporters are being successfully scraped. To monitor complex, multi-project environments, you should configure multi-project metrics scopes and resource groups to aggregate and query performance data from multiple Google Cloud projects in a single interface. This strategy allows your operations team to maintain a unified pane of glass without logging into separate project accounts.

Design and Configure Log-Based Metrics

Extracting Metrics from Logs

Log-based metrics allow you to generate custom metrics by evaluating log entries as they are processed by Cloud Logging. These metrics run on incoming log data to record occurrences of events or track numeric fields without requiring developers to change application code. Once configured, these metrics write directly to Cloud Monitoring and can be used to populate dashboards and trigger alerts.

Metric Types and Configurations

You can create three distinct types of log-based metrics depending on what telemetry you need to capture:

• Counter metrics track how many times a log entry matches a specific filter, which is ideal for measuring event frequency like error spikes.
• Distribution metrics gather numerical values from matching logs into histogram buckets, enabling you to calculate the mean, sum of squared deviations, and percentile ranges for latency analysis.
• Boolean metrics record a simple true or false value to indicate the presence or absence of a specific condition within the log stream.

User-Defined Labels

To segment your log-based metrics, you can define labels that organize your data into distinct time series. While default labels provide basic resource and log identifiers, you can create up to 10 user-defined labels using extraction expressions. You can extract an entire field directly or use a regular expression to pull out specific patterns, like extracting a user ID from a structured JSON log. Every unique combination of label values creates a new time series, so you should avoid using high-cardinality fields like timestamps or unique transaction IDs.

Design Trade-offs

When planning your telemetry strategy, you must evaluate the trade-offs between log-based metrics and direct application instrumentation. Log-based metrics are faster to deploy and require zero code modifications, but they are not retroactive and only start gathering data after they are created. Additionally, log-based metrics are calculated using all logs that reach the router, regardless of whether those logs are excluded from storage. Because Cloud Monitoring limits each log-based metric to 30,000 active time series, choosing to instrument the application directly is often the better design decision for high-cardinality tracking.