Instrumenting Telemetry with Ops Agent and OpenTelemetry

Collecting and importing logs is the end-to-end pipeline that moves host, application, audit, and network telemetry from where it is generated into Cloud Logging (opens in a new tab) for storage and analysis. Host data flows automatically from virtual machines (VMs), application data flows from instrumented code through receivers and collectors, and platform data flows from Google Cloud control planes and networks through audit and flow logs. Aggregated sinks then centralize those entries for security, troubleshooting, and compliance.

1.1Ops Agent Host Collection and Custom Log Pipelines

The Ops Agent is the primary single-process collector for Compute Engine VMs that gathers host metrics and system logs without extra configuration. The Ops Agent uses Fluent Bit for high-throughput multi-core log collection and the OpenTelemetry Collector for metrics and traces, and it installs with one download and one YAML configuration with proxy support. By default the Ops Agent collects host metrics for CPU, GPU, memory, and processes plus system logs such as Linux /var/log/syslog, /var/log/messages, and Windows Event Log, while VMs without the agent report only basic system signals.

Custom log collection in the Ops Agent is defined in config.yaml with logging receivers, processors, and pipelines that connect sources to Cloud Logging. A receiver defines where data comes from, a processor defines how entries are transformed or filtered, and a pipeline binds receivers and processors to an export destination in the VM project. Telemetry flows from the log source to the receiver, then through each processor in order, then from the pipeline to Cloud Logging.

Receivers cover sources beyond the defaults:

• File-based application logs with customizable paths and refresh intervals
• Journald and systemd logs from the Linux init system
• TCP network sources and Forward protocol sources used by Fluent Bit and Fluentd
• Curated third-party integrations for Postgres, MongoDB, and Java Virtual Machine with pre-configured dashboards and alert policies

Processors normalize entries before export by parsing text into structured logs with JSON or regular-expression parsing, modifying entries by removing or renaming fields, excluding unwanted logs by labels and expressions, and concatenating multiline Java, Python, and Golang exceptions. Applications should therefore write JSON-structured logs to a file first, then let the receiver tail that file while processors normalize severity and fields.

1.2OpenTelemetry Standardization and Compute Engine Export Pattern

OpenTelemetry (opens in a new tab) is the recommended vendor-neutral framework for generating metrics and traces and for structuring logs for Cloud Logging. For logs, applications use JSON-capable frameworks such as slog for Go, logging for Python, Pino for JavaScript, and SLF4J with Log4j2 for Java. For metrics, applications use OpenTelemetry or Prometheus client libraries that expose an HTTP endpoint for scraping, and for traces they use OpenTelemetry to create distributed traces where each trace is one end-to-end operation and each span is one function or operation.

Collection architecture determines whether code stays vendor-neutral and where transformation happens. In collector-based export, the application uses the OpenTelemetry software development kit (SDK) with an in-process OpenTelemetry Protocol (OTLP) exporter to send data to a collector, and the collector forwards it to Google Cloud. In direct export, the application uses Google-specific exporters to send data directly to the Telemetry API with no collector in between.

The Compute Engine pattern combines both paths: application logs flow from the application file to the Ops Agent logging receiver to Cloud Logging, while metrics and traces flow from the OpenTelemetry SDK to the Ops Agent OTLP receiver to Cloud Monitoring and Cloud Trace. The Ops Agent depends on the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs being enabled and on the VM service account having write permission to them. An alternative for Prometheus-format metrics uses the Ops Agent Prometheus receiver to scrape instrumented endpoints, and zero-code instrumentation can attach a runtime agent with environment variables instead of code changes.

1.3Cloud Audit Log Types, Access, and Querying

Cloud Audit Logs (opens in a new tab) is the platform audit system that writes immutable entries to Cloud Logging for Google Cloud API activity. Entries are organized into four types that differ by trigger and default enablement.

Access is split by role: applications need roles/logging.logWriter on the node service account to send entries, humans need roles/logging.viewer to view logs and logging.privateLogViewer for Data Access logs, and administrators need roles/logging.configWriter to create centralizing sinks. Collection depends on forwarders and buffers, such as an audit-proxy DaemonSet in Google Distributed Cloud that buffers Kubernetes API Admin Activity entries, with fallback to local disk and a 10 GB buffer that drops oldest entries first during network outages.

Troubleshooting flows from scope to log type to query in Logs Explorer:

1. Select scope: Open Logs Explorer and select the organization or project and resource type such as k8s_cluster.
2. Select log name: Choose activity, data_access, system_event, or policy.
3. Refine query: Filter on fields such as logName and protoPayload.serviceName.
4. Run and inspect: Run the query and inspect protoPayload for method, identity, and resource details.

Monitoring flows from matching entries to Cloud Monitoring through log-based alerting policies that separate immediate-investigation events such as unauthorized VM creation from low-priority events.

1.4Network Telemetry, Aggregated Sinks, and Log Destinations

Network telemetry extends audit history with evidence of packet movement and policy decisions. VPC Flow Logs record connection details for subnets, firewall logs record compute.googleapis.com/firewall allow and block decisions, NAT flow logs record gateway translations, and DNS query logs record resolver activity. Traffic flows from subnets, firewall rules, NAT gateways, and resolvers to Cloud Logging, where sampling rate and metadata annotations control how much detail is kept for latency diagnosis and boundary monitoring.

An aggregated sink is a Cloud Logging sink created at the folder or organization level that routes matching logs from all child projects to one destination. The sink is created with gcloud logging sinks create plus --include-children, and routing fails until the sink writerIdentity service account is granted write access on the destination. Routing copies entries without moving them, so Cloud Logging stores logs twice unless operators add an exclusion filter to child _Default sinks or disable _Default in organization defaults.

Destination choice determines which tools can query the data and how long it is kept:

• Logging bucket for impromptu investigation with Observability Analytics
• BigQuery dataset with daily partitioned tables for SQL analysis and per-table expiration
• Cloud Storage bucket with retention locks for low-cost long-term compliance retention
• Pub/Sub topic for real-time streaming to Elastic, Splunk, and Datadog

Reimporting returns exported objects from Cloud Storage to Cloud Logging through a Cloud Run job that reads objects, selects a log ID and time range, converts objects to LogEntry batches to save quota, and writes them back under normal routing rules. Imported entries are subject to quotas and 30-day retention, so entries older than 30 days at import time are not stored, and imported logs are excluded from log-based metrics unless a label brings them into metric data.

Optimizing logs is the practice of controlling ingestion volume, retention time, and export destinations so Cloud Logging preserves troubleshooting and compliance signals without overspending. Log data flows from the originating project, folder, or organization into Log Buckets and then optionally onward to analysis or archival destinations. Each stage of ingestion into Cloud Logging, retention in a bucket, and export to another service carries its own cost driver.

2.1Log Lifecycle, Buckets, Retention, and Tiered Exports

Default storage in Cloud Logging requires no configuration because each resource automatically stores the logs that originate in it, such as a Compute Engine instance storing logs in its own project. Every stored entry follows a retention lifecycle where it is kept for a retention period and then deleted, so excluding low-value entries and shortening retention are the two primary cost controls. Exclusion stops cost before ingestion and storage, while shorter retention stops ongoing accumulation of low-value history.

A Log Bucket is the Cloud Logging storage container that holds routed entries and enforces retention, region, and Identity and Access Management (IAM) access control. Every project includes _Required and _Default sinks that route entries to buckets, and sinks can be inherited from parent folders or organizations that may forward logs to unwanted destinations. A bucket can be upgraded to use Observability Analytics, formerly Log Analytics, for direct SQL querying at no extra upgrade cost, but an upgraded bucket cannot be downgraded.

Tiered exports separate hot operational storage in Log Buckets from colder or specialized storage elsewhere, which lets teams keep brief retention in Cloud Logging while preserving compliance history in cheaper destinations. The Log Router can route entries that pass filters to a destination in the same or a different project, and inherited sinks should be reviewed before adding exports to avoid duplicate charges.

Supported routing destinations include:

• Google Cloud project for forwarding to another project pipeline
• Log Bucket in the same or a different project for centralized retention
• BigQuery dataset for long-term SQL analysis where cost includes storage plus query computation
• Cloud Storage bucket for low-cost archival where Object Lifecycle Management can downgrade to Nearline, Coldline, or Archive
• Pub/Sub topic for streaming to Splunk or Datadog, often via Dataflow batching and compression

2.2Log Router Filtering, Exclusions, Sampling, and Transforms

The Log Router is the component that evaluates every incoming entry against sinks and forwards matching entries to destinations. A sink contains an inclusion filter that selects which entries to route and one or more exclusion filters that discard matching entries before storage or forwarding. When an entry matches an exclusion filter, the Log Router drops it so it consumes no bucket storage and incurs no downstream cost.

Source category shapes filtering decisions because different sources have different value and retention needs. Platform logs (opens in a new tab) are written by Google Cloud services for debugging, component logs come from Google-provided software on customer systems, security logs answer who did what through Cloud Audit Logs (opens in a new tab) and Access Transparency, user-written logs come from custom applications via the Ops Agent or API, and multicloud and hybrid logs come from other clouds or on premises. High-volume debug logs, successful health checks, and repetitive request logs are first candidates for exclusion, while audit and security logs often need longer retention despite cost.

Volume can also be reduced without dropping entire streams. Request logging can be sampled so only a percentage or only errors are recorded, transform logic can strip verbose fields or fully qualified domain names, and redact logic can mask sensitive fields before storage. The order of operations matters for savings: filter upstream at the source before data leaves Cloud Logging, drop at the router with exclusion filters, reduce within retained events with transform or redact, then verify with destination counts and ingestion metrics that required signals remain.

2.3Cost Control by Source, Exemptions, and Batching

Cost control by log source starts with ingestion metrics (opens in a new tab) that show volume by log type, resource, and project, which reveals the streams driving spend. High-volume low-value user-written and platform debug logs are tuned first with exclusion filters, sampling, and shorter retention, while audit logs are tuned selectively to preserve compliance. For exported data, teams apply time-to-live lifecycle rules, consistent tags for owner and retention, and Organization Policy constraints so FinOps reports can track spend by dataset.

Exemptions and exclusions control Data Access audit volume at different points, which matters most for services like Bigtable that can generate extreme DATA_READ and DATA_WRITE volume from application service accounts. Exempting a principal in the IAM audit policy prevents entries from being generated and incurs neither ingestion nor storage charges, while an exclusion filter discards already-generated entries from bucket ingestion after generation occurred. Operators first identify noisy principals through IAM policies, then exempt service accounts never needed for investigation and use exclusion filters for finer criteria such as log name or connection attributes.

Batch aggregation lowers quota and egress cost without changing content. The Cloud Storage to Cloud Logging reimport path uses a Cloud Run job that reads exported objects, selects a log ID and time range, converts objects to LogEntry structures and batches them to reduce API quota use with retries, then writes them where routing rules apply. The same tradeoff applies when forwarding through Pub/Sub and Dataflow to partners, where setting batchCount to its maximum of 1,000 improves egress efficiency. Imported entries with timestamps older than 30 days are not stored, and imported logs are excluded from log-based metrics unless a label records import time.

Collecting metrics is the practice of forwarding numeric time-series data from Google Cloud services, VMs, Kubernetes, on-premises infrastructure, and other clouds into one queryable store. Distributed collectors scrape or receive metrics near each workload, then data flows from each collector to Monarch, the globally scalable datastore behind Cloud Monitoring and Google Cloud Managed Service for Prometheus. A single Grafana instance, Metrics Explorer chart, or alerting policy can then query that global store with PromQL instead of connecting to individual servers.

3.1Monarch Storage, Metrics Scopes, and Hybrid Aggregation

Monarch (opens in a new tab) is the central storage and query layer that makes unified monitoring possible across regions, projects, and clouds. Data storage in Monarch retains Prometheus data for 24 months, while query evaluation in Monarch executes PromQL and unions results across all Google Cloud regions and across up to 3,500 Google Cloud projects. The metrics scope of a Google Cloud project controls which stored time series that project can chart or monitor, because by default the scope limits the project to series stored by that project alone. An operator expands visibility by adding other projects to the scope, so alerts, dashboards, and service-level objectives operate over the consolidated set.

Hybrid aggregation extends that pipeline beyond Google Cloud by running collectors near each workload and forwarding to Monarch for long-term storage and global querying. Collectors outside Google Cloud need only local network access to their targets, so workloads do not have to be exposed to the global query layer. Grafana then connects to Monarch rather than to each Prometheus server, so one Grafana instance provides a unified view when collectors are deployed in every environment.

3.2Ops Agent, BindPlane, and Managed Service for Prometheus Collectors

The Ops Agent is the collection pipeline for Linux and Windows VMs on Compute Engine, including third-party applications that basic system metrics do not cover. The agent collects host metrics for CPU, disk, memory, swap, GPU, and processes plus system logs, and it adds third-party integrations for Apache, Nginx, MySQL, Redis, MongoDB, PostgreSQL, and Java Virtual Machine. The same agent can scrape Prometheus metrics through its Prometheus receiver and receive OTLP metrics and traces through its OTLP receiver, so data flows from local endpoints to the agent to Cloud Monitoring without a separate Prometheus server.

BindPlane is the documented path for collecting on-premises and hybrid-cloud metrics into Cloud Monitoring alongside native metrics. BindPlane maps external infrastructure metrics into time series stored by a Google Cloud project, where they become subject to the same metrics scope, PromQL querying, dashboards, and alerting as Google Cloud service metrics. Operators choose BindPlane when on-premises sources must be analyzed together with cloud workloads for health, saturation, and availability.

Google Cloud Managed Service for Prometheus (opens in a new tab) is the fully managed multi-cloud solution that separates collection, query evaluation, rule evaluation, and storage. Collectors scrape local exporters and forward samples from any cloud or site to Monarch, Monarch evaluates queries and stores data, and rules are evaluated either by PromQL alerts in Cloud Monitoring or by a standalone evaluator that forwards fired alerts to Prometheus Alertmanager. Managed collection and the Ops Agent receive full Google Cloud support, while self-deployed collection and the OpenTelemetry Collector receive limited assistance.

3.3Service Mesh Observability with Envoy Telemetry

Network and Service Mesh observability via Cloud Service Mesh is the collection of telemetry from Envoy proxies in the data plane to understand transit latency and inter-service communication. The data plane runs Envoy proxies as sidecars beside each workload or as proxyless gRPC clients, while the managed control plane distributes routing and telemetry configuration. Telemetry flows from each proxy to Google Cloud Observability, where Cloud Monitoring, Cloud Logging, and Cloud Trace store it for dashboards and topology views.

Envoy proxies batch and export each signal on its own schedule, so logs arrive fastest while metrics and topology arrive about every minute and traces depend on sampling. Full telemetry can be delayed 1-2 minutes during startup, which is normal during rollout or restart. The reporting path depends on the proxy having the Google Cloud Observability filter configured and the Cloud Monitoring API enabled.

Envoy proxies report four signals:

• Logging exported about every 10 seconds for proxy and traffic events
• Metrics exported about every 1 minute for request and connection data
• Topology edges reported incrementally about every 1 minute and fully about every 10 minutes
• Traces exported by sampling, typically 1 out of every 100 requests

Automatic dashboards show HTTP service metrics and logs without custom instrumentation and let operators filter by source, destination, response code, and version. Because traffic flows from the client-side Envoy to the server-side Envoy to the destination endpoint, both sides can emit telemetry for the same request, which lets operators compare client-observed versus server-observed latency to separate transit faults from service faults. Dashboards display metrics only for HTTP(S) and gRPC services whose Kubernetes Service port is named http or starts with http-; TCP byte streams carry traffic but do not produce mesh metrics.

Troubleshooting missing metrics follows an ordered dependency chain:

1. Verify sidecar injection: Confirm pods include istio-proxy with kubectl get pods; without injection no proxy emits telemetry.
2. Verify telemetry filter: Dump proxy config and look for the Google Cloud Observability filter with stackdriver_inbound.
3. Verify HTTP identification: Confirm the Service port name is http or http- prefixed.
4. Verify API delivery: Confirm the Cloud Monitoring API is enabled and check for 429 quota errors.
5. Verify proxy logs: Review istio-proxy logs for reporting errors while ignoring normal mTLS PERMISSIVE warnings.

3.4Platform Scraping with PodMonitoring and Duplication Control

PodMonitoring and ClusterPodMonitoring are the managed-collection resources that define which Pods to scrape and how to filter their metrics on Google Kubernetes Engine (GKE) (opens in a new tab). PodMonitoring scrapes only Pods in the namespace where it is deployed, so the same manifest must be deployed per namespace for multi-namespace coverage. ClusterPodMonitoring scrapes matching Pods in any namespace from one cluster-scoped object, while ClusterNodeMonitoring scrapes node components such as kubelet and cAdvisor.

Filtering uses Prometheus relabeling inside the custom resource to keep only wanted series, which controls cardinality, cost, and duplication. A typical manifest selects an exporter by labels and scrapes a named port on an interval, such as scraping the NVIDIA DCGM exporter every 30 seconds for GPU utilization and health. The collector depends on the custom resource for discovery, the Pod depends on exposing a reachable port and path, and Monarch depends on the collector for delivery before series become globally queryable.

Targeted collection avoids ingesting a full built-in package when only one signal is needed, but keeping a metric already covered by an active package creates duplicate series. Built-in GKE packages already ingest curated sets for system components, control plane, kube-state, cAdvisor, kubelet, and DCGM. Operators therefore identify the metric and endpoint, list active packages with gcloud container clusters describe, check package coverage, verify the endpoint exposes the series, then apply a filtered PodMonitoring or ClusterPodMonitoring object and verify ingestion with a PromQL query in Metrics Explorer.

A Cloud Monitoring synthetic monitor is a proactive probe that simulates user or client behavior against application endpoints and workflows. The synthetic monitor runs as code on a schedule from global locations and validates availability, functionality, and performance before real users are affected. The same scheduled check can therefore detect regional failure, broken navigation, and degraded API contracts across distributed endpoints.

4.1Synthetic Monitor Types for Endpoints and User Journeys

Synthetic monitors (opens in a new tab) divide work between narrow deterministic checks and broad journey checks. A generic HTTP validation sends a real HTTP request to one URL or API and asserts on status and content, which fits pinging a website or validating an API contract. A broken-link checker starts at an origin page and validates many linked pages with browser rendering, which fits marketing sites, documentation hubs, or portals where one broken link blocks the next step.

A generic HTTP validation is a Mocha test that depends on Node.js dependencies declared in package.json and loaded with require, such as node-fetch for requests, chai for assertions, and clients for secrets or data. At runtime traffic flows from the Mocha it block to the target URL through fetch, then to an assertion such as expect(externalRes.ok).to.be.true, so a non-successful status fails the monitor. A broken-link checker is a browser-based monitor implemented with the Cloud Functions Framework and Synthetics SDK by calling runBrokenLinksHandler(options) inside functions.http, so scheduler requests flow to the handler, then to the origin page, then to each discovered link with Puppeteer rendering.

4.2Probe Options, Assertions, and Execution Constraints

A synthetic probe defines where to send traffic and how broadly to crawl through its options object. The options object defines the starting point, scope, selection logic, timing, and failure evidence for the crawl, including origin_uri for the start page, link_limit for how many links to test, query_selector_all and get_attributes for which elements supply links, link_order for FIRST_N or RANDOM sampling, link_timeout_millis per link, max_retries per failed link, wait_for_selector for dynamic pages, per_link_options for URL-specific expectations, total_synthetic_timeout_millis for the whole run, and screenshot_options for failure capture.

Assertions turn responses into pass or fail verdicts. The generic Mocha template asserts availability with expect(externalRes.ok).to.be.true, while the broken-link checker asserts per-link expectations through expected_status_code, which can be a specific code such as 304 or a class such as STATUS_CLASS_3XX or STATUS_CLASS_4XX. Latency assertions use two timeout boundaries that work together: link_timeout_millis, for example 30000 milliseconds, bounds each link probe, while total_synthetic_timeout_millis, for example 60000 milliseconds, bounds the entire run from origin load through the last link check. Traffic flows from the origin page to each extracted link in link_order, and each link is evaluated against its own expected status and timeout.

Execution constraints reflect that a Puppeteer browser needs more resources than a single fetch call. Probes that use Puppeteer need at least 1GiB of allocated memory because default memory, CPU, and security settings may be insufficient. Retries with max_retries separate transient slowness from persistent breakage but increase run time and must fit inside the total timeout. Per-link overrides handle nuance that not every non-200 response is a failure, so per_link_options maps a specific URL to its own expected status or timeout while strict success remains enforced elsewhere.

4.3Alerting, Result Analysis, and Cost Optimization

An alerting policy converts failed probes and degraded performance into notifications and follow-up work. The synthetic monitor depends on the alerting policy: the probe produces pass, fail, and latency signals, and the policy decides when those signals warrant human attention through thresholds, events, or anomaly conditions. Troubleshooting then uses collected logs and metrics to diagnose the issue behind a failed assertion, which minimizes downtime and connects the failed probe back to backend behavior.

Interpreting results means reading latency, success, and error signals together in Cloud Monitoring rather than reacting to one failed probe. Latency distributions show how response times spread across regions and time, success rates show the proportion of probes completing workflows, and error patterns show whether failures cluster by endpoint, step, geography, or time. Cloud Monitoring supports trend analysis to separate gradual degradation from sudden breakage, experiment analysis to judge changes, alerts for sustained shifts, and retrospective analysis to validate whether fixes held.

Correlation locates root cause by comparing synthetic failures against backend metrics for CPU, memory, queues, database latency, and downstream dependencies in the same window. When synthetic latency rises together with backend saturation, the cause likely sits inside the application or its resources, and when probes fail while backend health looks normal, the cause likely sits in routing, load balancing, authentication, or an external dependency. Load testing strengthens this diagnosis by showing capacity breakpoints that match probe failures under normal traffic.

Optimizing configuration balances detection sensitivity against quota, storage, and load on the application and third-party APIs. More frequent probes from more locations detect outages faster but consume monitoring quota, generate telemetry, and add load, while reduced frequency lowers cost but risks missing short or regional outages. Active Assist provides rightsizing and capacity recommendations that turn synthetic history into a tuning loop: keep sensitive frequent probing where latency and success protect revenue, and relax probing where stability is proven.

Custom metric pipelines in Cloud Monitoring (opens in a new tab) move data from writers to readers through an ordered chain. Data flows from an instrumented application, the Ops Agent, Cloud Logging for log-based metrics, or Google Cloud Managed Service for Prometheus into time series stored against a monitored resource. A query in Metrics Explorer, a custom dashboard, or an alerting policy then reads those time series, and the policy sends notifications when its condition is met.

5.1Instrumentation Paths and Monitored Resources

Instrumenting an application for custom collection means adding code to measure behavior that built-in metrics cannot capture, such as background routines spawned or internal function calls. User-defined metrics (opens in a new tab) cover any metric not defined by Google Cloud, including third-party metrics, and once stored they work like built-in metrics for charting, alerting, grouping, and filtering. The recommended path is vendor-neutral OpenTelemetry through an OTLP pipeline, while direct Cloud Monitoring API writes remain supported.

Common collection paths group by environment:

• OTLP receiver in the Ops Agent for OpenTelemetry applications on Compute Engine VMs
• Google Cloud Managed Service for Prometheus for Prometheus metrics on Google Kubernetes Engine and Kubernetes
• Prometheus receiver in the Ops Agent for Prometheus metrics on Compute Engine
• Cloud Monitoring API timeSeries.create calls that write points directly from code
• OpenTelemetry Collector as a sidecar for custom metrics on Cloud Run

Data flows from the instrumented application to the collector or agent, then to the Cloud Monitoring backend where time series are stored per project. Every custom point must attach to a monitored resource that identifies where code runs, and writing against the same resource that produced related system metrics gives better performance and avoids out-of-order writes. The global resource has only a project_id label, so many writers sharing one global object can collide and overwrite each other, while generic_task for task-like workloads and generic_node for node-like workloads provide multiple labels for unique objects.

5.2Descriptors, Kinds, Cardinality, and Querying

A MetricDescriptor is the Cloud Monitoring structure that defines metric type name, description, unit, labels, metric kind, and value type. Cloud Monitoring can create the descriptor implicitly from first written points, or owners can create it explicitly with metricDescriptors.create for tighter control. The type name must be unique among user-defined metrics in the project and use a user-defined prefix such as custom.googleapis.com/stores/daily_sales, with supported prefixes custom.googleapis.com/, workload.googleapis.com/, external.googleapis.com/user, and external.googleapis.com/prometheus.

Metric kind defines the relationship between adjacent measurements, and value type defines the data type of each measurement.

Value type must be INT64, DOUBLE, BOOL, STRING, or DISTRIBUTION, with DOUBLE common for sales in {USD} and DISTRIBUTION for latency distributions that support percentile analysis. A time series is one unique combination of monitored-resource values plus metric-label values, so each new label value creates additional active series that must be stored and paid for. Operators prefer separate metric types for unrelated activities needing independent alerts and prefer a shared metric with a low-cardinality label when comparison or ratio math matters, while avoiding unbounded labels such as user IDs or full URLs that multiply cardinality and can hit quotas.

Querying uses a menu-driven builder, PromQL, or Monitoring filter expressions, while Monitoring Query Language (MQL) piped queries can no longer be created in the console for new charts after July 22, 2025 but still work for existing objects and API creation. PromQL queries for Monitoring metrics must map names by replacing the first / with : and all other . and / with _, so workload.googleapis.com/otlp.test.gauge becomes workload_googleapis_com:otlp_test_gauge. When a metric supports multiple resource types, the PromQL query must also filter for the resource, for example {monitored_resource="gce_instance"}, while Prometheus metrics queried without PromQL must specify the prometheus_target resource with labels for project_id, location, cluster, namespace, job, and instance.

Diagnosing the pipeline starts with Metrics Management, which reports ingestion, cardinality, reads, policy use, and write errors for billable metrics over the previous day by default. Permission constraints commonly break writes because writers need roles/monitoring.metricWriter for timeSeries.create, while descriptor create and delete apply only to user-defined metrics and never to built-in metrics. Latency and retention complete the diagnosis because new points are not immediately queryable and data is kept only for the retention period unless manually copied to Cloud Storage or BigQuery.

5.3Log-Based Metrics Design, Types, and Trade-offs

Log-based metrics in Cloud Logging turn log entries into numeric time series in Cloud Monitoring without changing application code. The Log Router evaluates each incoming entry against a filter, and when an entry matches, Logging extracts a count or numeric value and writes it as metric data. Data flows from the log entry to the Log Router to the log-based metric to Cloud Monitoring, and a match counts only when billing is enabled, project-scoped metrics pass through that project sinks, or bucket-scoped metrics are stored in that bucket.

Project-scoped metrics count logs routed through that project, while bucket-scoped metrics count all logs destined for that bucket regardless of origin. Data comes only from entries received after the metric is created, so metrics are never backfilled, and user-defined metrics count both included and excluded logs while system metrics count included logs only. User-defined log-based metrics can be only counter or distribution types, while most system metrics are counters and some are boolean checks.

Distribution objects summarize many entries per interval with count, mean, sum of squared deviations, and histogram bucket counts with values per bucket. As each entry arrives its extracted number is added to the current distribution, then at intervals the distribution is written to Monitoring for heatmaps and percentile overlays. Teams tracking strict objectives often define custom buckets around service-level thresholds because boundaries shape percentile accuracy.

Labels allow one metric to produce multiple series by extracting values such as username or operation from matching entries, with a default log label holding the LOG_ID portion of logName. A filter should specify a single resource type such as resource.type="gce_instance" so Monitoring lists the metric under Compute Engine VMs; without a resource type the metric appears under multiple resources and creates charting ambiguity. Naming uses logging.googleapis.com/SYSTEM_METRIC_NAME for system metrics and logging.googleapis.com/user/USER_METRIC_NAME for user-defined metrics, and visibility follows the scoping project metrics scope. Log-based metrics cannot count by error group or extract an error group ID into a label.

Direct instrumentation through the API, OpenTelemetry, or Prometheus collectors emits high-frequency gauges, fine-grained histograms, and business metrics even when no log is written, with precise control over sampling and cardinality. Log-based metrics only observe what is logged, add load to ingestion and storage, and provide no historical data, so operators choose log-based metrics when the signal already exists in logs and speed matters, and choose direct instrumentation when control over timeliness or semantics is required.

Key Facts

• The Ops Agent on Compute Engine collects host metrics and system logs automatically and uses logging receivers, processors, and pipelines in config.yaml to send custom file-based application logs to Cloud Logging.
• An aggregated Cloud Logging sink with --include-children routes matching logs from all child projects to one destination, but originals remain in place unless exclusion filters are added to child _Default sinks.
• Cloud Audit Logs Admin Activity entries are always written and cannot be disabled, while Data Access entries are disabled by default because of volume and cost.
• Monarch retains Prometheus data for 24 months and evaluates PromQL across all regions and up to 3,500 projects in a metrics scope.
• User-defined log-based metrics can only be counter or distribution types, start counting only after creation, and are named logging.googleapis.com/user/NAME.