Collecting Metrics Across Hybrid and Multi-Cloud

Hybrid and multi-cloud metric aggregation is the practice of forwarding numeric time-series data from Google Cloud services, virtual machines (VMs), on-premises infrastructure, and other clouds into one queryable store in Cloud Monitoring (opens in a new tab). Distributed collectors scrape or receive metrics near each workload, then data flows from each collector to Monarch, the globally scalable datastore behind both Cloud Monitoring and Google Cloud Managed Service for Prometheus. A single Grafana instance, Metrics Explorer chart, or alerting policy can then query that global store with PromQL instead of connecting to individual Prometheus servers in each environment.

1.1Centralized Visibility with Monarch and Metrics Scopes

Monarch is the central storage and query layer that makes unified monitoring possible across regions, projects, and clouds. Data storage in Monarch retains Prometheus data for 24 months, while query evaluation in Monarch executes PromQL and unions results across all Google Cloud regions and across up to 3,500 Google Cloud projects. The metrics scope of a Google Cloud project controls which stored time series can be charted or monitored by that project, because by default the scope limits the project to time series stored by that project alone. An operator expands visibility by configuring the metrics scope to include time series stored by other projects, so alerts, dashboards, and service-level objectives (SLOs) operate over the consolidated set.

1.2Ops Agent Pipeline for VMs and Third-Party Applications

The Ops Agent is the collection pipeline for Linux and Windows VMs running on Compute Engine, including third-party applications that automatic Compute Engine system metrics do not cover. The agent collects host metrics for CPU, disk, memory, swap, GPU, and processes, plus system logs such as syslog and Windows Event Log, and it adds third-party integrations for software such as Apache Web Server, Nginx, MySQL, Redis, MongoDB, PostgreSQL, and Java Virtual Machine. The same agent can scrape Prometheus metrics through its Prometheus receiver and receive OpenTelemetry Protocol (OTLP) metrics and traces through its OTLP receiver, then send that data to the global datastore. Because the Ops Agent handles VM discovery directly on the instance, data flows from the local system and application endpoints to the agent to Cloud Monitoring without installing a separate Prometheus server in the VM environment.

Cloud Monitoring separates third-party collection by platform: third-party telemetry on Compute Engine is collected by the Ops Agent, while third-party telemetry on Google Kubernetes Engine (GKE) is collected by Google Cloud Managed Service for Prometheus. This division keeps host-level collection with the VM agent and keeps container collection with the Prometheus-compatible pipeline, even though both pipelines land in the same Monitoring backend where the same alerting, charting, and dashboard services apply.

1.3BindPlane Path for On-Premises and Hybrid Sources

BindPlane is the documented path for collecting on-premises and hybrid-cloud metrics into Cloud Monitoring alongside Google Cloud-native metrics. BindPlane extends the unified pipeline beyond environments where the Ops Agent or GKE-based collectors run natively, so diverse external infrastructure metrics are consolidated into time series stored by a Google Cloud project. Once ingested, those external series are subject to the same project metrics scope, PromQL querying, dashboards, and alerting policies as system metrics generated by Google Cloud services. Choose BindPlane when the source is on-premises or hybrid infrastructure that must be mapped into the same Monitoring project as cloud workloads for end-to-end health, saturation, and availability analysis.

1.4Google Cloud Managed Service for Prometheus Collection Modes

Google Cloud Managed Service for Prometheus is the fully managed, multi-cloud solution for Prometheus and OpenTelemetry metrics that separates collection, query evaluation, rule evaluation, and storage into different components. Data collection is handled by collectors that scrape local exporters and forward data to Monarch, and those collectors can run in GKE, on VMs, on-premises, and on any cloud, with outside-Google Cloud collectors sending to Monarch for long-term storage and global querying. Rule and alert evaluation is handled either by PromQL alerts in Cloud Monitoring that fully execute in the cloud, or by locally run rule-evaluator components that evaluate against the global Monarch store and forward fired alerts to Prometheus Alertmanager. Grafana then connects to the global Monarch store rather than to each Prometheus server, so one Grafana instance provides a unified view when collectors are deployed in every environment.

Collectors can be operated in three principal Kubernetes and VM patterns compared below. Managed collection uses a Kubernetes operator that generates scrape configuration, scales ingestion, and scopes rules through lightweight custom resources. Self-deployed collection runs the drop-in replacement binary instead of the upstream Prometheus binary, preserving existing deployment methods and configs. The OpenTelemetry Collector provides a single-agent strategy for metrics, logs, and traces in any compute or Kubernetes environment, including push-based patterns and stateless environments such as Cloud Run with a sidecar.

Accurate global querying depends on consistent resource mapping during ingestion. The OpenTelemetry path maps diverse sources by exporting with the googlemanagedprometheus exporter, extracting Kubernetes metadata with the k8sattributes processor, setting project_id, location, cluster, and namespace attributes, and populating top_level_controller_name and top_level_controller_type labels for workload identity. Managed collection discovers the same workload identity through Prometheus service-discovery metadata and adds top_level_controller_{name,type} to target labels during ingestion. First verify that a wanted metric is not already covered by an active GKE system or curated package, because filtering for a metric that is already ingested by an active package creates duplicate series in Monitoring.

Network and Service Mesh observability via Cloud Service Mesh is the collection and analysis of telemetry from Envoy proxies in the data plane to understand transit latency, traffic patterns, and inter-service communication. The data plane runs Envoy proxies as sidecars beside each workload, or as proxyless gRPC clients, while the managed control plane distributes routing and telemetry configuration. Telemetry flows from each proxy to Google Cloud Observability (opens in a new tab), where Cloud Monitoring, Cloud Logging, and Cloud Trace store it for dashboards, topology views, and deep filtering.

2.1Envoy Telemetry Reporting to Google Cloud Observability

Envoy telemetry reporting is the periodic push of signals from each sidecar proxy to the Google Cloud Observability APIs. The proxy batches and exports each signal type on its own schedule, so logs arrive fastest, metrics and topology edges arrive about every minute, and traces depend on sampling. Full telemetry can be delayed 1-2 minutes while clusters start, which is normal during rollout or restart. The reporting path depends on the proxy having the Google Cloud Observability filter configured and the Cloud Monitoring API enabled in the project.

Envoy proxies in Cloud Service Mesh report four telemetry signals:

• Logging: exported about every 10 seconds for proxy and traffic events.
• Metrics: exported about every 1 minute for service request and connection data.
• Edges for topology: incremental reports about every 1 minute, with full reports about every 10 minutes to build the service graph.
• Traces: exported according to configured sampling, typically 1 out of every 100 requests.

2.2Automatic Service Insights and Dashboards

Automatic service insights are preconfigured dashboards and topology maps in the Google Cloud console that are generated for workloads configured through the Istio APIs. Service metrics and logs for HTTP traffic inside the mesh GKE cluster are automatically ingested to Google Cloud without custom instrumentation. In-depth telemetry powered by Cloud Monitoring, Cloud Logging, and Cloud Trace lets operators filter and segment data on attributes such as source, destination, response code, and version. Service-to-service relationships show dependencies and who connects to each service, plus communication security posture and service level objectives (SLOs) for health.

Cloud Service Mesh also maintains a service registry of mesh services by name and endpoint, such as Kubernetes Pod IPs or Compute Engine managed instance group IPs. Because proxies run side-by-side with services, traffic flows from the client-side Envoy to the server-side Envoy to the destination endpoint, and both sides can emit telemetry for the same request. This two-sided view makes it possible to compare client-observed latency and errors against server-observed latency and errors to isolate transit versus service faults.

2.3Istio Telemetry API Configuration

The Istio Telemetry API is the custom resource that controls how proxies generate and customize telemetry in a managed Traffic Director control plane. In most channels only one cluster-wide configuration is supported, specifically a single Telemetry custom resource in the root namespace, default istio-system. In the rapid channel multiple configurations are supported at root, namespace, and workload-selector levels, with priority highest for selector-specific and lowest for root level. When only one configuration takes effect, any previous customization must be merged into the new resource or it will be lost.

To list the configuration, run kubectl -n istio-system get telemetry and verify that at most one object exists outside rapid channel. If multiple objects exist where only one is supported, only the first one takes effect and the others are ignored. This check matters when metrics, access logs, or tracing behavior change unexpectedly after a new Telemetry resource is applied.

2.4Diagnosing Latency, Traffic Patterns, and Communication Failures

Diagnosing mesh behavior means correlating request metrics, topology edges, logs, and sampled traces from Envoy proxies. Request metrics reveal volume, error rate, and latency distribution between services, while edges reveal the path traffic takes through the mesh over time. Trace sampling at about 1% provides end-to-end execution paths that link an ingress request to downstream calls, which helps separate a slow service from slow transit between services. When a failure affects one source-to-destination pair but not others, comparing region and zone dimensions points to routing, locality load balancing, or cross-region transit rather than a global outage.

The control plane also exports its own metrics to Cloud Monitoring for viewing in Metrics Explorer:

These control-plane metrics complement proxy telemetry: a drop in connected streams points to control-plane connectivity loss, while a shift in request count by region or zone with rising error status points to traffic shifting or failing across locations.

2.5Troubleshooting Missing or Incorrect Service Metrics

Troubleshooting missing service metrics is an ordered check from workload injection through proxy configuration to API delivery. A missing sidecar means no proxy exists to emit telemetry, a missing filter means the proxy cannot export to Google Cloud, and a wrong port name means the mesh treats the traffic as TCP rather than HTTP. Even with correct proxy state, disabled APIs, quota exhaustion, or reporting errors prevent metrics from reaching dashboards. Follow the checks in order because each later step depends on the earlier one succeeding.

1. Verify sidecar injection: Confirm pods have at least two containers including istio-proxy with kubectl -n YOUR_NAMESPACE get pods. A missing label for automatic injection or failed manual injection leaves only the application container with no mesh telemetry.
2. Verify telemetry filter in the proxy: Dump proxy config with kubectl debug targeting istio-proxy and fetching localhost:15000/config_dump. Look for the Google Cloud Observability filter with root_id and vm_id set to stackdriver_inbound.
3. Verify HTTP service identification: Confirm the Kubernetes Service port name is http or has an http- prefix. Without this name, metrics will not appear in the Services dashboard even though traffic flows.
4. Verify Cloud Monitoring API delivery: Confirm the Cloud Monitoring API is enabled, then check Traffic By Response Code for the API for errors. A large number of 429 responses indicates a quota issue, which is then checked under IAM and Admin Quotas for the Cloud Monitoring API.
5. Verify Envoy proxy logs: Review logs with kubectl -n YOUR_NAMESPACE logs YOUR_POD_NAME -c istio-proxy and search for error instances. Ignore the normal warning about mTLS PERMISSIVE mode allowing plaintext or TLS without a client certificate, and focus on reporting or configuration errors.

Google Cloud Managed Service for Prometheus (GMP) is Google Cloud's fully managed, multi-cloud solution for Prometheus and OpenTelemetry metrics. GMP collects metrics from Prometheus exporters and OpenTelemetry sources, stores the data globally, and lets teams query it with PromQL in Cloud Monitoring (opens in a new tab), Grafana, or any Prometheus API client. The service stays compatible with upstream Prometheus, supports Kubernetes, virtual machines (VMs), and Cloud Run, retains data for 24 months, and can be combined with over 6,500 free Cloud Monitoring metrics using PromQL.

GMP is built on Monarch, the globally scalable datastore also used by Cloud Monitoring. A standard Prometheus server handles collection, query, rules, and storage in one process, while GMP splits those duties. Collectors scrape local exporters and forward samples from any cloud or on-premises site to Monarch, Monarch evaluates queries across all regions and up to 3,500 Google Cloud projects in a metrics scope, Monarch stores the data, and rules are evaluated either by PromQL alerting policies in Cloud Monitoring or by a standalone rule evaluator that queries Monarch and forwards fired alerts to Prometheus Alertmanager. Grafana connects to Monarch instead of to individual servers, so one Grafana instance shows metrics from every deployment that forwards to Monarch.

3.1Collection Architecture and Collector Options

Collection architecture in GMP separates local scraping from global storage and querying. A managed collector, self-deployed collector, OpenTelemetry Collector, or Ops Agent runs near the workload, scrapes the /metrics or OTLP endpoint exposed by the application or exporter, and sends the time series to Monarch for long-term storage and global PromQL querying. Because only the collector needs local network access to the target, collectors running outside Google Cloud can still provide hybrid and multi-cloud visibility without opening the workload to the global query layer.

The four collectors differ in ownership, configuration style, and operational tradeoff:

Managed collection is the recommended approach for Kubernetes because the operator generates scrape configs, scales ingestion, and scopes rules to the right data. Self-deployed collection and the OpenTelemetry Collector receive limited assistance from Google Cloud support, while managed collection and the Ops Agent receive full assistance.

3.2Platform Scraping with PodMonitoring and ClusterPodMonitoring

PodMonitoring and ClusterPodMonitoring are the managed-collection custom resources that define which Pods to scrape and how to filter their metrics. PodMonitoring scrapes only Pods in the namespace where the PodMonitoring object is deployed, so the same manifest must be deployed in each namespace to cover multiple namespaces. ClusterPodMonitoring scrapes matching Pods in any namespace from a single cluster-scoped object. A third resource, ClusterNodeMonitoring, scrapes node components such as kubelet and cAdvisor rather than application Pods.

Filtering uses Prometheus relabeling rules inside the custom resource to keep only wanted series, which controls cardinality, cost, and duplication:

• PodMonitoring: configures collection from Pods in a specific namespace using a label selector, endpoint port, path, and interval.
• ClusterPodMonitoring: configures collection from Pods in any namespace using the same selector and endpoint fields but with cluster scope.
• ClusterNodeMonitoring: configures collection from node-level endpoints such as /metrics and /metrics/cadvisor for kubelet and cAdvisor signals.

A typical PodMonitoring manifest selects an exporter by labels and scrapes it on a named port every 30 seconds. For example, a DCGM manifest with matchLabels: app: nvidia-dcgm-exporter and endpoints: port: metrics, interval: 30s directs the managed collector to scrape the NVIDIA DCGM Exporter Pod and forward GPU utilization, performance, and health metrics to Monarch. The collector depends on the custom resource for target discovery, the Pod depends on exposing a reachable metrics port and path, and Monarch depends on the collector for delivery before the series becomes globally queryable.

3.3OpenTelemetry Instrumentation for High-Cardinality Application Metrics

OpenTelemetry instrumentation uses language SDKs in the application to emit detailed metrics and an OpenTelemetry Collector to route them to GMP. The SDK generates high-cardinality series with attributes such as HTTP route, tenant, or revision, the collector's Prometheus receiver scrapes Prometheus exporters or receives OTLP push traffic, and the googlemanagedprometheus exporter forwards the result to Monarch. This path supports push-based collection from short-lived or stateless workloads, preserves exemplars that link metrics to traces, and lets one collector carry metrics, logs, and traces to their respective backends.

Collector configuration for Kubernetes application identity relies on processors that attach resource metadata before export. The k8sattributes processor extracts metadata such as namespace from the environment, the transform/collision processor sets project_id, location, cluster, and namespace attributes, and the transform/aco-gke processor populates top_level_controller_name and top_level_controller_type labels. On Google Kubernetes Engine (GKE) clusters at version 1.32.1-gke.1439000 or later, GMP discovers those controller labels through Prometheus service-discovery metadata and, together with App Hub registration, Google Cloud Observability attaches application labels to the ingested series. For Cloud Run, the workload uses either the Prometheus sidecar for Prometheus-format metrics or the OpenTelemetry sidecar for OTLP metrics, then registers the service with App Hub so the same application attribution applies.

3.4Targeted Collection Workflow and Duplication Control

Targeted collection workflow configures GMP to ingest only specific emitted metrics instead of enabling a full built-in metrics package. Built-in GKE packages already ingest curated sets for system components, control plane, kube-state, cAdvisor and kubelet, and NVIDIA DCGM, so custom scraping is used to monitor a specific signal, to replace an unneeded package with a cheaper subset, or to collect an emitted metric that no package includes. If a custom relabeling rule keeps a metric that is also active in a package, such as kubelet_running_containers while the KUBELET package is enabled, Monarch receives duplicate series.

To configure a specific metric without duplication, follow the documented order:

1. Identify the metric and endpoint: find the metric name such as kubelet_working_pods and its endpoint such as /metrics or /metrics/cadvisor in component or cAdvisor reference docs.
2. List active packages: run gcloud container clusters describe with the monitoring component fields to see enabled packages such as SYSTEM_COMPONENTS, CADVISOR, KUBELET, or DCGM.
3. Check package coverage: compare the wanted metric against the available-metrics list for those packages and stop or adjust the plan if the metric is already ingested.
4. Verify endpoint availability: optionally query the endpoint with kubectl get --raw "/api/v1/nodes/NODE_NAME/proxy/metrics" and confirm the metric lines appear with HELP, TYPE, and samples.
5. Apply the filtered custom resource: create the PodMonitoring, ClusterPodMonitoring, or ClusterNodeMonitoring object with a relabeling rule that keeps only the wanted metric, then verify ingestion with a PromQL query for Prometheus Target data in Metrics Explorer.

This order matters because package state determines whether custom scraping will duplicate data, endpoint verification confirms the target actually exposes the series, and the custom resource only takes effect after it is applied to the correct scope. First the source must expose the metric, then the operator must select it, then Monarch must receive it before PromQL, alerting, or Grafana can use it globally.

Key Facts

• Monarch is the global datastore behind Cloud Monitoring and Google Cloud Managed Service for Prometheus that retains Prometheus data for 24 months and evaluates PromQL across all regions and up to 3,500 projects in a metrics scope.
• The Ops Agent collects third-party telemetry on Compute Engine VMs while Google Cloud Managed Service for Prometheus collects third-party telemetry on GKE, and BindPlane brings on-premises and hybrid-cloud metrics into the same Cloud Monitoring store.
• Envoy proxies in Cloud Service Mesh export logs about every 10 seconds, metrics and topology edges about every 1 minute, full topology every 10 minutes, and traces for about 1 in 100 requests.
• PodMonitoring scrapes Pods only in its own namespace while ClusterPodMonitoring scrapes matching Pods in all namespaces, and keeping a metric already covered by an active GKE package creates duplicate series in Monitoring.