Application and Platform Instrumentation with Managed Service for Prometheus and OpenTelemetry

Metric Collection Architecture

Google Cloud Managed Service for Prometheus (GMP) is a fully managed, multi-cloud solution that collects and monitors metrics across diverse environments. To standardize metrics collection, GMP integrates with the OpenTelemetry framework, which provides a unified agent for capturing metrics, logs, and traces. Within Kubernetes environments, teams configure managed data collection using PodMonitoring and ClusterPodMonitoring custom resources. These resources specify the targets, scraping intervals, ports, and paths for metrics collection, shielding operations teams from manually managing Prometheus servers.

Collection Options and Decision Criteria

DevOps teams can choose between several data collection methods depending on their infrastructure requirements:

• Managed collection is the primary choice for standard Kubernetes setups because custom resources automate scraping configurations.
• Self-deployed collection works best as a drop-in replacement for existing upstream Prometheus servers to preserve complex legacy configurations.
• OpenTelemetry Collector serves as a single-agent push-based solution for environments requiring unified handling of metrics, logs, and traces.
• Ops Agent simplifies the pipeline by capturing host-level metrics directly from Google Compute Engine virtual machines.

Storage, Querying, and Alerting

Once collected, the metrics flow into Monarch, Google's globally scalable data store, which retains metric data for up to 24 months. Users can query this centralized data using PromQL through the Cloud Monitoring UI, Grafana, or any tool compatible with the Prometheus API. Alerting policies can be created directly in Cloud Monitoring using PromQL, or executed locally via a standalone rule evaluator component that routes alerts to an Alertmanager. This dual path allows teams to maintain historical visibility and keep existing alerting workflows operational.

Advanced Workload Integration

This unified telemetry framework enables automated platform scaling and serverless monitoring. For instance, Horizontal Pod Autoscaling (HPA) in Google Kubernetes Engine (GKE) can scale workloads by leveraging custom Prometheus metrics. To achieve this, the system requires an active PodMonitoring resource and a Kubernetes Metrics Adapter to expose those metrics to the HPA controller. Similarly, Cloud Run services can deploy an OpenTelemetry Collector sidecar, which writes OTLP metrics directly to GMP to enable serverless workload observability.

Hybrid and Multi-Cloud Metric Aggregation and Integration

Secure Unified Pipelines

Establishing a single pane of glass across multi-cloud environments requires a pipeline that links external telemetry with Google Cloud. DevOps teams can use BindPlane by observIQ alongside the Ops Agent to collect, map, and consolidate diverse infrastructure metrics. This unified pipeline routes raw system data from on-premises servers and other cloud providers directly into Google Cloud Monitoring. Consolidating this data ensures that platform administrators have consistent dashboards to track overall platform health.

Hybrid Data Collection and Metadata Mapping

In Google Distributed Cloud environments, dedicated agents gather and translate local system performance data. The GKE Metrics Agent runs as an OpenTelemetry Collector-based DaemonSet to capture container metrics, while the Metadata Agent maps local Kubernetes resource metadata to those metrics. These agents work in tandem to append context, such as cluster names and namespaces, before securely forwarding the telemetry to the central Monarch data store. This structure preserves container relationship context as metrics cross the hybrid network boundary.

Long-Term Retention and Optimization

Because Google Cloud Monitoring stores metric data for only six weeks, long-term historical analysis requires an export strategy. Teams can set up a route to transfer aggregated metric data into BigQuery to perform deep trend analysis while keeping ingestion costs low. To manage these costs, administrators can use the Metrics Management page to audit metric cardinality, identify write errors, and exclude unnecessary metrics. This optimization step reduces the volume of redundant data sent to Monarch, ensuring cost-efficient storage for up to 24 months.

Network and Service Mesh Observability via Cloud Service Mesh

Telemetry Collection and Frequency

Cloud Service Mesh delivers out-of-the-box observability for distributed systems by automatically collecting HTTP traffic metrics, logs, and traces. The system relies on Envoy proxies running as sidecars to periodically report these telemetry signals to Google Cloud Observability APIs. Envoy sidecars emit log data approximately every 10 seconds, while metric collection occurs every minute, and topology edge data updates every 1 to 10 minutes. During cluster startup, this telemetry reporting flow can face a 1-to-2 minute delay, which engineers must account for when diagnosing initial connection issues.

Troubleshooting and Verification Steps

When service-to-service metrics are missing or incorrect, operators must troubleshoot the path from the workloads to the monitoring API:

• Verify that Sidecar proxies exist and are successfully injected by confirming the presence of the istio-proxy container in the application pods.
• Extract a configuration dump from the proxy to verify that the Google Cloud Observability filter is configured and active.
• Check that the Cloud Monitoring API is enabled on the Google Cloud project and is free of execution errors.
• Review Envoy proxy logs for transmission failures and confirm that the metric.mesh_uid is set correctly.

Proxyless gRPC Observability

For workloads that do not use sidecars, Cloud Service Mesh leverages the gRPC OpenTelemetry plugin to collect latency and payload metrics. This proxyless approach uses environment variables such as GRPC_XDS_BOOTSTRAP, POD_NAME, NAMESPACE_NAME, CSM_WORKLOAD_NAME, and OTEL_RESOURCE_ATTRIBUTES to discover and describe running containers. The mesh attaches detailed attributes to the exported metrics, including csm.mesh_id and csm.remote_workload_type, to model traffic patterns across different projects and clusters. These attributes map service relationships, letting teams analyze inter-service latency even when bypassing standard sidecar configurations.

Control Plane and Backend Integrations

The service mesh supports diverse monitoring backends, allowing operators to choose the most suitable platform. Teams can route HTTP and TCP metrics directly to Cloud Monitoring, scale metric collection through GMP, or integrate third-party tools such as Grafana, Jaeger, and Zipkin. While the Istio Telemetry API provides detailed configurations for access logs and traces, the TRAFFIC_DIRECTOR control plane only supports a subset of these capabilities. Therefore, engineers must align their telemetry policies with the limitations of their chosen control plane architecture.