Correlating Metrics with Metrics Explorer

Correlating metrics across services and resources is the practice of querying related time series together to trace user-visible degradation back to its origin. In Cloud Monitoring (opens in a new tab), no single chart explains an outage, so analysis compares frontend signals such as latency and errors against backend signals such as CPU utilization, memory pressure, and storage errors over the same time window. Causal confidence comes from aligning timing, scope, and distribution across those dimensions rather than from any single spike.

1.1Metric Sources That Feed Cross-Service Correlation

The correlated data set in Cloud Monitoring comes from several collection paths that are stored as numeric time series in a Google Cloud project. System metrics from Google Cloud services are collected automatically (opens in a new tab), while deeper guest and application signals require agents or instrumentation. All sources become comparable in Metrics Explorer because they share the same time-series model and metrics scope.

• Google Cloud service metrics: automatically collected system metrics such as the more than 25 Compute Engine metrics reported per virtual machine (VM) instance.
• Ops Agent metrics: CPU, disk, memory, and swap collected from Linux and Windows Compute Engine VMs, plus third-party telemetry from plugins such as Apache, Nginx, MongoDB, and PostgreSQL.
• Prometheus metrics: application and Kubernetes signals collected by Google Cloud Managed Service for Prometheus, or by the Ops Agent through the Prometheus receiver or OTLP receiver.
• User-defined metrics: application-specific measurements instrumented with OpenTelemetry or written through the Cloud Monitoring API.
• Log-based metrics: numeric counts derived from Cloud Logging, including Google-defined error counts and custom counts such as 404 responses from App Engine.
• External metrics: signals defined by open-source libraries or third-party providers.

The metrics scope of a project controls which time series can be charted together. By default the scope limits a project to time series stored by that project, and the scope must be configured to include time series stored by other projects for multi-project correlation. Cloud Monitoring retains metrics for several weeks for real-time health checks and recent investigations, while longer-term storage supports comparison across outages, promotional peaks, and architectural changes.

1.2Time-Series Labels and Kinds That Enable Multi-Dimensional Slicing

A Cloud Monitoring time series is identified by what is measured, where it was measured, and how adjacent points relate. The metric.type field names the service and measurement, such as compute.googleapis.com/instance/cpu/utilization, and the labels field carries key-value pairs defined by the MetricDescriptor, such as instance_name or instance_id. Cloud Monitoring writes one time series for each combination of resource and metric label values, so filtering by one label isolates a single VM while grouping by another label aggregates across machine types.

• GAUGE: stores the value at a moment in time, such as hourly temperature or current CPU utilization.
• CUMULATIVE: stores the accumulated total at a moment in time, such as a vehicle odometer.
• DELTA: stores the change over a period, such as gains or losses since the prior summary.

Interpreting kind correctly prevents false causation because a rising CUMULATIVE line may reflect normal accumulation while a rising GAUGE line reflects growing pressure. The valueType field defines the measurement type as INT64, DOUBLE, BOOL, STRING, or DISTRIBUTION, and DISTRIBUTION values preserve the full spread needed for percentile analysis. Grouping and aggregation then turn labeled series into comparable signals, such as average CPU utilization grouped by machine_type plotted beside request latency for the same interval.

1.3Comparing Related Metrics in Metrics Explorer

Metrics Explorer is the chart service for quickly visualizing and exploring time-series data without building a full dashboard first. The explorer selects a monitored resource type and metric, then refines the view with filtering, grouping, and aggregation before overlaying additional metrics on the same timeline. Charts can be saved to a custom dashboard once a useful correlation view is found.

1. Select resource and metric: choose the Configuration tab, select a resource type such as Cloud Run service or Compute Engine VM, and select a metric from the menu.
2. Refine scope: use Filter to isolate one instance or service, use Group By to split by resource or metric labels, and use Aggregator to align statistics across series.
3. Overlay and persist: add related metrics to the same chart, compare against previous periods, show outliers and percentiles, and save the chart to a custom dashboard.

Google Cloud-managed dashboards are created automatically for instantiated resources and for App Hub applications, services, and workloads, showing curated logs, metrics, and open alerts. Custom dashboards are required when managed views do not expose the needed trend, outlier, or multi-metric comparison, and custom dashboards can combine metric charts and tables with log data, error groups, alert charts, alert status, text, and events such as reboots or crashes. Comparing current behavior to previous behavior in the same view separates a genuine regression from normal daily variation.

1.4Connecting Infrastructure Signals to User-Facing Health

Root-cause analysis connects low-level saturation to user-facing service level objectives (SLOs) (opens in a new tab) through a small set of descriptive signals. No single metric reliably indicates an unhealthy service, so Cloud Monitoring encourages combining signals and creating custom metrics that capture the joint condition. Distribution matters because averages and medians can hide wide imbalances across instances, zones, or operations.

• Latency: time to complete work, often judged by 50th, 95th, and 99th percentiles rather than the average.
• Traffic: load or request rate placed on the service.
• Errors: rate of failed work, including backend error rates and log-derived error counts.
• Saturation: exhaustion of CPU, memory, disk, network, GPU, or queue capacity.

Google Kubernetes Engine (GKE) observability exposes these signals at infrastructure and application layers, including CPU and core request utilization, memory request utilization, cAdvisor CPU and memory and filesystem and network statistics, container restart counts, unschedulable pod counts, startup latency, control-plane health, and GPU and Tensor Processing Unit (TPU) utilization. Workload application dashboards report per-second request rate by operation, error rate by operation and response code, 50th and 95th percentile latency by operation, and CPU and memory use as a percentage of requested amount. SLOs defined through service monitoring turn those platform and service metrics (opens in a new tab) into reliability targets, and alerting policies on burn rate warn when infrastructure pressure is consuming the user-facing budget.

1.5Service Dependencies That Shape Correlation Paths

Correlation paths follow the dependency from the user-facing request layer to the storage and processing layers that complete the work. Google Cloud services expose Monitoring metrics automatically, so each layer contributes request, error, latency, and saturation signals that can be overlaid by time and label. Traffic flows from the request-response service to the data storage and retrieval service or data-processing service it depends on, and degradation propagates in the same direction as described in patterns for scalable and resilient apps (opens in a new tab).

End-to-end monitoring tests externally visible behavior the way a user sees it and can reveal latency or availability loss that fine-grained checks miss. Health checks pull status from an endpoint so the load balancer can decide whether an instance should continue receiving traffic and whether a managed instance group should autoheal unhealthy machines. When frontend errors rise without frontend saturation, the correlated backend error or saturation spike that starts first is the stronger candidate for the origin.

Differentiating metric types means recognizing what kind of numbers a metric carries so the visualization matches the analytical goal of state, throughput, or latency. Analysis in Metrics Explorer starts with this distinction because the kind determines whether a chart should show a current value, a rate of change, or a spread of measurements. Config Sync (opens in a new tab) custom metrics provide the working example, using the form custom.googleapis.com/opencensus/config_sync/METRIC where METRIC is a name such as reconciler_errors or reconcile_duration_seconds.

2.1Gauge and Last Value Metrics for System State

Gauge and Last Value metrics report the state at measurement time and can move up or down. The documented defaults include kcc_resource_count as Gauge and declared_resources, resource_count, ready_resource_count, resource_ns_count, cluster_scoped_resource_count, resource_group_total, last_sync_timestamp, pipeline_error_observed, and reconciler_errors as Last Value. A line chart over time is the natural visualization because each point answers how many resources exist, how many are ready, or when the last sync occurred. Almost all Config Sync metrics carry a reconciler label, so the same Last Value metric can be filtered to one RootSync reconciler such as root-reconciler or one RepoSync reconciler such as ns-reconciler-NAMESPACE, and the line then tracks health for that sync object alone.

2.2Count Metrics for Throughput and Error Volume

Count metrics record totals that grow, such as apply_operations_total, internal_errors_total, and resource_fights_total. A raw cumulative total answers how many operations have happened since counting began, while a delta or rate derived from that total answers throughput and error velocity over the selected window. Choose a line showing rate when the analytical goal is throughput or burn-down of errors, and choose a stacked bar broken down by labels such as component or status when the goal is to compare which part contributes most to the total. The reconciler_errors metric is labeled by component, so filtering and grouping by component shows where in the import, source, and sync pipeline the errors occurred.

2.3Distribution Metrics for Latency and Duration

Distribution metrics capture many individual observations as a histogram rather than a single number. The documented defaults include api_duration_seconds, apply_duration_seconds, and reconcile_duration_seconds as Distribution. Distributions are the correct kind for latency questions because averages alone hide slow outliers, while percentiles and heatmaps reveal how the spread shifts over time. Use a heatmap when the goal is to see the full distribution of durations across buckets, and use lines of selected percentiles when the goal is to track typical versus tail latency against a service objective.

2.4Selecting, Filtering, and Verifying Metrics in Metrics Explorer

Metrics Explorer selection combines the full metric name, labels, and chart form to match the analytical goal of state versus throughput versus latency. Metrics without data do not appear in Metrics Explorer, so an absent metric means no samples were written for the selected time and filters, not necessarily zero. The comparison below summarizes how the documented kinds map to goals and charts:

Filtering turns a fleet-wide metric into a diagnostic signal by naming both sides of the relationship: data flows from the selected reconciler Pod to the charted time series. To isolate one sync object in Metrics Explorer, complete the following ordered workflow:

1. Open Monitoring: Go to Monitoring in the Google Cloud console and open Metrics explorer.
2. Select metric: Enter custom.googleapis.com/opencensus/config_sync/reconciler_errors in the Select a metric control.
3. Add label filter: Choose reconciler in the Filter control to expose the reconciler identity dimension.
4. Set match: Set the operator to = and the value to the reconciler name, for example root-reconciler.
5. Apply: Click Apply so the chart queries only time series from that reconciler Deployment.
6. Interpret: Read the resulting line or bar as errors from that source of truth to cluster sync flow, not from the fleet.

Constructing and refining metric queries means selecting a precise metric name, isolating only the relevant time series with label filters, and then combining those series over resources and time to reveal health, trends, and anomalies. In Cloud Monitoring, Config Sync (opens in a new tab) metrics provide a complete example because almost every metric is tagged by reconciler and by component, and the metrics use different value types that determine how they should be read over time. A refined query therefore flows from metric selection to label filtering to time-window interpretation.

3.1Config Sync Metric Names and Value Types

Config Sync metric names in Cloud Monitoring always use the form custom.googleapis.com/opencensus/config_sync/METRIC. The prefix custom.googleapis.com identifies the time series as a custom metric, opencensus identifies the instrumentation library used by Config Sync, config_sync/ identifies the exporter, and METRIC is the specific measurement such as reconciler_errors or reconcile_duration_seconds. Selecting the full name is the first step in Metrics Explorer because only the exact match returns the intended streams.

The value type controls how a time series should be aggregated and interpreted across a window:

Counts expose rate and volume when summed over time, last values and gauges expose current state and drift when compared across windows, and distributions expose typical versus tail behavior when viewed by percentile bands. Choosing the wrong reading, such as treating a last-value resource count as if it were a summed event total, hides whether the system is stable, growing, or failing.

3.2Filtering Queries by Reconciler and Component Labels

Filtering by labels in Metrics Explorer isolates one data stream from many by keeping only time series where a resource label or metric label matches a value. Config Sync depends on the reconciler label because each RootSync object and each RepoSync object creates its own reconciler Deployment Pod that syncs manifests from a source of truth to a cluster. A RootSync named root-sync creates a reconciler named root-reconciler, while any other RootSync name creates root-reconciler-ROOT_SYNC_NAME; a RepoSync named repo-sync creates ns-reconciler-NAMESPACE, while any other RepoSync name creates ns-reconciler-NAMESPACE-REPO_SYNC_NAME-REPO_SYNC_NAME_LENGTH. Filtering by that exact reconciler name separates cluster-scoped RootSync activity from namespace-scoped RepoSync activity.

The reconciler_errors metric adds a second refinement dimension through its component label, which identifies where in the import, source, and sync pipeline the error occurred. Filtering first by reconciler isolates which sync loop is unhealthy, then filtering or grouping by component isolates which stage inside that loop is failing. This two-level refinement turns one noisy error stream into an actionable diagnosis of which object and which pipeline stage needs attention.

A reconciler-scoped query in Metrics Explorer follows an ordered workflow:

1. Open Monitoring and Metrics Explorer: Data flows from the Cloud Monitoring store into the Metrics Explorer query builder.
2. Select the metric: Enter custom.googleapis.com/opencensus/config_sync/reconciler_errors in the Select a metric control.
3. Add a filter: Open the Filter control and select the reconciler label.
4. Set the match: Choose = as the operator and enter the reconciler name, for example root-reconciler.
5. Apply the filter: Metrics Explorer retains only time series where the reconciler label equals that value.
6. Refine further: Add a component filter or change the time window to compare the filtered series against its baseline.

3.3Interpreting Time Series to Identify Baselines, Trends, and Anomalies

Time-series interpretation in Metrics Explorer compares current values against an established baseline to distinguish normal change from failure. Config Sync exposes both state metrics that define the baseline and event metrics that signal deviation from it. State metrics such as declared_resources, resource_count, ready_resource_count, resource_ns_count, and cluster_scoped_resource_count show how many resources should exist and how many are actually ready, while last_sync_timestamp shows whether the sync loop is still advancing. Event metrics such as internal_errors_total, resource_fights_total, apply_operations_total, and pipeline_error_observed show whether operations are succeeding or colliding over the same window.

A healthy pattern shows last_sync_timestamp advancing on schedule, ready counts tracking declared and total counts, and error counters remaining flat. An anomaly appears as a break in that relationship, such as ready counts falling below total counts while reconciler_errors rises for one reconciler, apply operations stalling while reconcile duration grows, or pipeline_error_observed changing state without a corresponding successful sync. Because the query is already filtered by reconciler and component, the analyst can attribute the anomaly to a specific RootSync or RepoSync loop and to a specific pipeline stage rather than to the cluster as a whole.

Key Facts

• Cloud Monitoring stores system, agent, Prometheus, user-defined, log-based, and external signals as numeric time series that Metrics Explorer can overlay by time and label for root-cause analysis.
• A metrics scope shows only its own project's time series by default and must be expanded to include time series stored by other projects for multi-project correlation.
• GAUGE stores a point-in-time value, CUMULATIVE stores an accumulated total, and DELTA stores change over a period, so a rising CUMULATIVE line does not imply growing pressure the way a rising GAUGE line does.
• Config Sync metrics use the form custom.googleapis.com/opencensus/config_sync/METRIC and use the reconciler label to separate RootSync activity from RepoSync activity.
• Distribution metrics such as reconcile_duration_seconds require heatmaps or percentile lines for latency analysis, while Count metrics require a rate for throughput and Gauge or Last Value metrics use a line for state.