Prepare and test your skills
Prepare and test your skills
Proactive issue detection means creating alerting policies that fire before users notice degradation. An alerting policy in Cloud Monitoring (opens in a new tab) continuously evaluates telemetry against a condition and creates a persistent incident when the condition is met. A mature strategy combines threshold checks for known limits, forecast and absence checks for exhaustion and silence, expressive queries for service-level objective violations, and synthetic probes for external availability.
Cloud Monitoring alerting connects data, policy, incident, and notification in a fixed flow. Data flows from time-series metric data stored by Monitoring or log data stored by Cloud Logging into an alerting policy, then from the policy to an incident record, then from the incident to each configured notification channel. The alerting policy describes when to respond and who to notify, the incident stores status, labels, duration, metric charts, and links to logs, and the notification channel defines delivery such as email, Cloud Mobile App, PagerDuty, or Slack.
An alerting policy can evaluate metric data, individual log entries, or SQL query results over logs. Metric-based policies monitor time-series data, log-based policies fire when a matching phrase appears in a log entry, and SQL-based policies in Public Preview fire when an aggregated query result over log data meets criteria. A classic example is HTTP response latency above two seconds for at least five minutes, which creates an incident and emails the support team while tolerating brief fluctuations.
An alerting policy is defined by its condition, its notification channels, and its documentation. The condition specifies the data source, filters and grouping, aggregation, and a static or dynamic threshold, and it can monitor a single metric, multiple metrics, or a ratio of metrics. Notification channels list one or more destinations for action, and documentation controls the notification subject and body where teams embed links to playbooks, custom dashboards, and Google Cloud pages.
Policies are managed through the Google Cloud console, the Cloud Monitoring API, the Google Cloud CLI, or Terraform, and integrations can supply pre-configured recommended policies. Once enabled, Monitoring continuously evaluates the condition with no schedule for business hours only, so to suppress notifications temporarily teams create a snooze rather than disabling evaluation. If an incident is open and the metric condition stops being met, Monitoring moves the incident to closed automatically and sends a closure notification.
Time-series conditions in Cloud Monitoring provide four distinct triggers for early detection. Each condition type watches the same underlying time-series store but answers a different failure question, from immediate breach to predicted exhaustion to missing data. Threshold conditions catch known bad states, while forecasted conditions buy lead time for capacity action before the limit is reached.
| Condition type | What triggers it | Proactive use |
|---|---|---|
| Metric-threshold condition | Metric value is above or below a static threshold for a retest window | Detect saturation and latency, such as response latency at 500ms or higher for five consecutive uptime checks over 10 minutes |
| Metric-absence condition | No data arrives for a retest window, up to 24 hours maximum | Detect silent failures, such as a resource with no HTTP responses for five minutes |
| Prometheus Query Language (PromQL) condition | A PromQL expression evaluates to true in real time, including ratios and dynamic thresholds | Detect SLO burn, error-ratio shifts, and patterns spanning more than 25 hours of data |
| Forecasted metric-value condition | Forecast predicts a threshold violation within a forecast window of 1 hour to 7 days | Detect resource exhaustion early, such as disk likely to reach 80% usage within the next 24 hours |
Absence conditions are essential because a dead agent or stalled pipeline produces no bad values to threshold on, only silence. Forecasted conditions extend detection days ahead for disk, memory, or queue growth, while PromQL conditions handle complex signals that a single threshold cannot express.
Exam tip: Choose a forecasted metric-value condition when you must act before exhaustion, such as disk at 80% within 24 hours, and choose a PromQL condition when the signal needs a ratio, combined metrics, or more than 25 hours of data.
A PromQL condition in Cloud Monitoring evaluates time-series data with any valid expression, including metric combinations, ratios, and dynamic thresholds based on another metric. Every alerting policy or chart starts from a query that selects the data to monitor, built from console menus or written directly in the query editor with syntax checks and suggestions. Prometheus metrics reach Monitoring through Google Cloud Managed Service for Prometheus or through the Ops Agent with the Prometheus receiver or OpenTelemetry Protocol (OTLP) receiver, so the same expression style can be reused from public repositories.
Precise queries depend on correct label selection when resource and metric labels overlap. If a monitored resource includes a cluster label and the metric includes a similar kubernetes_cluster_name label, filter on the resource cluster label to select time series consistently. Do not rely on system- or user-defined resource metadata labels for sensitive alerting policies, because metadata labels are added asynchronously and are not subject to the same latency expectations as metric-type or monitored-resource-type labels, which can make triggering erratic when population is delayed.
SLO-focused alerting in Cloud Monitoring translates user-facing reliability goals into metric and log signals. Effective targets follow the four golden signals of latency, traffic, error rate, and saturation, with concrete SLO examples such as 99.9% successful API calls, 95th percentile inference latency below 300 ms, time to first token below 500 ms for 99% of requests, and harmful-output rate below 0.1%. Burn-rate or ratio alerts built with PromQL detect error-budget consumption across services, while threshold alerts on saturation metrics such as CPU utilization above 80% catch resource pressure.
Abnormal behavior that evades simple thresholds needs log-aware and predictive conditions. Log-based policies fire when an individual entry matches, such as a message containing specific product IDs, and SQL-based policies fire on aggregations such as more than 50 WARNING entries in the last 60 minutes. Forecasted conditions extend this to capacity by predicting disk, memory, or queue growth days ahead, and metric-absence conditions catch instrumentation loss. Together these cover the progression from early anomaly to SLO breach to user impact.
Synthetic monitors in Cloud Monitoring test availability, consistency, and performance from outside the service before real traffic fails. An uptime check probes HTTP, HTTPS, and TCP endpoints for responsiveness and opens an incident path when an endpoint fails to respond, while a broken-link checker crawls a web page and notifies when broken links are detected. Because probes run continuously and independently of user load, they detect network, DNS, certificate, and routing failures that internal metrics may miss.
Synthetic results feed the same alerting workflow as metric data. A failing probe triggers its alerting policy, creates an incident with latency and failure details, and notifies the configured channels, so teams triage external symptoms alongside internal saturation and error signals. Teams create and manage synthetic monitors with the console, API, CLI, or Terraform, just like alerting policies, to keep proactive checks versioned with the services they protect.
Optimizing observability data means shaping logs, metrics, and traces so failures can be found quickly without storing and querying unnecessary telemetry. In Google Cloud, in-process instrumentation generates telemetry inside the application and exports it to a Google Cloud project, where Cloud Logging (opens in a new tab) stores log data, Cloud Monitoring stores metric data, and Cloud Trace stores trace data for correlated troubleshooting. Most services generate basic logs and metrics automatically, so optimization focuses on adding detail where it speeds diagnosis and aggregating or excluding it elsewhere.
A structured log entry in Cloud Logging is a timestamped record of a single event that combines a message with machine-readable fields such as severity, customer ID, error message, stack trace, and source-code location. Severity and labels add troubleshooting context to the raw message, so a label that indicates severity or a customer ID lets operators filter for high-impact failures and related activity. Error Reporting analyzes log entries from Cloud Logging, annotates the associated entries, and groups them into an error group that shows cause and history.
Log data carries rich detail about what happened in one part of an application, but log data alone does not show how a change in one component relates to activity in others. Logs Explorer is the interface for viewing individual log entries and finding related entries, and it annotates entries when they belong to an error group or when trace data is available. Observability Analytics is the interface for querying log data with SQL to find trends and patterns, such as average HTTP latency for a URL over time and whether latency fluctuates.
To turn log data into action, Cloud Logging supports two different notification paths. A log-based alerting policy notifies when a particular pattern appears, which fits important but rare events such as a service-account key access in an audit log or a deployment message. A log-based metric counts matching entries or extracts numeric values into histograms, and an alerting policy then monitors that metric for trends such as error counts or latency shifts. The choice depends on whether the team needs to act on one entry or on a trend across many entries.
| Notification path | Monitors | Choose when |
|---|---|---|
| Log-based alerting policy | Single matching log entry | The event is rare but important, such as a security access or deployment |
| Log-based metric with alerting policy | Count or distribution derived from many entries | The concern is a trend, such as rising error counts or changing latency |
Exam tip: Use a log-based alerting policy for a single rare event in log data, and use a log-based metric for counts and latency trends that need charts and threshold alerts.
Custom metrics are application-defined numeric signals measured at regular intervals that reveal behavior system metrics cannot, such as request latency, queue depth, token usage, or business-specific success rates. Instrumentation is code inside the application process that interacts with the application to generate telemetry, and the instrumentation framework then exports that telemetry to a configured storage system such as a Google Cloud project. OpenTelemetry (opens in a new tab) is the recommended vendor-neutral framework for this work because it is not tied to one vendor, provides a standardized way to collect telemetry across languages, and offers interoperable libraries for traces and metrics.
Google Cloud collects system metric data by default and ingests custom and third-party metric data through dedicated paths. Google Cloud Managed Service for Prometheus handles third-party and user-defined metric data for Google Kubernetes Engine (GKE), including exporters for software such as Postgres, MongoDB, and Redis. On Compute Engine, the Ops Agent can collect Prometheus metric data and OpenTelemetry Protocol (OTLP) metric data in addition to host CPU, GPU, memory, and process metrics. On Cloud Run, a service that writes Prometheus metrics uses a Prometheus sidecar to send data to Cloud Monitoring, while a service that writes OTLP metrics uses an OpenTelemetry sidecar.
Custom metrics are most useful when they cover the four golden signals for the application. Charting these signals in Metrics Explorer or a custom dashboard lets teams detect unexpected changes, compare patterns over time to anticipate resource needs, and link a metric shift to related logs and traces.
A trace in Cloud Trace represents the path of a single end-to-end request across the components of a distributed application. Each trace is composed of spans, where each span records a single function or operation, so following spans in order shows how the request flows from service to service and where latency accumulates. For agent applications, traces also capture agent actions such as Model Context Protocol (MCP) calls, and trace analysis can derive metrics such as number of model calls and total token usage.
Trace data bridges the gap left by logs and metrics by connecting isolated events into one causal path. Instrumentation collects domain-specific detail from the language runtime, framework, or application logic, then exports spans to the Google Cloud project where Cloud Trace visualizes latency per span. When spans point to slow function blocks, Cloud Profiler analyzes CPU and memory usage to identify opportunities to improve performance and reduce execution time. Correlating the trace with structured logs that share the same request context then reveals the exact error message and code location behind the slow or failed span.
Balancing fidelity with cost means collecting detailed telemetry where it speeds diagnosis and aggregating or excluding it where full detail adds cost without value. Most Google Cloud services automatically generate log data and metric data, so Compute Engine, GKE, and Cloud Run provide system metrics and logs by default, while Cloud Run functions and App Engine automatically generate metric, log, and trace data. Additional collection is explicit and therefore controllable: Compute Engine VMs need the Ops Agent for host details, third-party integrations, and application logs from stdout or stderr; GKE clusters use observability metric packages such as control-plane metrics for availability and latency SLOs; Cloud Run needs a sidecar for custom Prometheus or OTLP metrics.
The tradeoff is that verbose logs and full trace capture preserve every troubleshooting clue but increase ingestion, storage, and query load. Reducing cost without losing diagnosis favors aggregation and selection: log-based metrics retain counts and latency histograms for alerting and charts, preconfigured dashboards and alert policies for third-party integrations reuse curated signals, and prompt-and-response data is kept for quality evaluation rather than mixed into every operational log. The result is high fidelity for errors, high-severity events, and latency outliers, with lower-volume summaries for routine successful operations.
Diagnosing root causes from telemetry data anomalies is the practice of correlating signals across metrics, logs, and traces to separate infrastructure failures from application failures. Google Cloud Observability (opens in a new tab) provides the single pane of glass and storage for metrics, logs, traces, and events, with dashboards, reporting, and alerting built on top. A metric spike shows what changed and when, a log pattern deviation shows what the system recorded at that time, and a trace latency increase shows where time was spent along the request path.
Correlation in Google Cloud Observability starts with a shared request identity that links the three signals together. Cloud Logging (opens in a new tab) access log entries carry trace, spanId, traceSampled, request_id, and logName fields, where trace has the form projects/YOUR-PROJECT/traces/TRACE-ID. Cloud Trace stores the end-to-end trace for that same trace ID, while Cloud Monitoring shows the aggregated metric spike for the same time window. From Logs Explorer you can expand a log and select View trace details, or from Trace explorer you can select a span and open the Logs and Events tab to see the linked logs.
Traffic flows from the client workload to the server workload, and each side records its own view that must be compared during diagnosis. Server access logs give the server-side view of requests and are attached to the k8s_container monitored resource under server-accesslog-stackdriver. Client access logs give the client-side view and are attached to the k8s_pod monitored resource under client-accesslog-stackdriver. Both log types contain HTTP request properties such as ID, URL, size, latency, and headers, plus mesh identity labels such as source and destination workload, namespace, principal, and response_flag.
| Log type | Monitored resource | View it provides |
|---|---|---|
server-accesslog-stackdriver | k8s_container | Server-side view of request handling, routing, and failures |
client-accesslog-stackdriver | k8s_pod | Client-side view of outbound requests and observed latency |
Comparing the two views shows whether latency or errors were seen by the client, the server, or both, which separates network and mesh problems from application handling problems.
Latency diagnosis for APIs and Services moves from aggregate metrics to specific requests in a fixed order. First, open APIs and Services in the console, select Enabled APIs and Services, open Metrics, and select the API method under Methods to view the Traffic by API method graph. Then enable audit logging by selecting Admin Read, Data Read, and Data Write so method calls emit detailed audit entries. Only after the metric spike and method are identified, drill into individual high-latency conversations by time and method name.
The drill-down depends on Cloud Logging Log Analytics and Logs Explorer querying the same audit data. In Log Analytics, filter on TIMESTAMP_DIFF to find conversations longer than a threshold, for example longer than 10 seconds on a particular day. In Logs Explorer, filter on protoPayload.methodName for the method name and on Timestamp for the spike window, then inspect results in Log fields. The workflow depends on audit logging being enabled beforehand, because without Data Read and Data Write entries there are no per-conversation records to correlate with the metric spike.
Control plane telemetry distinguishes a slow configuration rollout from a data plane application failure. When Cloud Service Mesh (opens in a new tab) uses the in-cluster control plane, istiod exports metrics prefixed with istio.io/control to Google Cloud Observability, covering proxies connected per control plane instance, configuration events, pushes, and validations. The pilot_proxy_convergence_time metric specifically measures the delay between a configuration change and all proxies converging. A rise in convergence time alongside proxy disconnects or failed pushes points to the control plane, while flat control plane metrics with rising data plane latency points to the application or backend.
You can inspect control plane state by installing the sample Istio Control Plane Dashboard and by scraping the pilot endpoint directly. The dashboard shows connected clients, pushes, and validations, while the direct scrape shows current convergence measurements.
gcloud monitoring dashboards create.http://localhost:15014/metrics, then filter for convergence to read pilot_proxy_convergence_time.Missing logs are themselves an anomaly that can mimic or hide an application failure, so diagnosis must verify the pipeline before concluding the application is silent. Configuration causes include a disabled Cloud Logging API for the project, cluster-level logging disabled for required log types, a node pool lacking the required access scope, and a node service account missing permission shown as 401 or 403 errors in agent logs. Resource and performance causes include exceeding the Cloud Logging API write quota shown as RESOURCE_EXHAUSTED, or a node exceeding logging agent throughput limits and CPU or memory during high load.
Filtering and application behavior causes explain why only specific logs are absent while other cluster logs remain present. These causes sit on the customer side of the trust boundary and must be checked before assuming a platform outage.
stdout, so container logs appear delayed until container exit.stdout or stderr, so no container logs exist for that Pod.Platform causes sit on the other side of the trust boundary from customer configuration. Logs older than the retention period have been deleted by Cloud Logging, widespread loss or delay indicates a Cloud Logging service issue or ingestion delay, and version limitations indicate an unsupported Google Kubernetes Engine version. The gcpdiag open source tool can automatically inspect the project and cluster for common misconfigurations when logs are missing or incomplete.
Exam tip: Log entries can be visible in Logs Explorer several minutes before the matching trace is viewable in Trace explorer, so wait and retry trace lookup before concluding the trace is missing.
Reliable diagnosis depends on retaining the right telemetry and being able to query it quickly under load. For compliance retention, set up log sinks for the organization, and for fast security and access analysis export logs to BigQuery. To analyze logs stored in a log bucket, run SQL queries through Observability Analytics, and for sensitive projects enable Data Access audit logs to track who accessed data. To remove sensitive fields such as Social Security numbers, credit card numbers, and email addresses, filter with a custom Fluent Bit configuration or with ingestion exclusions, while exporting unfiltered logs separately when compliance requires it.
The Metrics Management page in Cloud Monitoring reports diagnostics that help control billable metric spend without losing observability. You can use that page to exclude unneeded metrics and eliminate ingestion cost while keeping the signals used in alerts and dashboards.
The same tradeoff applies to hybrid collection: Google Cloud Managed Service for Prometheus collects all exposed metrics by default, each becoming chargeable by samples ingested, so implement cost controls to avoid unexpected charges. Fluent Bit and BindPlane logs imported into Cloud Logging are chargeable at standard Logging rates, and the log forwarder buffers locally and retries for up to four hours before dropping logs when it cannot reach the Cloud Logging API.
Use a log-based alerting policy in Cloud Logging when you need to act on a single rare but important log entry, such as service-account key access in an audit log or a deployment message. Use a log-based metric with an alerting policy when you need to monitor a trend across many entries, because the metric counts matching entries or extracts numeric values into histograms for charts and threshold alerts on signals such as error counts or latency shifts.
Use a forecasted metric-value condition in Cloud Monitoring when you must act before exhaustion, because it predicts a threshold violation within a forecast window of 1 hour to 7 days such as disk likely to reach 80 percent usage within the next 24 hours. Use a Prometheus Query Language (PromQL) condition when the signal needs a ratio, combined metrics, or more than 25 hours of data, such as SLO burn or error-ratio shifts.
In Cloud Monitoring alerting, data flows from time-series metric data stored by Monitoring or log data stored by Cloud Logging into an alerting policy, then from the policy to an incident record, then from the incident to each configured notification channel. The alerting policy describes when to respond and who to notify, the incident stores status, labels, duration, metric charts, and links to logs, and the notification channel defines delivery such as email, Cloud Mobile App, PagerDuty, or Slack.
A team of DevOps engineers is redesigning the observability architecture for a distributed microservices application running on Google Cloud. The application currently produces hundreds of gigabytes of unstructured string logs per day, experiences high Cloud Trace ingestion costs due to un-sampled tracing, and struggles with query latency during production incident investigations.
Which strategy should the team implement to optimize observability telemetry for efficient troubleshooting while balancing operational costs?
Last updated