Prepare and test your skills
Prepare and test your skills
Monitoring and alerting strategies in Cloud Monitoring (opens in a new tab) turn raw service behavior into dashboards and timely alerts. Metrics flow from applications and infrastructure into Cloud Monitoring, dashboards visualize health and performance, and alerting policies notify operators when thresholds or service-level objectives (SLOs) are at risk. This loop supports proactive management because problems are detected from data before users report widespread impact.
Reliability goals connect business outcomes to measurable technical targets for a service. For each key performance indicator, such as customer satisfaction or successful task completion, teams define technical reliability metrics and quantify them with SLO targets. Example targets from reliability guidance (opens in a new tab) include 99.9% of API calls returning success, 95th percentile inference latency below 300 ms, time to first token below 500 ms for 99% of requests, and harmful output below 0.1%.
SLOs focus on four main areas:
SLO monitoring in Cloud Monitoring helps monitor microservice health by setting alerting policies on SLO performance. SLO monitoring provides automatic support for Cloud Service Mesh, Istio on Google Kubernetes Engine (GKE), App Engine, and Cloud Endpoints, and supports custom services for GKE. Use of this feature requires no code changes in applications.
The four golden signals are the core metrics for monitoring a user-facing system in Cloud Monitoring. Latency, traffic, errors, and saturation together show whether a service is fast, used, correct, and close to capacity limits. Teams also track infrastructure metrics across resources used by the system, including CPU, GPU, and TPU usage, memory usage, network throughput and latency, and disk I/O.
The four golden signals are:
A single average value can hide wide imbalances, so Cloud Monitoring strategies consider metric distribution. The 99th percentile is often more informative than the average because it reveals poor experiences affecting a meaningful share of requests. Teams can create custom metrics in Cloud Monitoring to capture combinations of conditions that no single built-in metric describes.
Custom dashboards in Cloud Monitoring collect, store, and visualize infrastructure and application metrics tailored to a workload. Pre-built dashboards cover Google Cloud services, while custom dashboards focus on workload-specific performance indicators and infrastructure health. Metrics from specialized frameworks can be integrated through Google Cloud Managed Service for Prometheus, including vLLM and NVIDIA Triton Inference Server, and Agent Platform metrics exported to Cloud Monitoring can also be charted.
Cloud Monitoring retains metrics for several weeks for real-time health checks and recent investigations. For longer-term analysis, metrics can be exported and stored to study outages, identify bottlenecks and dependencies, and validate scaling during high-traffic events. Complementary signals come from Cloud Logging for event and error context, Cloud Trace for request flow and latency across distributed microservices, and Cloud Profiler (opens in a new tab) for bottlenecks within function blocks in application code.
| Tool | What it collects | Use it to |
|---|---|---|
| Cloud Monitoring | Infrastructure and application metrics | Build dashboards and define threshold alerts |
| Cloud Logging | Detailed application and infrastructure logs | Add context around events and errors |
| Cloud Trace | Distributed latency and request traces | Pinpoint latency and follow multi-service flows |
| Cloud Profiler | Function-level performance profiles | Identify code bottlenecks and optimize execution time |
Alerting policies in Cloud Monitoring compare a metric to a threshold and generate an incident when the condition is met. Throughput can be monitored with appengine.googleapis.com/http/server/response_count and error rate with logging.googleapis.com/log_entry_count. For example, error rate can be monitored by filtering log_entry_count for App Engine apps with severity of ERROR, then creating an alerting policy that fires when the count exceeds the threshold.
Alerting policies can also be created on SLOs to warn when an SLO is in danger of violation, including alerts on budget burn rate. During resilience and failover testing (opens in a new tab), teams observe changes in latency, error rates, and throughput and verify that logs are generated and alerts trigger for key events such as outages or failovers. Health signals can drive recovery as well, since checks can autoheal managed instance groups by re-creating unhealthy machines, while Kubernetes liveness and readiness probes help the orchestrator manage pods and requests.
Exam tip: Filter logging.googleapis.com/log_entry_count by severity ERROR to alert on error rate, and use SLO burn-rate alerts when the goal is protecting the SLO rather than firing on every short metric spike.
Optimizing performance through trace and profiler analysis means using two complementary signals to find why an application is slow. Cloud Trace (opens in a new tab) shows where time is spent as a request travels across distributed services, while Cloud Profiler (opens in a new tab) shows which function blocks inside the code consume the most resources and execution time. Together they connect outward latency with inward resource use, so fixes target the true bottleneck instead of guessing.
Cloud Trace is a distributed tracing system that collects latency data from applications and displays it in near real time in the Google Cloud console. Each request becomes a trace made of spans, where each span records one individual operation such as an API call, database query, or downstream service call. Because spans from different microservices are linked into one end-to-end view, Cloud Trace reveals request flow across complex pipelines such as agent conversations or multi-component inference paths.
To examine a trace, data flows from the instrumented application to the Trace explorer in the console:
The waterfall view in Cloud Trace separates total time from its parts. The top-level parent bar shows total request duration, or the full time the caller waits for the operation to finish. Child bars show network and server latency (opens in a new tab) for each downstream step, such as the time a command spends traveling across the network and running on the server. Common bottleneck patterns appear as shapes in that timeline:
Cloud Profiler analyzes performance bottlenecks within function blocks in application code to guide CPU and memory optimization. While Cloud Trace answers which service or call was slow, Cloud Profiler answers which function consumed the resource and increased execution time. That inward view helps teams rewrite, cache, or parallelize the hottest code paths to lower resource usage and shorten run time.
Cloud Profiler connects directly to saturation, or how close critical resources are to capacity limits. Saturation covers processor usage including CPU, GPU, and TPU, plus memory usage, network throughput and latency, and disk I/O. When saturation is high on GKE nodes, Cloud Run instances, or training and serving infrastructure, Profiler data shows whether the pressure comes from a few expensive functions or from broad inefficiency across the codebase. Fixing the functions that dominate CPU and memory lowers both cost and latency for later requests.
Choosing between Trace and Profiler depends on the performance question being asked, and Cloud Monitoring provides the surrounding metrics and alerts. Cloud Trace fits outward latency that may span services, networks, pools, or retries, while Cloud Profiler fits inward cost where CPU, memory, or execution time is high inside application code. Teams use reliability signals (opens in a new tab) from Cloud Monitoring to decide which tool to open first.
| Tool | Primary unit | Choose when |
|---|---|---|
| Cloud Trace | Trace with parent and child spans | Latency is high and the cause may span services, networks, pools, or retries |
| Cloud Profiler | Function block profile | CPU, memory, or execution time is high and the cause is inside application code |
Cloud Monitoring ties the two tools to the four golden signals for application health: latency, traffic, error rate, and saturation. Latency measures time to respond, traffic measures request volume, error rate measures failed operations, and saturation measures utilization of CPU, memory, and accelerators. Teams collect these infrastructure and application metrics in Cloud Monitoring dashboards, collect event context in Cloud Logging, then drill into Cloud Trace for request flow and Cloud Profiler for code hotspots. A sampled database query can also link outward, since its End-to-end Trace tab shows a Gantt chart of spans with attributes and metadata, and View in Cloud Trace opens the same trace in the Trace explorer for deeper analysis.
Exam tip: Never rely on the average for latency; graph p95 or p99 percentile latency to reveal the slow requests that averages hide.
Diagnosing application failures with logging and error reporting means using telemetry to find what broke, where it broke, and which defect to fix first. Google Cloud Observability (opens in a new tab) provides integrated monitoring, logging, and trace managed services for applications and systems running on Google Cloud and beyond. Cloud Logging (opens in a new tab) captures log entries from infrastructure, platforms, and applications, while Error Reporting surfaces problems related to mishandling of exceptional conditions so developers can prioritize critical software defects.
Google Cloud Observability is the suite that connects system health, logs, and performance signals into one troubleshooting workflow. Log data flows from workloads and agents to Cloud Logging, metrics flow to monitoring systems, and traces and profiles explain latency and resource use. The suite includes the following components:
Because the components share the same projects and resources, a failure seen as an error can be correlated with CPU, memory, network, and disk signals. For day 2 operations, Active Assist provides recommendations to reduce costs, increase performance, and improve security. Choose Cloud Logging and Error Reporting when the question is what failed in code, and add Monitoring, Trace, and Profiler when the question is how system health and performance contributed to the failure.
Cloud Logging is the service that ingests log entries through the Cloud Logging API and makes them searchable in Logs Explorer. Application containers should write to stdout or stderr, the logging agent on the node forwards those entries to the Cloud Logging API, and Logs Explorer displays the entries that match the selected scope and query. The pipeline depends on three controls: the Cloud Logging API must be enabled for the project, the node service account must have permission to write logs, and no log exclusion filter may drop the entries before storage.
When expected logs do not appear, narrow the cause by symptom before changing code. Permission errors such as 401 or 403 in agent logs point to a missing service-account permission, while intermittent loss with RESOURCE_EXHAUSTED points to exceeded Cloud Logging API write quota. Consistently missing patterns often mean a log exclusion filter is dropping them, an overly restrictive Logs Explorer query is hiding them, or the entries passed their retention period and were deleted. From an experiment or resource view, select View in Log Explorer to expand scope and adjust filters without losing context.
Error Reporting is the observability capability for detecting and triaging application exceptions and mishandled error conditions. Errors detected in logs and runtime signals are grouped so teams can see which failure affects the most requests and should be fixed first. This makes Error Reporting the complement to raw log search: Cloud Logging shows the sequence of events, while Error Reporting highlights the defect behind repeated crashes, caught exceptions, and failed requests.
Use Error Reporting together with resilience controls (opens in a new tab) when failures reach users. Traffic management with Cloud Load Balancing, filtering with Cloud Armor (opens in a new tab), and log-based metrics and alerting policies in Cloud Monitoring can reduce impact while the code fix is prepared. Create logs-based metrics and alerting policies that notify operations when error volume rises, and export critical logs to a central security or operations system when audit and long-term analysis are required.
Verifying compute and cluster signals confirms whether an application failure comes from code or from exhausted infrastructure. Install the Ops Agent to expose full instance metrics such as memory and disk space utilization, then open performance data in the console by going first to VM Instances, then to the instance Details page, then to the Observability tab:
On the Network Summary page, Sent to and Received from VMs, External, and Google destinations, plus Packet Mean Size and Firewall Incoming Packets Denied, show whether traffic was blocked or abnormal. On the Disks Performance page, I/O Size Avg, Queue Length Avg, and I/O Latency Avg show whether small random operations were limited by IOPS or large sequential operations were limited by throughput. For Google Kubernetes Engine (GKE) workloads, also verify cluster logging configuration, node pool access scopes, node throughput and CPU or memory pressure, and container buffering that delays output until container exit.
Exam tip: A 401 or 403 from the logging agent means fix service-account permissions, while RESOURCE_EXHAUSTED means fix Cloud Logging API quota — both hide logs but require different fixes.
stdout or stderr through the Cloud Logging API for search in Logs Explorer, while Error Reporting groups exceptions so teams fix the most impactful defect first.The four golden signals are latency, traffic, error rate, and saturation, which together show whether a service is fast, used, correct, and close to capacity limits. The 99th percentile latency is often more informative than the average because it reveals poor experiences affecting a meaningful share of requests that a single average value can hide.
Use Cloud Trace when latency is high and the cause may span services, networks, connection pools, or retries, because it shows where time is spent as a request travels across distributed services with parent and child spans. Use Cloud Profiler when CPU, memory, or execution time is high and the cause is inside application code, because it identifies which function blocks consume the most resources and execution time.
Alerting policies in Cloud Monitoring compare a metric to a threshold and generate an incident when the condition is met. Policies can also be created on service-level objectives to warn when an objective is in danger of violation, including alerts on budget burn rate.
Missing logs often mean the Cloud Logging API is not enabled, the node service account lacks permission to write logs, a log exclusion filter dropped the entries, the Logs Explorer query is too restrictive, or the entries passed their retention period. Permission errors such as 401 or 403 in agent logs point to a missing service-account permission, while intermittent loss with RESOURCE_EXHAUSTED points to exceeded Cloud Logging API write quota.
Last updated