Prepare and test your skills
Prepare and test your skills
Centralized log routing with sinks is the practice of collecting logs from many projects into one central place for search, analysis, and retention. A log is a record of system or application activity made of timestamped entries where each entry describes one event. Cloud Logging (opens in a new tab) stores, searches, analyzes, and alerts on log data from Google Cloud and AWS. Data flows from Google Cloud resources to Cloud Logging, then a sink routes matching entries to a chosen destination, while a copy can remain in a log bucket for short-term viewing.
Google Cloud resources in two projects send logs to central Cloud Logging, which keeps a copy in a log bucket and uses an organization-level sink to route matching entries to Cloud Storage for archival, BigQuery for SQL analysis, and Pub/Sub for real-time streaming.
Organization-level sinks support centralization because entries from many projects flow to one central destination owned by the organization. Sinks route to different services for different needs:
Controlling logging costs with exclusion filters uses rules that prevent selected log entries from being stored in a log bucket. The filters apply before storage, so excluded entries do not consume bucket retention or incur bucket storage charges. This tradeoff lowers ingestion and storage costs but removes detail, so exclusions should target noisy logs that are not needed for troubleshooting or compliance.
Cost control depends on knowing that high-volume logs drive most of the bill. Data Access audit logs can be very large and create extra storage costs, especially in busy projects. To stay on budget, teams view usage data, configure alerting policies on usage, and exclude audit logs that are not useful, such as Data Access audit logs in development projects.
Exam tip: Data Access audit logs are the most expensive to keep because of their volume — exclude them in development projects where they are not needed for compliance.
Audit logs for security and compliance are records of administrative actions, data access, system events, and policy decisions in Cloud Audit Logs. Analyzing these logs in Logs Explorer and Log Analytics shows who performed an action, on which resource, and whether access was allowed or denied. Because logs are centralized at the organization or project level, security teams can review activity across the whole environment from one query interface.
Cloud Logging organizes audit logs into four peer types that appear as log names in Logs Explorer:
activity and record configuration and administrative changes.data_access and record reads and writes of user-provided data.system_event and record Google Cloud system actions.policy and record access denials based on policy checks.Querying and monitoring audit logs is the process of finding matching entries in Logs Explorer and turning important events into alerts in Cloud Monitoring. Logs Explorer reads entries stored in log buckets, so only log types that are present and retained can be selected. If a log name option does not appear, then no audit logs of that type are available in the selected organization. Cloud Monitoring (opens in a new tab) collects metric data and creates incidents when a log entry matches an alerting policy filter, then sends a notification to the configured channel.
To retrieve audit log entries for an organization, follow this order:
activity, data_access, system_event, or policy for the audit log type you need.Policies should separate events that need immediate investigation from low-priority events, such as alerting when a specific data-access message appears in an audit log.
Retention and protection of log data controls how long logs are kept and how they are encrypted at rest. Retention is controlled by rules on the log bucket where the logs are held. Custom retention can be configured between 1 day and 3650 days to meet compliance needs. Each custom rule applies to all logs in that bucket, regardless of log type or whether the log was copied from another location.
Protection at rest relies on encryption managed by Google by default, with the option for customer control. Customer-managed encryption keys (CMEK) let an organization control and manage its own key encryption keys that protect log storage instead of Google managing the keys. Use CMEK when the organization has advanced encryption requirements that default encryption at rest does not satisfy.
Defining Service Level Indicators (SLIs) and Service Level Objectives (SLOs) turns a business key performance indicator (KPI) into a measurable reliability target from the user's perspective. A Service Level Indicator is the specific technical metric that is measured, such as request success rate or inference latency. A Service Level Objective is the target value for that metric over time, such as 99.9% success. Teams first identify the KPI affected by the system, such as revenue from recommendations, then choose the technical metrics that drive that KPI.
Common SLO targets include 99.9% successful API calls, 95th percentile inference latency below 300 ms, time to first token below 500 ms for 99% of requests, and harmful-output rate below 0.1%. Google Cloud services expose built-in metrics that can be used directly as SLIs, and custom behavior can be captured with user-defined metrics, logs-based metrics, Prometheus metrics, or OpenTelemetry instrumentation. Google Cloud Managed Service for Prometheus provides a fully managed backend for Prometheus metrics at scale, including serving frameworks such as vLLM and NVIDIA Triton Inference Server.
Monitoring at every layer means collecting infrastructure, application, service-to-service, and end-user signals so no single blind spot hides a failure. Infrastructure metrics track processor usage for CPU, GPU, and TPU, memory usage, network throughput and latency, and disk I/O across managed environments and self-managed resources such as GKE nodes and Cloud Run instances. Application metrics are defined by the team to reflect key workflows, such as time to execute a query or complete a sequence of service calls.
The four golden signals provide a shared vocabulary for user-facing health:
Because a single metric rarely tells the full story, teams should watch the distribution of a metric, not just its average, because averages and medians can hide wide imbalances across users.
| Monitoring layer | What it measures | Example in Google Cloud |
|---|---|---|
| Infrastructure monitoring | Baseline health of machines and accelerators | CPU load, GPU or TPU utilization, memory, and disk writes from Compute Engine VMs |
| Application monitoring | Performance of code-defined workflows | Query execution time captured with OpenTelemetry |
| Service monitoring | Interactions between distributed services | Error and latency between microservices observed with Cloud Service Mesh |
| End-to-end monitoring | User-perceived ability to complete critical actions | Synthetic check that checkout or login completes within thresholds |
Dashboards for real-time visibility bring related metrics, logs, and alerts onto one screen for investigation. As resources are created or applications are registered with App Hub, Google Cloud automatically creates managed dashboards that show curated health, logs, metric data, and open alerts for the application and each service and workload. When managed views are not enough, teams create a custom dashboard to control exactly what data appears and how it is displayed. A custom dashboard can display metric charts and tables, log data and error groups, alerting-policy charts, alert information, explanatory text, and events such as a reboot or crash.
Data flows into these views from system metrics, Ops Agent metrics on Compute Engine VMs, user-defined metrics, external metrics, Prometheus metrics, and logs-based metrics. An alerting policy defines the metric condition to watch, such as VM CPU utilization exceeding 80%, and the people to notify through email, Cloud Mobile App, PagerDuty, or Slack. When the condition is met, the policy creates an incident that stores status, duration, labels, a chart of metric data, and links to related logs. Health checks and synthetic monitors feed the same loop by pulling endpoints, routing around unhealthy VMs, autohealing managed instance groups, and probing HTTP, HTTPS, and TCP endpoints with uptime checks.
Distributed troubleshooting with correlated telemetry links metrics, logs, and traces around the same workload so an operator can pivot from symptom to cause. Cloud Trace is a distributed tracing system that collects latency data and displays it in near real time in the Google Cloud console. An instrumented application emits telemetry data, and Cloud Trace assembles that data into traces made of spans that show how a single request flows from service to service. The recommended approach is to instrument with OpenTelemetry, an open-source and vendor-neutral framework, instead of vendor-specific APIs.
Error Reporting aggregates and displays errors produced in running applications and groups repeated occurrences with the same signature into error groups. Application Monitoring creates dashboards and topology maps for App Hub-registered applications, while log-based metrics count matching structured log entries over time so text events flow from Cloud Logging to Cloud Monitoring for visualization and alerting. Monitoring, Logging, and Trace are enabled by default when a Google Cloud project is created and automatically collect telemetry and audit logs for most services.
| Signal | Primary data | Choose when |
|---|---|---|
| Cloud Trace | Latency data organized as traces and spans | The need is to follow a request across services and find the slow hop |
| Error Reporting | Aggregated error groups from exceptions | The need is to find which errors are frequent, new, or growing |
| Log-based metrics in Cloud Monitoring | Time series derived from structured logs | The need is to track volumes, latency, and error rates and alert on trends |
Continue with this study guide: Monitoring and logging
Benchmark targets in Google Cloud are defined with service level objectives (SLOs), service level indicators (SLIs), and service level agreements (SLAs). An SLO states a measurable target such as 99.9% availability or average response time under 200 ms, and an SLI is the specific metric tracked over time to measure that SLO. An SLA is the contractual commitment that describes the services provided, the expected level of service, the responsibilities of provider and customer, and penalties for noncompliance. Cloud Profiler (opens in a new tab) attributes CPU and memory consumption to source code, while Cloud Monitoring collects metrics for availability, performance, and latency to check whether critical services meet their SLOs and SLAs.
Performance baselines with continuous profiling are built by observing production applications while they serve real traffic. Test environments usually fail to reproduce production pressure, and microbenchmarks of isolated code usually fail to reproduce real workload behavior, so continuous profiling reveals where CPU cycles and memory are actually consumed. A single profile is created by collecting data usually for 10 seconds, every 1 minute, for a single instance of a configured service in a single Compute Engine zone. For example, if a Google Kubernetes Engine (GKE) service runs 10 pod replicas, roughly 10 profiles are created in a 10-minute period and each pod is profiled approximately once, with randomized timing.
Matching resource types to workload requirements means choosing Google Cloud resources based on availability, scalability, security, performance, and cost needs. A stateless application does not need the same availability as a stateful backend, and processor instruction sets benefit some workloads more than others. For globally distributed mission-critical workloads, Spanner keeps data reliable and consistent in all regions without complex database deployments. For workloads with fluctuating load, autoscaling adds capacity when load rises and removes it when load falls, with limits to stay within budget.
| Workload type | Workload requirements | Resource options |
|---|---|---|
| Mission-critical | Continuous availability, robust security, and high performance | Premium resources and managed services like Spanner for high availability and global consistency |
| Non-critical | Cost-efficient and autoscaling infrastructure | Basic features and ephemeral resources like Spot VMs |
| Event-driven | Dynamic scaling based on current demand | Serverless services like Cloud Run and Cloud Run functions |
| Experimental | Low cost and flexible environment for rapid iteration | Basic features, Spot VMs, and sandbox environments with spending limits |
Load and stress testing validates whether applications and infrastructure can handle peak demand and maintain performance. Load testing simulates realistic traffic patterns to confirm the system meets its baseline and SLOs under expected conditions. Stress testing pushes the system to its limits to identify bottlenecks and performance limitations that do not appear at normal load. Tools like Cloud Load Balancing and load testing services generate real-world traffic for both types of tests.
Based on test results, teams optimize by adjusting resource allocation, tuning configurations, or adding caching. For example, if an application slows during high traffic, the team might increase the number of virtual machines (VMs) or containers or adjust the web server or database configuration to improve performance.
Exam tip: Load testing simulates realistic traffic patterns, while stress testing pushes the system to its limits to identify bottlenecks.
Continuous observability with Google Cloud Observability tracks whether live behavior still matches the established baseline. Google Cloud Observability is a unified platform for collecting, analyzing, and visualizing metrics, logs, and traces from Google Cloud services, applications, and external sources. Data flows from resources and applications to Cloud Monitoring, where teams gain insight into resource utilization, performance characteristics, and overall health. Comprehensive monitoring tracks system health indicators that align with established baselines:
By correlating resource utilization with application performance, teams can decide whether to move to less expensive resources without harming user experience. Active Assist provides intelligent recommendations such as VM rightsizing insights to avoid unnecessary spending.
Correlating profiling data with system-wide observability means linking a hot function in Cloud Profiler to service-level and business-level signals from monitoring. Profiling answers which code consumed resources, while utilization metrics, traces, and logs answer whether that consumption created a bottleneck, left resources idle, or changed user experience. Traffic flows from users to microservices to infrastructure, the profiling agent reports code-level cost to Cloud Profiler, and the services report utilization and performance to Cloud Monitoring. Effective correlation starts with business-relevant metrics that show impact, not just infrastructure activity:
Use observability for resource footprint optimization by monitoring CPU and memory utilization to find idle VM resources, viewing GKE cost breakdown and cost-optimization metrics, and using Active Assist recommendations. The key decision rule is to correlate resource utilization with application performance before moving to less expensive resources.
Performance regression detection is the comparison of current profiles and service metrics against a pre-deployment baseline to find new hot spots. Cloud Profiler supports focusing the flame graph to isolate a service or function, comparing profiles from different time periods or versions, and viewing historical trends to see when consumption shifted. When a new deployment increases CPU time, heap allocation, wall time, or contention in a specific call stack and latency or error rate rises at the same time, the changed code is the likely cause. SRE error budgets and DORA change failure rate help decide whether a regression requires a rollback, a fix-forward, or acceptance within the safety margin.
Resource footprint optimization is the continuous adjustment of allocation to match actual demand so services avoid both over-provisioning and under-provisioning. Over-provisioning creates unnecessary cost, while under-provisioning creates performance bottlenecks that affect user experience. Cloud Monitoring provides real-time visibility into usage patterns, Recommender analyzes that data to suggest right-sizing, and custom metrics with alerts trigger timely adjustments.
| Option | Unit scaled | Scales in response to |
|---|---|---|
| Compute Engine managed instance groups (MIGs) | Group of VMs managed as a single entity with minimum and maximum size | CPU utilization thresholds and other autoscaling policies |
| Google Kubernetes Engine (GKE) autoscaling | Nodes and pods through Cluster Autoscaler, Horizontal Pod Autoscaler, Vertical Pod Autoscaler, and Node Auto-Provisioning | Pod resource demands, CPU, memory, custom metrics, and usage patterns |
| Cloud Run | Serverless container instances | Incoming traffic volume, scaling up on growth and down on decline |
Cost strategies complement right-sizing by changing how capacity is purchased and tracked. Options include committed use discounts for committed usage, sustained use discounts for consistent Compute Engine usage, and Spot VMs for unused capacity at lower cost.
Profile types in Cloud Profiler describe which resource is measured, and support for each type depends on the program language. CPU time measures processor consumption, heap measures memory in use, and wall time measures total elapsed time including waiting. A CPU-heavy function points to compute bottlenecks that require more vCPU or more replicas, while high heap use points to memory pressure that can force larger machines. Wall time reveals slow paths that include waiting for locks, I/O, or downstream calls, even when CPU use looks low.
| Language | CPU time | Heap | Wall time |
|---|---|---|---|
| Go | Yes | Yes | Not supported |
| Java | Yes | Yes | Yes |
| Node.js | Not supported | Yes | Yes |
| Python | Yes | Not supported | Yes |
The profiling agent is the language-specific library attached to an application that collects data as the application runs. The agent runs on Linux with glibc for Go, Java, Node.js, and Python, and on Linux with musl with Alpha limitations for Java and Python. Supported locations include Compute Engine, GKE, App Engine flexible and standard environments, Managed Service for Apache Spark for Java, Dataflow for Java and Python, and environments outside Google Cloud. The profiling workflow moves data from code to analysis in order:
Collection overhead is less than 5 percent during CPU and heap collection and commonly less than 0.5 percent when amortized over time and across replicas, which makes always-on profiling affordable.
Continue with this study guide: Profiling and benchmarking
Cloud Monitoring (opens in a new tab) alerting connects a policy, an incident, and a notification channel into one response flow. The alerting policy defines what data to watch, what condition means trouble, and how to notify responders. When data from Monitoring or Cloud Logging meets that condition, Monitoring creates an incident that records what was monitored and when the condition was met, and then sends notifications to each channel in the policy. A single policy can list one or more notification channels, so the same incident can email my-support-team@example.com and post to #my-support-team at the same time.
After a policy is enabled, Monitoring evaluates its conditions continuously, and when the condition for a metric-based policy is no longer met, Monitoring automatically closes the incident and sends a closure notification if configured. By default, notifications are sent only when incidents are opened, and responders are notified about closure only if Notify on incident closure is selected. An alerting policy can evaluate time-series metric data, individual log entries, or the results of a SQL query against logs in Observability Analytics.
Metric-based alerting policies monitor numerical time-series data for threshold violations, missing data, complex expressions, and forecasted problems. Each policy condition selects a resource type and metric, narrows the time series with filters and group-by aggregation, and then compares the aligned values against a static or dynamic threshold for a specific duration. For example, a policy can watch HTTP response latency and create an incident only when latency stays above two seconds for at least five minutes, which prevents a brief spike from paging the team.
The four time-series condition types differ by what pattern they detect and how far they look back:
| Condition type | What triggers it | Example use |
|---|---|---|
| Metric-threshold condition | Metric value is above or below a threshold for a specific retest window | Notify when response latency is 500ms or higher for five consecutive checks over 10 minutes |
| Metric-absence condition | Monitored time series has no data for a specific retest window, up to 24 hours | Open an incident when a resource receives no HTTP requests for five minutes |
| Prometheus Query Language (PromQL) condition | PromQL expression using combinations, ratios, or dynamic thresholds evaluates to true | Alert on a ratio of two metrics or evaluate more than 25 hours of data |
| Forecasted metric-value condition | Forecast predicts a threshold violation within a forecast window of 1 hour to 7 days | Warn when disk usage is likely to reach 80% within the next 24 hours |
Choose a metric-threshold condition for simple high-or-low checks, a metric-absence condition for silent resources, a PromQL condition for ratios and multi-metric logic, and a forecasted condition for capacity problems that have not happened yet. Use resource labels from the monitored-resource type to select time series, because resource metadata labels are added asynchronously and can make sensitive policies behave erratically if their population is delayed.
Log-based alerting policies notify responders when a specific message appears in included logs stored by Cloud Logging. The condition is met when a log entry matches the policy filter, such as a message field containing specific product IDs. This approach fits immediate events like an authentication failure, a failed integration execution, a specific severity log, or an execution time that exceeds a value.
To monitor recurring events in logs over time, log-based metrics first convert logging data into numerical time-series data that a metric policy can evaluate and chart. Log-based metrics are suitable when you want to count message occurrences like warnings or errors and notify when the number crosses a threshold, observe trends in log-extracted values like latency and notify if values change unacceptably, and create charts that display numeric data extracted from logs for trend analysis and dashboards.
Exam tip: A log-based alerting policy fires on an individual matching log entry, while a log-based metric counts or trends recurring log events over time before a metric threshold fires.
Accurate alerting policies combine a precise filter, an appropriate threshold duration, and clear response information. The condition defines the data source, threshold, and aggregation through filters and group-by, and it can monitor a single metric, multiple metrics, or a ratio of metrics. Longer durations and retest windows, such as requiring five minutes above threshold instead of firing on one point, reduce false positives while still catching sustained latency, error-rate, or utilization problems.
Clear operations practices keep alerts actionable after they are created:
A policy cannot be scheduled to monitor only during certain hours, so create a snooze to silence a policy for a defined period, and rely on automatic incident closure to confirm when the system has returned to normal.
SLO-based alerting defines reliability from the user's perspective and alerts only when that reliability is at risk. The SRE approach turns reliability into a measurable engineering property by defining Service Level Objectives that quantify acceptable reliability and performance from the user's perspective. Clear SLOs align engineering work with business needs, so reliability efforts focus on the system behavior that most affects users. Google Cloud Monitoring and Cloud Logging together provide the metrics, dashboards, and alerting mechanisms used to track those goals.
For each business KPI, such as revenue influenced by AI recommendations, customer churn, or user engagement, teams define corresponding technical metrics such as request success rate, latency, harmful-response rate, and successful task completion. Those metrics are quantified with targets such as 99.9% successful API calls, 95th percentile latency below 300 ms, time to first token below 500 ms for 99% of requests, and harmful-output rate below 0.1%. The four golden signals track health as latency, traffic, error rate, and saturation of CPU, memory, GPU, or TPU. OpenTelemetry captures metrics, traces, and logs, while Managed Service for Prometheus provides a managed backend at scale, Cloud Trace pinpoints request flows, Cloud Profiler finds code bottlenecks, and Error Reporting groups errors.
Alerting policies in Cloud Monitoring evaluate different kinds of data:
| Policy type | Data evaluated | Choose when |
|---|---|---|
| Metric-based alerting policy | Time-series metric data stored by Monitoring | Response latency stays above 2 seconds for 5 minutes or CPU, error rate, or quota usage crosses a threshold |
| Log-based alerting policy | Individual log entries or log-based metrics stored by Cloud Logging | A specific error message appears, failed executions spike, or latency values from logs change unacceptably |
| SQL-based alerting policy | Results of a SQL query in Observability Analytics against logs | The answer requires aggregating and analyzing log data with SQL before deciding to alert |
Effective SLO-based alerts prioritize symptoms that burn the error budget and affect users, with clear, actionable information and appropriate severity levels.
A multi-channel notification strategy routes the same incident to different tools based on urgency, role, and location of the responder. Log alerts and metric alerts both use notification channels to notify the on-call team when a policy is triggered, and supported integrations include email, Pub/Sub for private endpoints and automation, PagerDuty for paging and escalation, Slack for team chat, SMS for urgent mobile alerts, and webhooks for custom HTTP endpoints. For third-party tools, use webhooks to send to any public endpoint such as ServiceNow, Datadog, or OpsGenie, and use Pub/Sub to send to any private endpoint. Alerts are also sent to essential contacts configured in the project, so project contacts and policy channels work together.
Alert documentation is user-defined text stored with the alerting policy and included in the notification message. The Documentation field lets teams add context such as what the alert means, likely causes, triage steps, and links to runbooks or playbooks. Because this text travels with the incident notification, a responder who receives email, Slack, or PagerDuty sees the same instructions without searching for them. Alertmanager receives alerts from defined rules and handles routing, grouping, silencing, and inhibiting to reduce noise, where grouping combines related alerts, silencing pauses notifications, and inhibiting suppresses lower-priority alerts when a higher-priority condition explains them.
Continue with this study guide: Alerting strategies
Cloud Storage receives log data for low-cost, long-term archival and compliance preservation, BigQuery receives log data for large-scale SQL analysis and security investigation, and Pub/Sub receives log data for real-time streaming to other applications or third-party tools. Organization-level sinks support centralization because entries from many projects flow to one central destination owned by the organization.
Exclusion filters are rules that prevent selected log entries from being stored in a log bucket, so excluded entries do not consume bucket retention or incur bucket storage charges. This lowers ingestion and storage costs but removes detail, so exclusions should target noisy logs that are not needed for troubleshooting or compliance, such as Data Access audit logs in development projects.
A Service Level Indicator is the specific technical metric that is measured, such as request success rate or inference latency, while a Service Level Objective is the target value for that metric over time, such as 99.9% success. Teams first identify the business key performance indicator (KPI) affected by the system, such as revenue from recommendations, then choose the technical metrics that drive that KPI.
Last updated