Prepare and test your skills
Prepare and test your skills
Centralized Logging and Resource Optimization is the practice of collecting logs from many projects into one central place to search, analyze, retain, and control costs. Cloud Logging (opens in a new tab) stores, searches, analyzes, monitors, and alerts on log data and events from Google Cloud and AWS. A log is a record of system or application activity over time, made of timestamped entries where each entry describes one specific event.
Centralized log routing with sinks uses a routing rule in Cloud Logging to send matching log entries to a chosen destination for storage or analysis. Data flows from Google Cloud resources to Cloud Logging, then the sink routes matching entries to the destination. A copy can remain in a log bucket for short-term viewing, and organization-level sinks support centralization because entries from many projects flow to one central destination owned by the organization.
Sinks route to different Google Cloud services for different retention and analysis needs:
Optimizing logging costs with exclusion filters uses rules that prevent selected log entries from being stored in a log bucket. The filters apply before storage, so excluded entries do not consume bucket retention or incur bucket storage charges. This tradeoff lowers ingestion and storage costs but removes detail, so exclusions should target noisy logs that are not needed for troubleshooting or compliance.
Cost control depends on knowing that high-volume logs drive most of the bill. Data Access audit logs can be very large and create extra storage costs, especially in busy projects. To stay on budget, teams view usage data, configure alerting policies on usage, and exclude audit logs that are not useful, such as Data Access audit logs in development projects.
Exam tip: Data Access audit logs are the most expensive to keep because of their volume — exclude them in development projects where they are not needed for compliance.
Audit logs for security and compliance are records of administrative actions, data access, system events, and policy decisions in Cloud Audit Logs. Analyzing these logs in Logs Explorer and Log Analytics shows who performed an action, on which resource, and whether access was allowed or denied. Because logs are centralized at the organization or project level, security teams can review activity across the whole environment from one query interface.
Cloud Logging organizes audit logs into four peer types that appear as log names in Logs Explorer:
activity and record configuration and administrative changes.data_access and record reads and writes of user-provided data.system_event and record Google Cloud system actions.policy and record access denials based on policy checks.Querying and monitoring audit logs is the process of finding matching entries in Logs Explorer and turning important events into alerts in Cloud Monitoring. Logs Explorer reads entries stored in log buckets, so only log types that are present and retained can be selected. If a log name option does not appear, then no audit logs of that type are available in the selected organization.
To retrieve audit log entries for an organization, follow this order:
activity, data_access, system_event, or policy for the audit log type you need.Monitoring continues the flow from Cloud Logging to Cloud Monitoring when a log entry matches an alerting policy filter. Cloud Monitoring then sends a notification to the configured channel. Policies should separate events that need immediate investigation from low-priority events, such as alerting when a specific data-access message appears in an audit log.
Retention and protection of log data controls how long logs are kept and how they are encrypted at rest. Retention is controlled by rules on the log bucket where the logs are held. Custom retention can be configured between 1 day and 3650 days to meet compliance needs. Each custom rule applies to all logs in that bucket, regardless of log type or whether the log was copied from another location.
Protection at rest relies on encryption managed by Google by default, with the option for customer control. Customer-managed encryption keys (CMEK) let an organization control and manage its own key encryption keys that protect log storage instead of Google managing the keys. Use CMEK when the organization has advanced encryption requirements that default encryption at rest does not satisfy.
Performance monitoring and alerting strategies connect business goals to technical behavior and catch problems before users feel them. Cloud Monitoring (opens in a new tab) collects, stores, and visualizes metric data from Google Cloud services, while Cloud Logging, Cloud Trace, and Cloud Profiler add event context, request-flow detail, and code-level bottlenecks. Together they support a cycle where teams define reliability targets, watch system layers, build dashboards for real-time visibility, and fire alerts that create actionable incidents.
Defining Service Level Indicators (SLIs) and Service Level Objectives (SLOs) turns a business key performance indicator (KPI) into a measurable reliability target from the user's perspective. A Service Level Indicator is the specific technical metric that is measured, such as request success rate or inference latency. A Service Level Objective is the target value for that metric over time, such as 99.9% success. Teams first identify the KPI affected by the system, such as revenue from recommendations, then choose the technical metrics that drive that KPI, such as success rate, time to first token (TTFT), token streaming rate, harmful-output rate, or training throughput like model FLOPS utilization (MFU) and tokens per second.
Common SLO targets include:
Google Cloud services expose built-in metrics that can be used directly as SLIs, and custom behavior can be captured with user-defined metrics, logs-based metrics, Prometheus metrics, or OpenTelemetry instrumentation. Request-response services include Cloud Endpoints, Cloud Run, Cloud Run functions, App Engine, and Google Kubernetes Engine (GKE). Data storage and retrieval services include Cloud Storage, Bigtable, Spanner, and Datastore, and data-processing services include Dataflow and Managed Service for Apache Spark. Google Cloud Managed Service for Prometheus provides a fully managed backend for Prometheus metrics at scale, including specialized serving frameworks such as vLLM and NVIDIA Triton Inference Server.
Monitoring at every layer means collecting infrastructure, application, service-to-service, and end-user signals so no single blind spot hides a failure. Infrastructure metrics track processor usage for CPU, GPU, and TPU, memory usage, network throughput and latency, and disk I/O across managed environments and self-managed resources such as GKE nodes and Cloud Run instances. Application metrics are defined by the team to reflect key workflows, such as time to execute a query or complete a sequence of service calls. Service monitoring watches interactions between microservices, while end-to-end monitoring tests externally visible behavior the way a user sees it and can reveal availability problems that finer checks miss.
The four golden signals provide a shared vocabulary for user-facing health across all layers:
Because a single metric rarely tells the full story, Cloud Monitoring supports custom metrics that combine factors into one condition that reflects a real workflow. Teams should also watch the distribution of a metric, not just its average, because averages and medians can hide wide imbalances across users.
| Monitoring layer | What it measures | Example in Google Cloud |
|---|---|---|
| Infrastructure monitoring | Baseline health of machines and accelerators | CPU load, GPU or TPU utilization, memory, bytes written to disk from Compute Engine VMs |
| Application monitoring | Performance of code-defined workflows | Query execution time or multi-step inference pipeline latency captured with OpenTelemetry |
| Service monitoring | Interactions between distributed services | Error and latency between microservices observed with Cloud Service Mesh on and off Google Cloud |
| End-to-end monitoring | User-perceived ability to complete critical actions | Synthetic check that a checkout or login completes within defined thresholds |
Creating dashboards for real-time visibility brings related metrics, logs, and alerts onto one screen for investigation. As resources are created or applications are registered with App Hub, Google Cloud automatically creates managed dashboards that show curated health, logs, metric data, and open alerts for the application and each service and workload. When managed views are not enough to see trends, outliers, or cross-service detail, teams create a custom dashboard to control exactly what data appears and how it is displayed. Metrics Explorer supports this work by letting teams chart time-series data, compare current data to previous periods, show outliers and percentiles, display multiple metrics together, and save the chart to a custom dashboard.
A custom dashboard can display charts and tables of metric data, log data and error groups, charts for alerting policies, information about alerts, explanatory text, and events such as a reboot or crash that affect operation. Data flows into these views from system metrics generated by Google Cloud services, system and application metrics collected by the Ops Agent on Compute Engine VMs, user-defined metrics created through the Cloud Monitoring API or OpenTelemetry, external metrics, Prometheus metrics, and logs-based metrics that count events in Cloud Logging. For Compute Engine, installing the Ops Agent adds host metrics, system logs, third-party integrations for software such as PostgreSQL and MongoDB, and Prometheus and OpenTelemetry Protocol (OTLP) receivers. GKE sends system logs and metrics by default and uses Google Cloud Managed Service for Prometheus for third-party and user-defined metrics.
Establishing alerting policies and proactive checks notifies responders when performance data meets defined criteria so triage can start before an SLO is violated. An alerting policy defines the metric condition to watch, such as CPU utilization of a VM exceeding 80%, and the list of people or groups to notify through channels including email, Cloud Mobile App, PagerDuty, or Slack. When the condition is met, the policy creates an incident, which is a persistent record that stores status, duration, labels, a chart of recorded metric data, and links to related logs for troubleshooting. Policies can also alert on SLO burn rate, which warns that errors are consuming the allowed error budget too fast, and many Google Cloud integrations offer recommended alerts where only the notification list must be added.
Proactive checks and health signals feed the same response loop by exposing which parts of the system are healthy. Health checks typically pull data from an endpoint to decide whether a component should keep receiving traffic, and load balancer health checks use that result to route around unhealthy virtual machines and to autoheal managed instance groups by recreating unhealthy machines. In GKE, an ingress resource that load-balances external traffic automatically creates appropriate load balancer health checks, and Kubernetes liveness and readiness probes let the orchestrator decide how to manage pods and requests when applications expose the right endpoints. Synthetic monitors extend coverage outward by probing HTTP, HTTPS, and TCP endpoints with uptime checks and by crawling pages with a broken-link checker, then triggering notifications when an endpoint fails to respond.
Exam tip: Prefer a percentile such as the 99th percentile over an average or median for latency SLOs, because an average can hide slow outliers that still violate user experience.
Distributed Troubleshooting and Application Insights is the practice of using correlated telemetry to find why a distributed application is slow or failing. Cloud Trace (opens in a new tab) follows requests across services, Error Reporting groups application errors, Cloud Monitoring visualizes performance metrics, and Cloud Logging preserves detailed event records. Together with Cloud Profiler for code-level performance, these services bridge structured logs and performance monitoring so operators can move from symptom to cause.
Observability services collect, analyze, and correlate log data, metric data, and trace data to maintain application reliability. They help teams achieve five related goals:
Cloud Trace for distributed request flows is a distributed tracing system that collects latency data from applications and displays it in near real time in the Google Cloud console. An instrumented application emits telemetry data, and Cloud Trace assembles that data into traces made of spans that show how a single request flows from service to service. When one incoming remote procedure call (RPC) fans out into multiple calls, Cloud Trace starts traces, generates spans, and propagates the tracing context so each downstream hop remains linked to the original request. Operators examine the result in the Trace explorer, which lets them view traces, spans, and metadata for latency analysis.
Cloud Trace depends on instrumentation, which is code added to an application to emit telemetry data. The recommended approach is to instrument with OpenTelemetry, an open-source and vendor-neutral framework, instead of vendor-specific APIs or client libraries. Microservices observability wraps OpenCensus plugins for metrics, traces, and gRPC logging into one unified plugin that retrieves project and deployment metadata with minimal configuration. That plugin also attaches resource labels, such as Compute Engine node name or Google Kubernetes Engine namespace, and custom labels, such as service version or canonical service name, to spans, metrics, and log entries for filtering.
Error Reporting for aggregated application errors is the service that aggregates and displays errors produced in running applications. The Error Reporting library automatically reports exceptions captured from an application Rack stack in frameworks such as Ruby on Rails and Sinatra, and developers can also call a manual report method to send a specific exception. Error Reporting groups repeated occurrences with the same signature into error groups, so teams see counts and patterns instead of thousands of separate log lines. Teams view and filter those groups on the Error Reporting page in the Google Cloud console.
Error Reporting works best when combined with logs and traces because traces locate where a failure occurred and log entries explain the details of that failure. Microservices observability automatically correlates logs with traces, using upstream sampling decisions or user-defined sampling rates to decide which traces are sampled. For RPC workloads, teams can also inspect transport-level events such as headers, trailers, messages, OK status, error status, finishing events, serialized message length, authority, client and server addresses, and whether the client or server canceled the RPC. Explicit settings can enable payload logging for messages or headers with size limits and method filters.
Log-based metrics that connect logs to monitoring turn text events in Cloud Logging into time series in Cloud Monitoring. Cloud Logging stores, searches, analyzes, monitors, and alerts on log data and events, while Cloud Monitoring collects metrics, events, and metadata and generates insights through dashboards, charts, and alerts. A log-based metric counts or measures matching structured log entries over time, so text events become a time series that flows from Cloud Logging to Cloud Monitoring for visualization and alerting. Teams can then create alerting policies that notify them when log data or metric data meets conditions they specify.
This bridge is central to service mesh and microservices telemetry, which emphasizes request volumes, latency, and error rates integrated with Cloud Monitoring. The observability plugin provides first-party measures at message level, RPC level, or method level, from message counts and message sizes to latencies, and uploads those measurements to Cloud Monitoring. Teams explore the collected metrics in Metrics Explorer, which includes predefined dashboards with pre-existing charts, or in custom dashboards. Cloud Logging also supports SQL-based analysis of log data and trace data, such as comparing URLs in log data against a dataset of known malicious URLs, plus suggested queries and synthetic monitoring with uptime checks.
Correlated telemetry for application insights links metrics, logs, and traces around the same workload so an operator can pivot from symptom to cause. Application Monitoring creates dashboards and topology maps for App Hub-registered applications, services, and workloads, displaying telemetry data, AI resource metric data such as error rates and token usage, and information such as open incidents or alerts. Monitoring, Logging, and Trace are enabled by default when a Google Cloud project is created, and they automatically collect telemetry data and audit logs for most Google Cloud services. Because the services share labels and trace context, an operator can start from a latency spike or error group and pivot to the matching spans and log entries.
The three troubleshooting signals differ by the question each answers:
| Signal | Primary data | Choose when |
|---|---|---|
| Cloud Trace | Latency data organized as traces and spans | The need is to follow a request across services and find the slow hop |
| Error Reporting | Aggregated error groups from exceptions | The need is to find which errors are frequent, new, or growing |
| Log-based metrics in Cloud Monitoring | Time series derived from structured logs plus RPC measures | The need is to track volumes, latency, and error rates and alert on trends |
Exam tip: Use Cloud Trace to explain slow requests across services, Error Reporting to triage grouped crashes, and log-based metrics to alert on patterns in structured logs.
A Service Level Indicator (SLI) is the specific technical metric that is measured, such as request success rate or inference latency, while a Service Level Objective (SLO) is the target value for that metric over time, such as 99.9% success. Teams first identify the business key performance indicator (KPI) affected by the system, then choose the technical metrics that drive that KPI.
Route log data to Cloud Storage for low-cost, long-term archival and compliance preservation, to BigQuery for large-scale SQL analysis and security investigation, and to Pub/Sub for real-time streaming to other applications or third-party tools. Sinks use a routing rule in Cloud Logging to send matching log entries to the chosen destination, while a copy can remain in a log bucket for short-term viewing.
Exclusion filters are rules that prevent selected log entries from being stored in a log bucket, and because the filters apply before storage the excluded entries do not consume bucket retention or incur bucket storage charges. This lowers ingestion and storage costs but removes detail, so exclusions should target noisy logs that are not needed for troubleshooting or compliance, such as Data Access audit logs in development projects which can be very large and create extra storage costs.
The four peer types are Admin Activity audit logs which appear as activity and record configuration and administrative changes, Data Access audit logs which appear as data_access and record reads and writes of user-provided data, System Event audit logs which appear as system_event and record Google Cloud system actions, and Policy Denied audit logs which appear as policy and record access denials based on policy checks. To retrieve them, open Logs Explorer and select your organization, choose the Google Cloud resource type, choose the log name for the type you need, and run the query to display results.
Your enterprise organization consists of hundreds of Google Cloud projects organized within a folder hierarchy. Your security compliance team requires all Cloud Audit Logs across the entire resource hierarchy to be aggregated into a dedicated central logging project for long-term security analysis and auditing.
To control ingestion and storage costs, your team also needs to optimize log volume by sampling high-volume VPC Flow Logs at 50% and excluding low-severity development logs, while guaranteeing that critical security audit logs cannot be missed.
What should you configure to meet these requirements?
Disable the _Default and _Required log sinks in all child projects, and deploy a Cloud Logging agent to stream logs directly to a central BigQuery dataset.
Configure an exclusion filter on the _Required bucket in the central project to drop Admin Activity audit logs and retain only VPC Flow Logs.
Deploy a project-level log sink in each child project routed to a Pub/Sub topic, and use a Cloud Function to filter out 50% of the VPC Flow Logs before writing to Cloud Logging buckets.
Create an aggregated log sink at the organization or folder level pointing to the central project, configure inclusion filters for audit logs, and apply exclusion filters with the sample function for high-volume logs.
Last updated