Prepare and test your skills
Prepare and test your skills
Ops Agent (opens in a new tab) and OpenTelemetry (opens in a new tab) divide telemetry work on Compute Engine between automatic host collection and standardized application collection. The Ops Agent is the primary collector that runs on Compute Engine virtual machines (VMs), while OpenTelemetry provides vendor-neutral libraries and collectors for application logs, metrics, and traces. Together they let host data flow automatically to Cloud Logging and Cloud Monitoring while application data flows from instrumented code through a receiver to the same Google Cloud project.
The Ops Agent is the primary agent for collecting telemetry from Compute Engine instances in a single process. The Ops Agent uses Fluent Bit for logs, which supports high-throughput logging across multiple cores, and the OpenTelemetry Collector for metrics and traces. It installs with a single download and uses one YAML-based configuration with proxy support.
By default the Ops Agent automatically collects host metrics such as CPU, GPU, memory, and process metrics with no extra configuration. The Ops Agent also automatically collects system logs such as syslog from Linux VMs, specifically /var/log/syslog and /var/log/messages, and Windows Event Log from Windows VMs. VMs without the Ops Agent report only basic system metrics and logs, so installing the agent is what enables full troubleshooting, performance monitoring, and alerting.
Custom log collection in the Ops Agent is defined in config.yaml through logging receivers, processors, and pipelines that connect them to Cloud Logging. A logging receiver defines where log data comes from, a processor defines how entries are transformed or filtered, and a pipeline binds receivers and processors to an export destination. Telemetry flows from the log source to the receiver, then through each configured processor in order, then from the pipeline to Cloud Logging in the VM's project.
Receivers cover file and system sources beyond the defaults:
Processors standardize entries before export by parsing text into structured logs with JSON-based or regular-expression-based parsing, modifying entries by removing, renaming, or setting fields, excluding unwanted logs based on labels and regular expressions, and concatenating multiline language-exception logs from Java, Python, and Golang. This structure matters because an application should first output JSON-structured logs to a file, then the Ops Agent receiver tails that file and the processors normalize severity, message, and fields for Cloud Logging.
OpenTelemetry is the recommended open-source framework for generating vendor-neutral metric and trace data and for structuring log data for Cloud Logging. For logs, applications should use a framework that can output JSON-structured logs, including slog for Go, logging for Python, Pino for JavaScript, and SLF4J with Log4j2 for Java. For metrics, applications should use OpenTelemetry libraries or Prometheus client libraries that expose metrics at an HTTP endpoint for scraping. For traces, applications should use OpenTelemetry to create distributed traces where each trace represents one end-to-end operation and each span records a single function or operation.
The Google-Built OpenTelemetry Collector (opens in a new tab) is Google's open-source build of the upstream collector, built on Google infrastructure with a secure supply chain. The collector configuration file controls how telemetry is received, processed, and exported, and in general both receiving and exporting should use the OpenTelemetry Protocol (OTLP). The Google Cloud Telemetry OTLP API implements this protocol and supports ingesting OTLP logs, metrics, and traces, which allows existing collectors to migrate to OTLP exporters without rewriting application code.
Collection architecture determines whether application code stays vendor-neutral and where transformation happens. In collector-based export, the application uses the OpenTelemetry software development kit (SDK) with an in-process OTLP exporter to send data to a collector, and the collector forwards it to Google Cloud. In direct export, the application uses the OpenTelemetry SDK with Google-specific exporters to send data directly to the Telemetry API with no collector in between.
| Approach | Application dependency | Choose when |
|---|---|---|
| Collector-based export | In-process OTLP exporter to Ops Agent receiver or Google-Built OpenTelemetry Collector | The environment can run a collector and code must stay vendor-neutral |
| Direct export | Vendor-specific Telemetry API exporter in application process | Running a separate collector process is not practical |
Exam tip: Choose collector-based export on Compute Engine to keep application code vendor-neutral; the Ops Agent receives OTLP and sends metrics through the Cloud Monitoring API and traces through the Cloud Trace API.
The Compute Engine collection pattern combines file-based log collection with OTLP reception in the Ops Agent. Application logs flow from the application file to the Ops Agent logging receiver to Cloud Logging, while application metrics and traces flow from the OpenTelemetry SDK to the Ops Agent OTLP receiver to Cloud Monitoring and Cloud Trace. The Ops Agent depends on the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs being enabled and on the VM service account, by default the Compute Engine default service account, having permission to write to them.
To implement this pattern, first instrument the application (opens in a new tab) and configure JSON-structured output, then install and configure collection:
An alternative exists only for Prometheus-format metrics, where the Ops Agent Prometheus receiver scrapes metrics instrumented with Prometheus client libraries or the OpenTelemetry SDK. Zero-code instrumentation is another option that attaches a runtime agent and uses environment variables or system properties instead of code changes, such as for Java workloads. Use the full Ops Agent plus OpenTelemetry SDK approach when consistent logs, metrics, and traces with centralized processing and vendor-neutral code are required.
Cloud Logging (opens in a new tab) sinks route copies of log entries from projects, folders, or organizations to analysis and retention destinations. A sink defines an inclusion filter that selects which logs to export, an exclusion filter that drops noisy entries, and a destination such as a Logging bucket, BigQuery, Cloud Storage, or Pub/Sub. This design supports microservices and network telemetry, including audit logs, VPC Flow Logs, and Firewall Rules Logs, by centralizing them for security investigation, SQL analysis, and long-term compliance retention.
Centralized collection gives a security team one restricted-access record of audit logs outside the control of workload teams. Three enterprise patterns divide this work by where logs are analyzed and how much control is required.
| Option | Use when | Avoid when |
|---|---|---|
| Aggregated sink in Google Cloud | A central team needs all Admin Activity or Data Access audit logs in a Logging bucket or Cloud Storage bucket | Each project owner fully manages their own audit logs |
| Direct ingestion to Google Security Operations | Logs must be analyzed in Google Security Operations as a security information and event management (SIEM) system in real time | More granular control over ingestion is required than direct ingestion provides |
| Export outside Google Cloud | One central SIEM must hold logs from all clouds and environments | Existing systems lack budget, capacity, or a clear use goal for large cloud log volumes |
A common combination is an aggregated sink for platform teams who work in Google Cloud plus direct ingestion for security operations teams who work only in Google Security Operations. Exporting everything outside Google Cloud can be expensive from network egress and storage costs, so selective filters are recommended over exporting every available log.
An aggregated sink is a Cloud Logging sink created at the organization or folder level that routes matching logs from all child projects into one destination. The sink is created with gcloud logging sinks create SINK_NAME DESTINATION --log-filter="LOG_FILTER" --organization=ORGANIZATION_ID --include-children, where the --include-children flag is what pulls in logs from every child project. For BigQuery, the command also adds --use-partitioned-tables so data is partitioned by day on the log entry timestamp field, which allows different partitioned tables to have different expirations such as three years for audit logs and 90 days for VPC Flow Logs.
Creating the sink follows an ordered identity workflow where routing fails until the sink identity receives destination permission:
gcloud logging sinks create command with the destination, log filter, and --include-children.gcloud logging sinks list --organization=ORGANIZATION_ID to confirm the aggregated sink exists.gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID to get the writerIdentity service account, which looks like serviceAccount:p1234567890-12345@logging-o1234567890.iam.gserviceaccount.com.For Compute and network telemetry, the inclusion filter selects resource types and log names such as resource.type="gce_instance" for audit and serial console logs, resource.type="gce_subnetwork" with logName:"vpc_flows" for VPC Flow Logs, and logName:"compute.googleapis.com/firewall" for firewall logs. Filters can be narrowed further with jsonPayload fields such as jsonPayload.connection.dest_ip, jsonPayload.connection.src_ip, and jsonPayload.rule_details.action=("ALLOW" OR "BLOCK").
Exclusion filters prevent duplicate storage because routing logs to a new destination does not move them. Cloud Logging stores logs twice: once in the parent project and again in the sink destination. To avoid duplicate storage costs, operators add an exclusion filter to the _Default sink in every child project to drop logs that were already centralized. To stop logs from being ingested into _Default sinks of future projects, operators disable the _Default sink in the organization default settings.
Exclusion filters are also used for cost control on large log types such as Data Access audit logs. Data Access audit logs help Google Support troubleshoot issues and should be enabled when possible, but they can be large and increase storage cost. A practical pattern is to validate the organization-level audit policy in a test project before applying it to developer and production projects, then exclude audit logs that have no operational use.
Exam tip: An aggregated sink copies logs to the new destination and leaves the originals in place; add an exclusion filter to child _Default sinks when duplicate storage is not wanted.
Destination targets separate real-time investigation, SQL analysis, low-cost retention, and streaming integration. The sink destination determines which tools can query the data and how long it can be kept for compliance.
The Cloud Logging service account used by the sink needs write permission on the destination, such as roles/storage.admin on the Cloud Storage bucket. For Google Security Operations ingestion from Cloud Storage, the Google Security Operations service account needs Read or Read and Write on the bucket, and the bucket URI must end with a trailing slash such as gs://compute-logs/compute-logs/.
Reimporting returns previously exported logs from Cloud Storage back into Cloud Logging for incident investigation or audits of past events, such as reviewing database connections from an earlier quarter. The exported objects must remain in the expected Cloud Storage export organization so the importer can locate entries by log ID and time range. A Cloud Run job performs the import work and Cloud Logging ingests the results under normal routing and storage rules.
The reimport workflow moves data from stored objects back through the logging API:
LogEntry structures and aggregates multiple entries into batches to reduce API quota consumption, with retry handling for quota errors.Imported entries are subject to Cloud Logging quotas and a 30-day retention period, so entries with timestamps older than 30 days from import time are not stored. Imported logs are not included in log-based metrics because their timestamps are in the past, unless a label records the import time and brings them into metric data. Audit log IDs may be renamed during import, so queries in BigQuery or Logs Explorer must account for the new names.
Log storage controls meet latency, availability, residency, and encryption requirements for enterprise workloads. Logging buckets are regional resources, so the infrastructure that stores, indexes, and searches logs sits in a specific geographic location. A default resource location can automatically apply a storage region to new _Default and _Required buckets, and Pub/Sub message storage policies can similarly restrict where streamed log data is stored.
Custom retention on a log bucket retains all logs in that bucket from 1 day to 3650 days, regardless of log type or whether the log was copied from another location. By default, Cloud Logging encrypts content at rest, while customer-managed encryption keys (CMEK) through Cloud Key Management Service (Cloud KMS) give control over key lifecycle, hardware security module protection, or external key management. Access can be limited with log views that hide entire LogEntry objects or field-level access controls that hide individual fields such as email addresses in the payload, but Observability Analytics cannot be used on buckets with field-level access controls configured.
Hybrid telemetry collection combines Google Cloud telemetry with on-premises collection layers to keep familiar tools while preserving Google Cloud troubleshooting access. Prometheus (opens in a new tab) serves as the on-premises metrics collection layer and Grafana serves as a single pane of glass for both Google Cloud and on-premises metrics, while logging is configured separately with Fluentd, Fluent Bit, or the Cloud Logging plugin for Grafana. Fluentd and Fluent Bit read many inputs including system logs and support many outputs including BigQuery, Elastic, and Splunk, but they handle logs only, so monitoring must be configured separately.
Exporting from Cloud Logging to partner services uses the Cloud Monitoring API for metrics and Pub/Sub log sinks for high-volume real-time log ingestion into Datadog, Elastic, and Splunk. This pattern lets teams keep existing tools and keeps logs available to Google Cloud Support, but externally hosted dashboards are not directly available to support and partner solutions may add licensing fees or fail to collect during network disruptions. When regulation requires customer data to stay on premises, a useful pattern separates sensitive application data from lower-risk operations data and routes telemetry with the OpenTelemetry Collector or BindPlane, keeping sensitive data local while streaming operations data to the cloud.
Cloud Audit Logs (opens in a new tab) and VPC Flow Logs combine control-plane audit history with network flow evidence. Cloud Audit Logs records who did what to which Google Cloud resource and when, while flow and firewall logs show how traffic moved between sources and destinations inside and across Virtual Private Cloud boundaries. Together the two sources let operators track administrative actions, audit data access patterns, diagnose latency and connectivity failures, and monitor security boundaries.
Cloud Audit Logs is the platform audit system that writes immutable entries to Cloud Logging for Google Cloud API activity. Entries are organized into four log types that differ by what triggered them and whether they are enabled by default:
| Log type | Records | Enablement and use |
|---|---|---|
Admin Activity (activity) | Control-plane writes such as creating a VM, changing IAM policy, or deploying a cluster | Always written; cannot be configured, excluded, or disabled, even if the Logging API is disabled |
Data Access (data_access) | Reads of user data, metadata, and configuration, such as get, list, or BigQuery select | Disabled by default; enable selectively because volume and storage cost can be large |
System Event (system_event) | Google-initiated system actions outside direct user control | Available for system-generated maintenance and lifecycle events |
Policy Denied (policy) | Access attempts denied by policy checks | Available to audit denied access and policy enforcement |
Administrative actions therefore flow from the Google Cloud service to the Admin Activity log without configuration, while data access flows from the service to the Data Access log only after an administrator enables that category. For threat detection, authentication and authorization services warrant full data-access auditing to catch discovery, credential access, and privilege escalation, storage services warrant auditing of user-data access to catch collection and exfiltration, and infrastructure services warrant auditing of metadata access to catch configuration scanning.
Access to audit telemetry is split between application write access and human read and admin access. Applications and agents need the roles/logging.logWriter role on the service account attached to the node pool to send entries to Cloud Logging, while humans need roles/logging.viewer to view logs and logging.privateLogViewer to view Data Access logs. Administrative control depends on roles/logging.configWriter to create sinks that centralize logs and roles/logging.admin for full logging administration.
Collection depends on forwarders and buffers between the source and Cloud Logging. In Google Distributed Cloud, an audit-proxy DaemonSet runs on control-plane nodes to buffer and write Kubernetes API server Admin Activity entries to the configured Google Cloud project and location. If Cloud Audit Logs is explicitly disabled, entries fall back to disk under /var/log/apiserver/ with up to 1 GiB retained, and if export is blocked by a network outage, entries are cached in a 10 GB disk buffer with oldest entries dropped first when full. One project supports approximately 1000 service accounts for this path, and multiple clusters can share the same service account.
Exam tip: Admin Activity logs are always on and cannot be excluded, while Data Access logs are off by default and should be excluded in development projects to control cost.
Retention for Cloud Logging buckets is controlled per bucket, not per log entry, and applies equally to copied logs. Operators configure custom retention between 1 day and 3650 days to meet compliance needs, and Cloud Audit Logs entries in the default path are retained for 400 days. Encryption defaults to Google-managed keys at rest, and organizations with advanced requirements configure customer-managed encryption keys (CMEK) to control the key encryption keys that protect stored logs.
Cost control depends on understanding that Data Access volume drives storage and routing charges. Operators estimate bills from usage data, configure alerting on usage, and use exclusions to drop audit logs that are not useful, such as verbose Data Access logs in development projects. When logs are routed onward, BigQuery, Cloud Storage, and Pub/Sub pricing also apply to the destination.
The Logs Explorer is the console interface for retrieving audit entries by resource, log name, and query filters. A troubleshooting query flows from resource selection to log-type selection to query execution:
k8s_cluster for Kubernetes clusters.logName and protoPayload.serviceName, for example protoPayload.serviceName="anthosgke.googleapis.com".protoPayload for method, identity, and resource details.Monitoring flows from log entries to Cloud Monitoring through log-based alerting policies. Logging creates alert data for Monitoring whenever a matching event appears, so operators create policies that distinguish immediate-investigation events such as unauthorized VM creation or IAM privilege escalation from low-priority events. Routing flows from Cloud Logging through sinks to supported destinations, including Cloud Storage for long-term preservation, BigQuery and Observability Analytics buckets for analysis, Pub/Sub for third-party SIEM ingestion, and direct ingestion to Google SecOps for security analytics.
Network telemetry extends platform audit history with evidence of packet movement and policy decisions. VPC-related sources include VPC Flow Logs for connection records, firewall logs such as compute.googleapis.com/firewall, NAT flow logs such as compute.googleapis.com/nat_flows, and DNS query logs such as dns.googleapis.com/dns_queries. Traffic flows from subnets, firewall rules, NAT gateways, and resolvers to Cloud Logging, where sampling rate and metadata annotations determine how much detail is kept for latency diagnosis, connectivity tracing, and boundary monitoring.
Centralized ingestion aggregates both platform and network logs from across the organization into one destination. An aggregated sink collects entries from multiple projects into a single Cloud Logging bucket for Observability Analytics or into a single BigQuery dataset for SQL analysis, while the same sink model can forward to Pub/Sub or Cloud Storage before a third-party SIEM pulls the data. For Google SecOps, direct ingestion can automatically aggregate a predefined export filter that includes cloudaudit.googleapis.com/activity, system_event, data_access, policy, access_transparency, firewall and NAT flows, DNS queries, requests, syslog, authlog, and Windows event logs, with the filter narrowed to exclude noisy reads such as storage get and list or BigQuery select.
Collector-based export uses an in-process OpenTelemetry Protocol (OTLP) exporter to send data to a collector such as the Ops Agent receiver or the Google-Built OpenTelemetry Collector to keep application code vendor-neutral, while direct export uses Google-specific exporters to send data directly to the Telemetry API with no collector in between. Choose collector-based export when the environment can run a collector and code must stay vendor-neutral, and choose direct export when running a separate collector process is not practical.
An aggregated Cloud Logging sink created at the organization or folder level with the include-children flag routes matching logs from all child projects to one destination but leaves the originals in place. To avoid duplicate storage costs, add an exclusion filter to the Default sink in every child project to drop logs that were already centralized and disable the Default sink in the organization default settings for future projects.
Admin Activity logs record control-plane writes such as creating a virtual machine, changing an Identity and Access Management policy, or deploying a cluster and are always written and cannot be configured, excluded, or disabled. Data Access logs record reads of user data, metadata, and configuration such as get, list, or BigQuery select and are disabled by default and should be enabled selectively because volume and storage cost can be large.
Your DevOps team is investigating intermittent connectivity timeouts and potential egress policy violations between Compute Engine workloads in a production VPC network. You need to enable VPC Flow Logs across the entire network to analyze traffic patterns while strictly controlling Cloud Logging ingestion costs.
Your solution must meet the following requirements:
0.5) to balance visibility with log volume.src_instance,dst_instance).Which command should you execute to implement this configuration?
Execute gcloud network-management vpc-flow-logs-configs create prod-vpc-flow-config --location=global --network=projects/my-prod-proj/global/networks/prod-vpc --aggregation-interval=interval-30-sec --flow-sampling=0.5 --metadata=custom-metadata --metadata-fields=src_instance,dst_instance and explicitly enable Data Access audit logs for the Network Management API.
Execute gcloud network-management vpc-flow-logs-configs create prod-vpc-flow-config --location=global --network=projects/my-prod-proj/global/networks/prod-vpc --aggregation-interval=interval-30-sec --flow-sampling=50 --metadata=exclude-all-metadata --metadata-fields=src_instance,dst_instance
Execute gcloud network-management vpc-flow-logs-configs create prod-vpc-flow-config --location=global --network=projects/my-prod-proj/global/networks/prod-vpc --aggregation-interval=interval-30-sec --flow-sampling=0.5 --metadata=custom-metadata --metadata-fields=src_instance,dst_instance
Execute gcloud compute networks subnets update prod-subnet --enable-flow-logs --logging-aggregation-interval=interval-30-sec --logging-flow-sampling=0.5 --logging-metadata=include-all-metadata
Last updated