Configuring log exports (log sinks and aggregated sinks).

Configure and Manage Aggregated Sinks for Hierarchical Data Collection

Aggregated sinks let administrators collect and route log entries from an entire resource hierarchy—organization, folder, or project levels—to a single central destination. This design supports organization-wide auditability, compliance monitoring, and centralized threat detection. When you create a sink at the organization or folder level, its rules apply to all child resources beneath it, ensuring uniform log collection policies without needing to configure each project individually.

Log routing relies on two kinds of filters: an inclusion filter that decides which logs enter the sink, and optional exclusion filters that drop specific logs before they are routed. Google Cloud automatically creates two system sinks: the immutable _Required sink, which captures critical audit logs, and the customizable _Default sink for all other log entries. The pipeline evaluates logs in a fixed order: first, the inclusion filter checks whether a log entry matches; if it does, the system then applies any exclusion filters. Exclusion filters do not reduce write quota usage because they act after the log has already been ingested. Sinks forward matching entries to destinations such as Cloud Storage, BigQuery, Pub/Sub, or log buckets, regardless of the child project’s own sink settings.

Administrators can use an intercepting aggregated sink by choosing a central Google Cloud project as the destination. This prevents child resources from duplicating log ingestion because the intercepting sink captures the logs and stops further routing. To ensure the delivery works, the Cloud Logging service account must be granted the correct IAM permissions on the target resource. Centralized logs require proper governance, including storage region selection and retention periods from 1 day to 3,650 days. You can protect logs further with customer-managed encryption keys (CMEK). Log views and field-level access controls let you restrict or redact sensitive fields such as IP addresses or credentials, keeping operational telemetry safe while meeting compliance rules.

Analyze Security and Compliance Implications of Log Export Architecture

The configuration of log sinks and aggregated sinks directly affects data residency, access control, and the integrity of the audit trail. For example, sending logs to a bucket in a specific geographical region helps satisfy data sovereignty laws, while a misconfigured sink could expose sensitive logs to unauthorized users. Every sink has a writer identity—a service account that the Logging system uses to write logs to the destination. The principle of least privilege means this identity should only have the bare minimum permissions needed to write, such as roles/storage.objectCreator for Cloud Storage or roles/logging.bucketWriter for a Log Bucket. Crucially, the writer identity must never be granted read permissions, because that would create a path for log exfiltration or tampering. The design ensures logs are written in a secure, append-only manner.

From a compliance perspective, the log export architecture must provide complete audit coverage for frameworks like PCI DSS, HIPAA, or SOC 2. This means capturing all relevant audit logs—Admin Activity, Data Access, System Event—and routing them to an immutable destination. Aggregated sinks at the folder or organization level simplify this by centralizing logs from many projects into one governed location. However, you must verify that no critical logs are excluded by overly broad filters and that the destination’s retention settings match regulatory requirements. Managing the lifecycle of sinks matters too: custom sinks can be disabled, updated, or deleted, and any change to a sink’s filter or destination can create gaps in monitoring. When exporting logs to an external SIEM, such as in a Google Distributed Cloud air-gapped scenario, you must secure additional components like authentication tokens and network connectivity. The overall goal is to prevent log loss, enforce data residency, restrict access through precise IAM, and guarantee an immutable record for both security operations and compliance obligations.

Design Log Sink Destinations and Filtering Strategies

Log sinks consist of three elements: a sink name, a destination, and a filter that determines which logs are included or excluded. You can route logs to Cloud Storage buckets, BigQuery datasets, Pub/Sub topics, or other log buckets within your organization. Each destination serves a different purpose, and the filter defines exactly which log entries are forwarded.

Cloud Storage for Long-Term Retention

Cloud Storage buckets are a common destination for log exports when you need long-term retention, compliance, or forensic analysis. The bucket must be created with the correct location type, storage class, and access controls. The Cloud Logging service account needs IAM permissions such as roles/storage.admin to write logs. Organizations often choose Cloud Storage when they must preserve logs for years or meet regulatory requirements.

BigQuery for Analytics and Pub/Sub for Real-Time Processing

BigQuery is ideal for advanced SQL-based analysis. When logs are routed to BigQuery, they are organized into tables that support complex queries and aggregations, helping security teams spot patterns and threats. Pub/Sub destinations stream logs in real time to subscribers, enabling immediate action such as security orchestration or integration with third-party SIEM systems. BigQuery excels at historical analysis, while Pub/Sub supports real-time event processing.

Log Filtering Strategies

Effective filtering manages log volume, reduces costs, and ensures only relevant logs are exported. Filters use the Logging query language to match fields like log name, resource type, severity, and payload attributes. For example, a filter such as logName="*compute*" captures compute-related logs, while more complex filters can target specific network fields or security actions like jsonPayload.rule_details.action="ALLOW". You can also create exclusion filters to drop unwanted log entries, which helps control costs and focuses on high-priority security events.

Aggregated Sinks for Organization-Wide Collection

Aggregated sinks extend log routing across the entire organization, collecting logs from multiple projects, folders, or billing accounts into centralized destinations. This capability is essential for security operations teams that need comprehensive visibility. Aggregated sinks can be configured at the organization or folder level to route logs to a central Cloud Storage bucket, BigQuery dataset, or Pub/Sub topic. System-created sinks like _Required and _Default automate handling of audit logs: _Required routes critical audit logs to a required bucket, and _Default handles all other logs. Understanding how to configure and manage these sinks is vital for maintaining comprehensive security monitoring and meeting compliance requirements.