Log Exports Using Sinks and Aggregated Sinks

Log export architecture in Google Cloud is the controlled pipeline that moves security telemetry from Cloud Logging to a Cloud Storage (GCS) bucket and then to Google SecOps. Cloud Logging evaluates each entry against a sink inclusion filter and routes matches to the bucket, the sink writer identity writes objects into the bucket, and a Google SecOps feed reads those objects for detection. This separation of writer, storage, and reader creates the boundaries used for data residency, access control, audit completeness, and tamper resistance.

The export follows a fixed order. First, log entries that match the inclusion filter flow from Cloud Logging to the configured GCS destination in hourly batches. Then exported entries land as sharded JSON files organized by log type and date, where bucket location and protection settings apply. Finally, data flows from the GCS bucket to Google SecOps when the feed service account reads files from the bucket URI. New sinks that route to Cloud Storage buckets might take several hours to start routing, and the first entries can take 2 to 3 hours to appear, which creates a temporary monitoring gap for incident response and compliance timelines.

1.1Data Residency and Destination Control in Log Exports

Data residency and destination control in log exports is determined by the Cloud Storage bucket location, because object data is permanently stored in the selected location. The bucket configuration chooses a Location type of Region, Dual-region, or Multi-region and a specific Location such as us-central1. A regional choice keeps logs in one geography, while dual-region or multi-region choices spread durability across geographies. For regulated workloads, the bucket location must be selected closest to the compliance boundary or Google SecOps instance.

The destination also controls retention and sovereignty. Buckets support Standard storage for frequently accessed logs, Uniform access control for consistent bucket-level Identity and Access Management (IAM), object versioning or retention policy under protection tools, and a selectable encryption method. Public access prevention sits as a boundary control between the private log bucket and the internet, and it can be locked by organization policy so project owners cannot expose the bucket. Together, location plus retention, versioning, and encryption decide where logs legally reside, how long they remain available, and whether they can be altered after arrival.

1.2Access Control with IAM Separation for Writers and Readers

Access control with IAM separation for writers and readers uses two different service accounts on opposite sides of the bucket trust boundary. The sink writer identity, typically service-PROJECT_NUMBER@gcp-sa-logging.iam.gserviceaccount.com, must be granted permission on the destination bucket after the sink is created. The Google SecOps reader is a unique per-instance account, typically chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com, that must be granted permission to read the same bucket.

Least privilege requires different roles for each side. The Cloud Logging writer needs only Storage Object Creator (roles/storage.objectCreator) to create objects, while the Google SecOps reader needs only Storage Object Viewer (roles/storage.objectViewer) to list and read objects. Some procedures grant broader Storage Object Admin or roles/storage.admin, but broader write plus delete plus permission-management rights increase blast radius and should not be the default. Creation of sinks itself requires roles/logging.configWriter, so sink management, bucket writing, and log consumption remain three distinct ownership domains.

1.3Audit Coverage and Inclusion Filters for Compliance

Audit coverage and inclusion filters for compliance decide which entries flow from Cloud Logging to the bucket and onward to the SIEM. Only entries matching the sink inclusion filter are exported, so an overly narrow filter silently drops security-relevant telemetry while a comprehensive filter preserves activity, network, and threat evidence. Filters use resource.type and logName to select exact sources, so precision directly controls completeness for regulatory frameworks.

The documented patterns show how coverage maps to control domains. App Engine (opens in a new tab) request and application logs use resource.type="gae_app" with logName values for request_log, stdout, and stderr. Compute (opens in a new tab) and network telemetry use resource.type=("gce_instance" OR "gce_subnetwork" OR "gce_network" OR "gce_firewall_rule") with audit, VPC Flow, firewall, and serial console log names. Threat and identity telemetry use dedicated selectors such as Cloud Intrusion Detection System (IDS) log names, DNS Threat Detector (opens in a new tab) resource types, custom DNS (opens in a new tab) sources, and resource.type="audited_resource" with protoPayload.serviceName="cloudidentity.googleapis.com". Because exports are organized by log type and date with sharded files such as 08:00:00_08:59:59_S0.json, missing names or types create predictable blind spots. The feed also applies Maximum File Age, default 180 days, which bounds how far back ingestion will include modified files after a sink outage.

1.4Prevention of Log Exfiltration and Tampering with Least Privilege

Prevention of log exfiltration and tampering relies on write-only sink permissions, read-only consumer permissions, and bucket protection controls. The sink writer can create objects but cannot read or delete prior evidence when limited to Storage Object Creator, which prevents a compromised logging pipeline from altering history. The SecOps reader can ingest evidence but cannot modify the source bucket when limited to Storage Object Viewer, which prevents the consumer from becoming a tampering path. Cloud IDS (opens in a new tab) evidence exported under these roles follows the same boundary: the logging pipeline writes it and the detection platform only reads it.

Deletion behavior is the tradeoff between storage hygiene and evidence preservation. The Google SecOps feed offers three source deletion options that change both lifecycle and required privilege:

Choosing either delete option gives the reader delete rights over security evidence, so a compromised or misconfigured feed could destroy the authoritative GCS copy. The Never option is safest for retention and testing, especially with bucket retention policy, object versioning, Uniform access, public access prevention, and customer-controlled encryption. The bucket URI used by the feed, for example gs://appengine-logs-export/, must end with a trailing slash, and access should remain private so logs flow only from Cloud Logging to the designated bucket and then to the designated SecOps instance.

Design of log sink destinations and filtering strategies chooses where routed logs land and which log streams are exported. A log sink routes copies of log entries from Cloud Logging (opens in a new tab) to a destination, while an aggregated sink created at the organization or folder level routes matching logs from all child projects when --include-children is used. Centralized designs commonly use one aggregated sink for Administrator Activity audit logs into a central analytics bucket and a second sink for Data Access audit logs from sensitive projects into long-term storage.

2.1Selecting Export Destinations by Retention, Analysis, and Integration Needs

Selecting export destinations by retention, analysis, and integration needs matches each destination to how logs will be kept, queried, and shared. Cloud Storage holds logs as objects for low-cost, long-term retention and later re-import, BigQuery stores logs in partitioned tables for SQL analysis with per-table retention, Pub/Sub streams logs in real time to SIEM and partner platforms, and a Cloud Logging bucket in a central project provides restricted, query-ready storage for investigations outside workload-team control. Choose centralized sinks when the security team needs a consistent central record in a restricted environment, and avoid them when project owners fully own their audit logs or no central requirement exists.

Exporting outside Google Cloud after centralizing into an aggregate sink supports multi-cloud SIEM consolidation, but network egress plus external storage and per-format integration work make it expensive. Be selective about which log types leave Google Cloud, and retain the Google Cloud copy so Google Cloud Support retains access for troubleshooting, which is lost when logs are only sent through third-party collectors such as Fluentd.

2.2Configuring Aggregated Sinks and Routing Behavior

Configuring aggregated sinks and routing behavior creates an organization-level rule that collects logs centrally without removing them from source projects. Logs are stored twice, first in the parent project and again in the sink destination, so duplicate storage and cost continue unless child _Default sinks exclude the routed entries and future projects disable the _Default sink in organization default settings. The sink cannot write until its writerIdentity service account is granted write access on the destination bucket, dataset, or topic, so authorization is a dependency for log flow.

Create and activate an organization aggregated sink in order:

1. Create the sink: Run gcloud logging sinks create SINK_NAME DESTINATION --log-filter="LOG_FILTER" --organization=ORGANIZATION_ID --include-children, using a Logging bucket URI for analytics or a BigQuery dataset URI with --use-partitioned-tables for daily partitioning.
2. Verify creation: Run gcloud logging sinks list --organization=ORGANIZATION_ID to confirm the sink exists at the organization scope.
3. Retrieve the writer identity: Run gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID and copy the full writerIdentity value starting with serviceAccount:.
4. Grant destination write access: Grant that service account write permission on the destination, then control duplicate retention with exclusion filters on child _Default sinks.

2.3Constructing Advanced Inclusion Filters in Logs Explorer

Constructing advanced inclusion filters in Logs Explorer isolates specific log streams by resource, log name, payload fields, and hierarchy level before export. Every exported element must be explicitly scoped with log_id() or logName, and filters are limited to 20,000 characters with rejection possible for deeply nested AND and OR combinations even under that limit. Test the query in Logs Explorer with Run Query and confirm the matched entries are exactly the intended export set before pasting it into the sink filter.

Compute and network (opens in a new tab) examples show how resource and payload fields combine to narrow a stream. resource.type="gce_instance" selects VM logs, while resource.type=("gce_instance" OR "gce_subnetwork" OR "gce_network" OR "gce_firewall_rule") broadens to related network resources, and logName selects audit, flow, firewall, or serial output such as logName:"cloudaudit.googleapis.com/activity", logName:"vpc_flows", logName:"compute.googleapis.com/firewall", and logName:"serialconsole.googleapis.com". Payload conditions such as jsonPayload.connection.dest_ip:*, jsonPayload.dest_instance.project_id:*, and jsonPayload.rule_details.action=("ALLOW" OR "BLOCK") further isolate traffic by addresses, instance identity, or firewall decision. Tuning Data Access audit logs reduces high-volume exports without losing coverage, for example by keeping log_id("cloudaudit.googleapis.com/data_access") while excluding routine Storage get and list, repetitive SQL select, and Spanner streaming, transaction, and commit operations. Hierarchy scoping uses full logName paths such as projects or folders paths for access_transparency to include or exclude one project or folder while an organization-wide OR of log_id() values covers everything else.

Configuring and managing aggregated sinks for hierarchical data collection centralizes logs from many projects into one destination. An aggregated sink in Cloud Logging (opens in a new tab) is a log sink created at an organization, folder, or billing account that collects matching entries from that resource and all of its child resources. A security team uses aggregated sinks to centralize audit logs into one destination for compliance, investigation, and restricted-access storage. The sink applies one inclusion filter, optional exclusion filters, and routes copies to a Logging bucket, BigQuery dataset, Cloud Storage bucket, Pub/Sub topic, or Google Cloud project.

3.1Aggregation Scope and the Include-Children Rule

Aggregation scope and the include-children rule define which hierarchy level owns the sink and which children contribute logs. An organization-level sink collects logs from all folders and projects under the organization, while a folder-level sink collects only from projects nested under that folder. Log flow from children to the parent sink depends on the --include-children flag in the gcloud logging sinks create command, so the flag must be included to collate logs from all child projects. The sink inclusion filter, often generated by the log scoping tool, then selects which hierarchical logs are routed, such as Admin Activity audit logs for investigations or Data Access audit logs for sensitive workloads.

3.2Copy Semantics and Duplicate Storage in Aggregated Routing

Copy semantics and duplicate storage in aggregated routing mean the sink copies entries instead of moving them. An entry that matches the aggregated sink is stored twice: once by the sinks in its parent project and again in the aggregated sink destination. To avoid duplicate storage, an administrator adds an exclusion filter to the _Default sink in every child project to exclude the centrally routed entries. To prevent ingestion into _Default sinks in future projects, the administrator disables the _Default sink in the default settings of the organization.

3.3Inclusion and Exclusion Filters That Govern Routing Decisions

Inclusion and exclusion filters that govern routing decisions decide whether an entry flows from the sink to its destination. Filters are written in the Logging query language, and a sink with no inclusion filter matches every entry. Exclusion filters are applied after entries are received by the Logging API, so they do not reduce entries.write API quota consumption or the number of entries.write calls.

• A log entry that does not match the sink inclusion filter is not routed to the sink destination.
• A log entry that matches the sink inclusion filter and matches at least one sink exclusion filter is not routed to the sink destination.
• A log entry that matches the sink inclusion filter and does not match any sink exclusion filter is routed to the sink destination.

These three rules determine precedence inside a single sink, with exclusion filters overriding the inclusion filter when both match.

3.4System-Created Sinks That Guarantee Local Routing

System-created sinks that guarantee local routing ensure every entry originating in a project, folder, billing account, or organization is routed by either _Required or _Default to a log bucket in the same resource. The _Required sink handles a fixed subset of audit logs and cannot be modified or deleted, while the _Default sink handles the remainder and can be modified, retargeted, or disabled. An entry routed from project A to project B by another sink is not reprocessed by the _Required sink in project B, because _Required matches only entries that originated in its own resource.

Because the two filters are complementary, every originating entry is captured locally unless an administrator changes _Default.

3.5Creating and Authorizing an Aggregated Sink

Creating and authorizing an aggregated sink establishes the routing rule and then enables log flow by granting write permission. The sink creates its own service account, identified by writerIdentity, and routing fails until that identity is granted write access to the log bucket, dataset, or other destination. Verification confirms the sink exists at the intended organization or folder before traffic is expected.

1. Create the sink: Run gcloud logging sinks create SINK_NAME DESTINATION --log-filter="LOG_FILTER" --organization=ORGANIZATION_ID --include-children, where the destination is a Logging bucket path or a BigQuery path and LOG_FILTER is the scoped filter.
2. Verify creation: Run gcloud logging sinks list --organization=ORGANIZATION_ID and confirm the new sink appears.
3. Retrieve the writer identity: Run gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID and copy the full writerIdentity string starting with serviceAccount:.
4. Grant destination write access: Grant the writer identity write access on the destination bucket or dataset, then confirm that routed entries arrive.

For BigQuery destinations, the creation command also includes --use-partitioned-tables so data is partitioned by day based on the entry timestamp field. Compute log routing (opens in a new tab) to BigQuery follows the same authorization dependency: no entries flow until the writer identity holds destination write permission.

3.6Central Destinations and Rerouting Through a Project

Central destinations and rerouting through a project let a central project own storage while child projects own workloads. A common pattern routes Administrator Activity audit logs from all projects to an Observability Analytics bucket in a central project for investigation, then routes Data Access audit logs from sensitive-workload projects to Cloud Storage for long-term retention. When the destination is a Google Cloud project, the sinks in that destination project can reroute the received entries to any supported destination except another project, including local log buckets needed for Cloud Logging analysis and Error Reporting. Choose centralized aggregation when the security team needs a consistent, restricted-access record outside workload-team control, and avoid it when project owners retain full responsibility for their own audit logs with no central consistency requirement.

Key Facts

• Log export flows in order from Cloud Logging through a sink inclusion filter to a Cloud Storage bucket and then to Google SecOps, with new Cloud Storage sinks potentially delayed several hours before routing starts.
• The sink writer identity requires Storage Object Creator for write-only export and the Google SecOps reader requires Storage Object Viewer for read-only ingestion, while either delete-transferred-files option requires Storage Object Admin.
• An aggregated sink requires --include-children to collect child-project logs and copies entries instead of moving them, so duplicates remain unless child _Default sinks add exclusions.
• The _Required sink cannot be modified and captures activity, system_event, and access_transparency logs, while the _Default sink matches all other entries and can be modified, retargeted, or disabled.