Prepare and test your skills
Prepare and test your skills
Securing log pipelines and implementing compliance retention policies means controlling who can write and read logs while keeping copies for the required time at the lowest cost. Cloud Logging (opens in a new tab) routes log entries through sinks to a central logging bucket, Cloud Storage, BigQuery, or Pub/Sub, and each destination enforces its own retention and access controls. Retention in Cloud Logging buckets is configurable from 1 day to 3650 days, while exported copies use destination controls such as retention locks and partition expiration for long-term compliance.
A log sink is the routing rule that sends matching log entries from Cloud Logging to a supported destination, including an aggregated sink that collects logs from all projects in an organization or folder. Every sink has a writer identity, which is a Google-managed service account created for that sink, and log flow from the sink to the destination only succeeds after that identity is granted write permission on the destination. Read access to the destination is configured separately, so the team that analyzes logs does not need the same permissions as the pipeline that writes them.
Securing the pipeline requires strict separation of IAM roles for sink management, destination access, and analysis:
roles/resourcemanager.projectIamAdmin) to grant the Logs Writer role on the destination log bucketroles/storage.admin) to grant Storage Object Viewer on the storage bucket hosting exported logsroles/iam.serviceAccountCreator) to create supporting service accountsroles/serviceusage.serviceUsageAdmin) to enable required APIsroles/logging.admin) to upgrade a bucket for analysis and delete imported logsroles/run.developer) to create, run, and modify import jobs that move logsAccess to audit logs must be restricted to authorized personnel with unique accountability. Admin Activity audit logs are always written and cannot be configured, excluded, or disabled, even if the Cloud Logging API is disabled, so monitoring and alerting for administrator activity depends on protecting and routing this immutable stream.
Bucket-level retention for compliance is defined by the log bucket that holds the logs, not by individual log types. A custom retention rule applies to all logs in a bucket for between 1 day and 3650 days, regardless of log type or whether the log was copied from another location. This bucket-level scope means that logs with different compliance needs must be placed in different buckets if they require different retention periods.
A central bucket for organization-wide analysis is created in a fixed location of global, us, or eu, and the region cannot be changed after creation. The workflow moves from source projects to the central destination in order. First a single destination is created in the aggregation project, then an aggregated sink is defined at the organization or folder with a filter, then the sink writer identity is granted write permission, then analysts are granted only needed read roles, and finally routing is verified for new entries. Retention policies on log buckets should use Bucket Lock to prevent early deletion, which enforces immutable storage for the retention period.
Exam tip: Custom retention in Cloud Logging applies to the entire log bucket for 1 to 3650 days; it cannot be set per log type inside the same bucket.
Locked exported storage extends retention beyond Cloud Logging buckets and adds analysis options for compliance, real-time operations, and archival. Cloud Storage provides low-cost, long-term archival for audit logs, BigQuery provides powerful SQL search and analytics over large log volumes, and Pub/Sub forwards logs to other applications, repositories, and third parties including third-party SIEM systems. For organization-wide compliance, an aggregated sink routes copies from every project to a single Cloud Storage bucket or a single BigQuery dataset.
| Destination | Primary compliance role | Retention and cost control |
|---|---|---|
| Cloud Storage | Long-term immutable archival of audit logs | Bucket retention policy with Retention Lock prevents deletion until expiry |
| BigQuery | Centralized analysis with SQL over large audit datasets | Dataset partition expiration automatically drops old partitions to optimize storage cost |
| Pub/Sub | Real-time forwarding to SIEM or custom processors | Short message retention; long-term retention depends on the downstream system |
In BigQuery, the dataset is created with a default partition expiration that automatically expires old log partitions, which keeps compliance data for the required window without paying for indefinite storage. In Cloud Storage, a retention policy with lock provides the equivalent immutability guarantee for archived files. Because Data Access audit logs can be very large, excluding non-useful Data Access logs, for example in development projects, controls ingestion and storage cost without weakening Admin Activity coverage.
Encryption boundaries for stored logs separate customer content in the bucket or exported destination from anyone without access to the key. Protection at rest starts with Google-managed encryption, with customer-managed encryption keys (CMEK) available when an organization must control its own key encryption keys. With CMEK for logs storage, the organization manages key rotation and revocation in Cloud KMS instead of relying on Google-managed keys, which meets advanced encryption requirements for audit logs.
Cost and compliance trade off directly because comprehensive Data Access auditing increases storage and analysis spend. Bills are estimated by viewing usage data and configuring alerting policies, with attention to Data Access volume pushing projects over log allotments. Sinks that exclude noisy Data Access logs from Logging while still exporting required Admin Activity logs preserve accountability while optimizing cost.
Evaluating destination-specific log architectures is the process of matching an operational requirement — compliance retention, analytical querying, or real-time response — to the Google Cloud destination that serves it through a Cloud Logging sink. A Cloud Logging (opens in a new tab) sink routes copies of log entries to Cloud Storage, BigQuery, or Pub/Sub, while an aggregated sink collects logs from any or all projects in an organization into one central destination. The choice determines how long logs are kept, how they are queried, how quickly they can leave Google Cloud, and what ingestion and storage costs apply.
Centralized routing from source projects to an aggregation project moves log data through a sink filter to a single central destination. Data flows from source projects to the sink filter and then to a single central Logging bucket, BigQuery dataset, Pub/Sub topic, or Cloud Storage bucket in a designated aggregation project. If Observability Analytics or BigQuery is already centralized in a Logging bucket or BigQuery dataset, the existing aggregated sink is updated with the new filter and, for Observability Analytics, the bucket is upgraded to use Observability Analytics.
Routing logs for organization-wide analysis follows an ordered workflow:
Direct built-in ingestion to Google Security Operations covers only a predefined subset of logs, while an expanded set or a third-party security information and event management (SIEM) system requires an aggregated sink to Pub/Sub or Cloud Storage first.
Storage, analysis, and streaming destinations serve different access patterns, and the routing path depends on the analytics tool that consumes the logs. Cloud Storage preserves files cheaply for infrequent access, BigQuery enables SQL over large volumes, and Pub/Sub streams messages in real time to outside consumers. The sink writes directly to Cloud Storage or BigQuery, while Pub/Sub requires subscribers to pull messages into the SIEM or custom pipeline.
| Destination | Primary strength | Choose when |
|---|---|---|
| Cloud Storage | Long-term, cost-effective preservation | Compliance requires keeping audit logs for a longer period with infrequent access |
| BigQuery | Multi-dimensional analytical querying | Teams need powerful search and SQL analysis across large, organization-wide log volumes |
| Pub/Sub | Real-time streaming integration | Logs must flow to other applications, other repositories, third parties, or an external SIEM |
For third-party SIEM use, the sink must first aggregate the selected logs in Pub/Sub or Cloud Storage before the analytics tool pulls them. If the predefined exportable set meets security needs, Google Security Operations can instead ingest those logs directly without a sink.
Exam tip: Route directly to BigQuery for analysis in BigQuery, but route through Pub/Sub or Cloud Storage when the consumer is a third-party SIEM that must pull logs from Google Cloud.
Location, retention, and encryption ownership for centralized logs sits in the aggregation project that owns the destination. Log bucket retention in Cloud Logging is controlled by retention rules that apply to the bucket where logs are held. Custom retention can be configured between 1 day and 3650 days to meet compliance needs, and the rule applies to all logs in the bucket regardless of log type or whether the log was copied from another location.
New buckets are created with a fixed location of global, us, or eu, with global used when no location is specified, and the region cannot be changed after the bucket is created. A separate bucket is recommended instead of reusing the _Default bucket so retention, location, and Observability Analytics upgrades can be isolated to the compliance or analytics use case. Cloud Logging encrypts customer content at rest by default, and customer-managed encryption keys (CMEK) let the organization control the key encryption keys when default encryption does not meet requirements. This separation lets source projects generate logs while compliance and security teams govern how long the central copies persist and who can query them.
Audit log volume and cost drivers center on Data Access audit logs because they can be large and can push projects over log allotments. Pricing follows the destination service, with separate Cloud Storage pricing, BigQuery data ingestion pricing, and Pub/Sub pricing applying in addition to Google Cloud Observability pricing, so bills are estimated by viewing usage data and configuring alerting policies. Costs are managed by creating sinks that exclude audit logs that are not useful, such as excluding Data Access audit logs in development projects while retaining them for sensitive workloads.
Scoping which Data Access logs are enabled controls both security coverage and export volume. Authentication and authorization services warrant auditing all data access operations to monitor keys, secrets, and IAM policies for discovery, credential access, and privilege escalation. Storage services warrant auditing operations involving user-provided data to monitor sensitive data for collection and exfiltration, while infrastructure services warrant auditing metadata or configuration operations to monitor scanning of workload configuration for discovery. The selected logs then flow through the aggregated sink to Cloud Storage for cheap retention, to BigQuery for analysis, or to Pub/Sub for streaming, so the granularity decision directly determines destination size, query cost, and streaming throughput.
Designing and configuring log sinks and routing filters controls where copies of log entries go after they are received by Cloud Logging. The Log Router (opens in a new tab) evaluates every incoming entry against log sinks, and each log sink with a matching filter forwards a copy to its configured destination while the original continues to its log bucket. This design lets teams centralize audit logs for compliance, distribute operational logs for real-time analysis, and archive logs for long-term retention from one routing layer.
How the Log Router evaluates sinks and filters determines how one entry can reach many destinations at once. A sink defines a destination resource, such as a Logging bucket, BigQuery dataset, Cloud Storage bucket, or Pub/Sub topic, plus a filter that selects some or all logs for export. When an entry arrives, the router checks it against every sink in scope, so one entry can be stored locally, copied to long-term storage, and streamed to another system at the same time. Routing to Pub/Sub also enables forwarding to other applications, other repositories, and third parties outside Google Cloud.
| Destination | Central resource | Choose when |
|---|---|---|
| Cloud Logging bucket | Logging bucket in a central project | Central analysis is needed with Observability Analytics |
| BigQuery | BigQuery dataset in a central project | SQL-based analysis across the organization is needed |
| Cloud Storage | Cloud Storage bucket | Low-cost, long-term preservation of audit logs is needed |
| Pub/Sub | Pub/Sub topic | Streaming to a third-party SIEM, custom app, or expanded Google Security Operations export is needed |
For third-party SIEM use, the sink must first aggregate the selected logs in Pub/Sub or Cloud Storage before the analytics tool pulls them. If the predefined exportable set meets security needs, Google Security Operations can instead ingest those logs directly without a sink.
Choosing between project-level and aggregated sinks determines whether routing covers one project or an entire organization. A project-level log sink routes only the logs generated in the project where the sink is created. An aggregated sink is created at the organization or folder level and routes logs from any or all child Google Cloud projects to a single destination. Aggregated sinks are the mechanism for managing audit logs across an entire organization from one control point.
To centralize analysis, an aggregated sink forwards organization-wide entries into one Logging bucket for Observability Analytics or into one BigQuery dataset for cross-project queries. If that central sink already exists, a new export does not require a new sink; the existing sink is updated with the new log filter. Upgrading an existing central log bucket to use Observability Analytics preserves the centralized pipeline while enabling the newer analysis features.
Controlling traffic with inclusion and exclusion filters determines which log entries consume traffic, storage, and cost. The sink inclusion filter selects which entries the sink forwards to its destination, from broad selection of all logs to a precise match on log name, service, severity, or resource. Exclusion filters prevent matched entries from being stored or forwarded, which reduces volume when noisy logs push projects over allotments.
Precise filters commonly target audit log types by logName, such as activity for Admin Activity, data_access for Data Access, system_event for System Events, and policy for Policy Denied logs. Data Access audit logs are especially important for filter design because they can be very large and drive storage cost, so development projects often exclude them while production projects route them to BigQuery or Cloud Storage. Exclusion is a cost and volume control, while inclusion is a distribution control, and both can apply to the same log stream.
Exam tip: Admin Activity audit logs are always written and cannot be configured, excluded, or disabled, even if the Cloud Logging API is disabled — only Data Access audit logs and other selectable logs can be excluded to control cost.
Building the ordered centralized export pipeline aggregates logs from many projects into one destination through a fixed sequence of dependencies. The pipeline depends on the destination existing first, the sink having write permission to that destination, and readers having read permission to query the result.
gcloud logging buckets create or the central BigQuery dataset with bq mk in the aggregation project.Bucket location is fixed at creation and constrains where aggregated data physically resides. Supported Logging bucket locations are global, us, or eu, with global used when no location is specified, and the region cannot be changed after the bucket is created. Log retention is governed by the retention rule on the log bucket, with custom retention between 1 day and 3650 days applying to all logs in the bucket, while routing copies to Cloud Storage, BigQuery, or Pub/Sub extends preservation beyond that period. By default, Cloud Logging encrypts stored content, and organizations with stricter requirements can apply customer-managed encryption keys (CMEK) to control their own keys.
A Cloud Logging sink forwards a copy of each matching log entry to its configured destination while the original continues to its log bucket. Every sink has a writer identity, which is a Google-managed service account, and log flow only succeeds after that identity is granted write permission on the destination, with read access configured separately.
Custom retention in a Cloud Logging bucket can be configured from 1 day to 3650 days and applies to all logs in the bucket regardless of log type or whether the log was copied from another location. Logs that require different retention periods must be placed in different buckets, and Bucket Lock can be used to prevent early deletion.
Route to Cloud Storage for low-cost, long-term preservation of audit logs with infrequent access, to BigQuery for powerful SQL search and analytics over large log volumes, and to Pub/Sub for real-time streaming to other applications, other repositories, and third parties including third-party security information and event management (SIEM) systems. An aggregated sink at the organization or folder level can route logs from all child projects to one central destination.
No, Admin Activity audit logs are always written and cannot be configured, excluded, or disabled, even if the Cloud Logging API is disabled. To control ingestion and storage cost, exclude noisy Data Access audit logs that are not useful, for example in development projects, while retaining required Admin Activity logs.
An enterprise organization is establishing a centralized logging strategy across its Google Cloud resource hierarchy. The DevOps and security engineering teams have established three specific requirements for handling incoming audit logs:
Which log routing architecture should the DevOps engineer implement to satisfy all three requirements?
Last updated