Designing Cloud Logging Collection and Retention

Configure Log Sources and Collection is the practice of enabling the right Google Cloud logs and routing them through Log Router to the correct destinations for security and retention. Logs flow from the producing service or network resource to Log Router in the source project, folder, or organization. Log Router then evaluates exclusions and sink filters, and matching entries flow from each sink to its destination bucket, warehouse, queue, or security platform.

1.1Critical Log Sources to Enable

Critical log sources are the audit and network telemetry that show who changed what and what traffic was allowed or denied. Cloud Audit Logs (opens in a new tab) record control-plane and data-plane activity across projects, while network logs record packet-level and rule-level decisions in the Virtual Private Cloud (VPC). Enabling both families gives detection and incident response visibility into identity actions and lateral movement paths. Most of these sources depend on explicit enablement per subnet, rule, service, or organization policy.

Key sources to configure include:

• Administrator Activity audit logs for configuration changes and administrative operations, aggregated centrally for investigations.
• Data Access audit logs for reads and writes to user and service data, enabled selectively for sensitive workloads because volume and cost are high.
• VPC Flow Logs for subnet-level network traffic, including source and destination IPs, ports, and instance context.
• Firewall Rules Logging for connections evaluated by firewall rules, including the ALLOW or BLOCK action from rule details.
• Load balancer logging, Google Kubernetes Engine (GKE) Cloud Logging, and Cloud Storage bucket logging for edge, container, and object-storage activity required by compliance controls.
• Compute and serial console logs for virtual machine (VM) operations, such as resource.type="gce_instance" activity logs and serial console output.

The compliance baseline also expects database statement and connection logging, such as PostgreSQL and AlloyDB flags, and Google Workspace Login, Admin, and Group logs when Workspace is in scope.

1.2Design Ingestion with Log Router, Sinks, and Exclusions

Log Router (opens in a new tab) is the ingestion pipeline that controls which log entries are kept, dropped, and exported. Every entry first arrives at Log Router, then Log Router applies exclusions first to discard noisy entries and applies each sink inclusion filter to decide what flows from the sink to its destination. Each sink depends on a writer identity with permission on the destination, so the Cloud Logging service account must be granted destination access such as roles/storage.admin on a Cloud Storage bucket.

An aggregated sink is a sink created at the organization or folder level that collects matching logs from all current and future child projects into one central project. Use aggregated sinks when the security team needs a central record outside the control of the workload teams, and avoid them when each project owner fully owns its own audit logs. A practical pattern is one aggregated sink that routes Administrator Activity audit logs to a Cloud Logging bucket enabled for Observability Analytics for fast search, plus a second aggregated sink that routes Data Access audit logs from sensitive projects to Cloud Storage for long-term retention with retention policy locks.

A sink inclusion filter selects entries by resource type and log name. For Compute and network collection, the build order is fixed:

1. Create the sink: Go to Logging > Log Router and create a sink with a name such as Compute-Logs-Sink.
2. Choose the destination: Select Cloud Storage, BigQuery, Pub/Sub, a Logging bucket, or a Google Security Operations (Google SecOps) feed, and enter the destination URI such as gs://compute-logs/compute-logs/.
3. Build the resource filter: Start with resource.type="gce_instance" for VM logs or add gce_subnetwork, gce_network, and gce_firewall_rule for broader network coverage.
4. Add log-type clauses: Add logName:"cloudaudit.googleapis.com/activity" for audit operations, logName:"vpc_flows" for flow logs, logName:"compute.googleapis.com/firewall" for firewall decisions, and logName:"serialconsole.googleapis.com" for serial output.
5. Grant the writer and validate: Grant the sink writer identity permission on the destination, then validate collection in a test project before applying the organization-level Data Access policy to production.

1.3Centralized Collection Patterns

Centralized collection patterns differ by where analysis happens and who controls access. Logs can stay in Google Cloud for platform teams, stream in real time to Google SecOps for security operations, or leave Google Cloud for a multi-cloud security information and event management (SIEM) platform. The patterns can combine when ownership is split, such as platform teams analyzing logs in the central Google Cloud project while the security operations center (SOC) works only in Google SecOps.

Export outside Google Cloud carries network egress, storage, and per-format integration cost, so export only required log types rather than every available log. Direct ingestion offers less granular control than Log Router sinks, so use custom feeds through feed management when log types beyond the supported export set are required.

1.4Control Storage, Retention, and Access

Log storage control determines where logs live, how long they are kept, and who can read sensitive fields. Cloud Logging buckets are regional resources, so the storage region must satisfy latency, availability, and compliance needs, and a default resource location can be set so new _Default and _Required buckets inherit the required region. Custom retention configures retention between 1 day and 3650 days, and that rule applies to every log in the bucket regardless of log type or copy source.

Protect and scope access after routing is configured. Default encryption at rest protects stored content, while customer-managed encryption keys (CMEK) let the organization control key lifecycle, including hardware security module (HSM) or external key management when compliance requires it. Log views restrict access to a subset of entire LogEntry objects in a bucket, while field-level access controls hide individual LogEntry fields such as email addresses in the payload for most users. Do not configure field-level access controls on a bucket used for Observability Analytics, because analytics cannot run on buckets with field-level controls.

Define Log Retention, Archival, and Access Controls is the discipline of deciding how long log data remains searchable, where it moves for low-cost storage, and who can read it. Cloud Logging (opens in a new tab) stores logs in regional log buckets with configurable retention, while log sinks route copies to central analysis or archival destinations. Identity and Access Management (IAM) plus bucket-level views and field controls enforce least privilege over that data.

2.1Define Data Retention Periods

Data retention periods are the retention rules on the log bucket where logs are held. Cloud Logging buckets are regional resources, so the infrastructure that stores, indexes, and searches logs lives in a specific geographic location chosen for latency, availability, or compliance requirements. An organization can configure a default resource location to automatically apply a storage region to new _Default and _Required buckets.

Custom retention can be configured between 1 day and 3650 days to meet compliance needs. A custom rule applies to all logs in a bucket regardless of log type or whether a log was copied from another location. For audit durability, best practice is to back up audit logs to a separate physical repository with its own retention schedule, integrity checks using checksums, monitoring, and tested backup and restore procedures.

2.2Archive Logs for Cost-Effective Long-Term Storage

Archival is the movement of log data from high-cost searchable buckets to lower-cost destinations through log sinks. A common pattern is to aggregate Administrator Activity audit logs from all projects into an Observability Analytics bucket in a central project for investigation, and to route Data Access audit logs from sensitive workloads into a Cloud Storage (opens in a new tab) bucket for long-term retention. Centralized sinks are appropriate when the security team needs a central record outside the control of the workload teams that produced the logs, and are unnecessary when project owners retain full responsibility for their own logs.

Cost-effective lifecycle management depends on being selective about what is retained and where:

• Central Analytics bucket receives aggregated admin activity logs from all source projects for centralized investigation with restricted access.
• Cloud Storage bucket receives data access logs from sensitive projects for long-term retention with retention policies and retention policy locks.
• Google Security Operations (Google SecOps) receives supported Cloud Logging, asset inventory, and Security Command Center findings in real time through direct ingestion for multi-cloud SIEM analysis.
• External storage outside Google Cloud receives contents of an aggregate sink for single-pane analysis across cloud providers, with network egress and external storage costs.

Data Access audit logs can be large and incur additional storage cost, and exporting every log outside Google Cloud can be very expensive due to egress plus external capacity. Manage cost by estimating bills with usage data and alerting policies, and by excluding audit logs that are not useful rather than ingesting everything without a clear use goal.

2.3Enforce Granular Access Controls with Least Privilege

Granular access controls restrict log data with IAM allow policies granted only to privileged users, with further granularity from conditions, boundaries, and log-specific controls. IAM policies with conditions manage granular access to data and configurations, while principal access boundary (PAB) policies limit what a principal can access and Privileged Access Manager provides just-in-time temporary elevation whose use is itself audited. Storage-level controls extend this model, including IAM on Cloud Storage buckets and managed folders and IAM plus row-level and column-level security in BigQuery.

Log views and field-level access controls provide two different levels of redaction inside Cloud Logging buckets:

Field-level controls offer more precise privacy than views, but they prevent use of Observability Analytics on that bucket. Centralized sinks also support least privilege by storing logs in an environment with restricted access outside the control of the teams that produced them.

2.4Protect Log Storage with Location and Encryption Controls

Protection of retained logs combines regional placement with encryption at rest managed to the organization's compliance level. Google Cloud automatically encrypts content stored at rest, which is sufficient when there is no requirement to manage keys. When auditors or regulations require self-managed keys, Cloud Key Management Service (Cloud KMS) (opens in a new tab) provides customer-managed encryption keys (CMEK) for log buckets, with optional hardware security modules (HSM) or external key management through Cloud External Key Manager (Cloud EKM).

The CMEK model shifts key lifecycle responsibility to the security team, which can enforce compliance by blocking creation of non-compliant resources with CMEK organization policies. Customer-supplied encryption keys (CSEK) are not recommended because Cloud EKM supports more services with higher availability. For field-level privacy before storage, an ingestion pipeline can tokenize sensitive data with Sensitive Data Protection before persisting the tokenized form to destinations such as BigQuery.

Design Log Analysis and Threat Detection is the practice of collecting audit and network logs centrally, routing them to an analytics platform, and correlating them to find threats. Cloud Logging (opens in a new tab) collects logs from projects, folders, and billing accounts across the organization, while Security Command Center (opens in a new tab) and Google Security Operations (Google SecOps) analyze those logs for threats, vulnerabilities, and misconfigurations. An effective design assumes an attacker is already inside the environment, so it establishes baselines for normal behavior and continuously looks for anomalies.

3.1Security Log Analytics Workflow

The security log analytics workflow is an ordered pipeline that moves data from source services to detection decisions. The workflow depends on Cloud Logging sinks to transport logs and on the chosen analytics tool to query them. Only after logs are enabled and routed can analysis correlate events from Cloud Logging, asset changes from Cloud Asset Inventory, and findings from Security Command Center into one view.

1. Enable logs: Prioritize and enable the security logs needed for analysis, since some logs are on by default and others incur ingestion costs.
2. Route logs: Aggregate organization-wide logs with an aggregated sink and send them to a single destination tied to the analytics tool.
3. Analyze logs: Query the routed logs to audit usage, detect threats, and correlate findings across sources.

3.2Enable and Structure Logs for Efficient Querying

Enabling and structuring logs means selecting the log types that answer specific security questions and formatting them so queries run efficiently. Cloud Audit Logs provide the foundation: Admin Activity audit logs are enabled by default, while Data Access audit logs must be manually enabled because they increase ingestion volume. Every audit entry is a structured LogEntry with a protoPayload containing an AuditLog object, including user identity in AuthenticationInfo, event type in methodName, timestamp, success or failure in response, request context in request and requestMetadata, and service details in serviceData.

Network and access logs add visibility beyond control-plane actions. Firewall Rules Logging records TCP and UDP connections at the individual firewall rule level inside a VPC network, which supports auditing and early warning of unapproved use. VPC Flow Logs record network flows sent or received by virtual machine instances, and each subnet enables Flow Logs independently so volume can be limited to in-scope subnets. Access Transparency and Access Approval close the provider-access gap by producing near real-time logs when Google administrators access customer content. Adding labels to workloads such as Google Kubernetes Engine (GKE) pods lets those labels percolate into metrics and logs, so queries can separate environments and differentiate workloads without parsing free text.

3.3Route Logs to Centralized Destinations

Routing logs is the process of copying organization-wide logs through a Cloud Logging aggregated sink to the destination that matches the analytics tool. The aggregated sink applies a log filter developed during scoping, often with help from a log scoping tool that maps each log to MITRE ATT&CK tactics, CIS benchmark checks, and Event Threat Detection dependencies. The sink needs write access to its destination, and analysts need separate read access to query the stored logs. Verification confirms that logs from all contained folders, projects, and billing accounts arrive in the destination.

Traffic flows from Cloud Logging to the sink destination, then from the destination to the analytics engine. Logs routed to Pub/Sub can be ingested directly by Google SecOps or streamed to Splunk through Dataflow, while logs in storage can be locked immutable for seconds to years to preserve the audit trail. Retention policies balance security needs for investigation history against storage and ingestion costs.

3.4Detect Anomalies with Monitoring and Security Command Center

Threat detection combines baseline monitoring, intrusion signals, and managed detectors that scan logs in near real time. Cloud Monitoring creates alerts from logs and metrics when behavior deviates from baseline, such as unusual protocols, unexpected connections, sudden spikes in data transfer, intrusion detection system (IDS) detections, or Firewall Rules Logging violations. Network Analyzer provides insight into network performance and security issues, while Cloud Intrusion Detection System (Cloud IDS) and the Cloud Next-Generation Firewall (NGFW) intrusion prevention service monitor traffic for malicious activity and automatically block or generate alerts for suspicious events.

Security Command Center automatically detects threats, vulnerabilities, and misconfigurations after activation and correlates its findings with ingested logs for broader context:

• Security Health Analytics detects common vulnerabilities and misconfigurations across Google Cloud resources.
• Event Threat Detection applies detection logic and threat intelligence against logs to identify threats in near real time.
• Virtual Machine Threat Detection detects potentially malicious applications running on virtual machines.
• Container Threat Detection detects common container runtime attacks.
• Web Security Scanner identifies OWASP Top Ten issues in web applications on Compute Engine, App Engine, and GKE.
• Attack path exposure simulates how an attacker could reach high-value resources from detected weaknesses.

Because Event Threat Detection depends on specific logs, the logs it relies on must be enabled and routed or its rules will have no data to evaluate.

3.5Correlate Events and Hunt Threats in SIEM Platforms

SIEM correlation joins logs, asset changes, and Security Command Center findings into timelines that support advanced threat hunting. Google Security Operations ingests Google Cloud data to correlate security events from various sources, provide real-time analysis of alerts, and facilitate incident response. Analysts hunt in Google SecOps with YARA-L rules, while analysts using Observability Analytics or BigQuery hunt with SQL queries from the open-source Community Security Analytics (CSA) project that provide a reusable baseline. Third-party SIEM platforms receive the same centralized stream through Pub/Sub integration.

Effective hunting starts from baselines for normal network behavior and then looks for suspicious patterns across identity, network, and workload signals. Continuous analysis of traffic flows exposes lateral movement, command-and-control traffic, port scans, cryptomining, brute-force SSH, and data exfiltration attempts that single-event alerts might miss. Exported Security Command Center findings flow to triage through Pub/Sub or console roles, so detection, investigation, and response share the same correlated data.

Key Facts

• Log Router applies exclusions first to discard noisy entries, then applies each sink inclusion filter to decide what flows from the sink to its destination.
• An aggregated sink created at the organization or folder level collects matching logs from all current and future child projects into one central project.
• Cloud Logging buckets are regional resources with custom retention between 1 day and 3650 days that applies to every log in the bucket.
• Log views hide entire LogEntry objects while field-level access controls hide individual LogEntry fields, and Observability Analytics cannot query buckets with field-level controls.
• Admin Activity audit logs are enabled by default, while Data Access audit logs, VPC Flow Logs, and Firewall Rules Logging must be explicitly enabled.