Exporting and retaining logs (e.g., routing to BigQuery, Pub/Sub, Cloud Storage)

Evaluating and Selecting Destination-Specific Log Architectures

When designing a log management strategy, you must choose the right destination for your logs based on operational needs. You route logs using sinks to services like Cloud Storage, BigQuery, and Pub/Sub. Each destination serves a different primary purpose, and your choice balances cost, analysis needs, and speed of access.

Cloud Storage is best for cost-effective, long-term retention to meet compliance rules. You can set retention policies to keep logs for a specific number of days. However, logs stored here are not easily searchable for deep analysis. In contrast, BigQuery is a data warehouse designed for multi-dimensional analytical querying. You can run complex SQL queries on your log data to find trends and perform security investigations, making it ideal for historical reporting.

For situations requiring immediate action, Pub/Sub enables real-time log streaming. It acts as a message broker, sending logs directly to external Security Information and Event Management (SIEM) systems or custom applications for live monitoring. A key architectural decision is whether to use organization-level or folder-level aggregated sinks to collect logs from many projects into one central destination, which simplifies management.

A complete strategy often uses multiple destinations together. For example, you might send all logs to Cloud Storage for compliance, route important logs to BigQuery for your analysts, and stream security logs via Pub/Sub to your SIEM. You must also manage costs by estimating usage, setting budget alerts, and using filters to exclude unnecessary logs from expensive destinations.

Designing and Configuring Log Sinks and Routing Filters

Log sinks are the routers that direct log entries from Google Cloud resources to destinations like Cloud Storage, BigQuery, or Pub/Sub. Configuring these sinks correctly is essential for maintaining a complete audit trail and meeting compliance rules. You can create sinks at the project level or use aggregated sinks at the folder or organization level to centralize logs from many projects into a single destination.

To control which logs are sent and to avoid unnecessary costs, you use inclusion filters and exclusion filters. These filters use the Logging query language to evaluate each log entry. For example, you might exclude high-volume Data Access audit logs in development environments to save money, while ensuring critical security logs are always routed.

Cloud Logging automatically uses system log buckets. The _Required bucket stores critical Admin Activity and System Event logs for a fixed 400-day retention; you cannot disable or bypass routing for these logs. The _Default bucket holds other logs like Policy Denied and Data Access logs with a default 30-day retention that you can change. You can also create user-defined buckets with custom retention periods from 1 to 3650 days.

You can turn log data into operational insights using log-based metrics and alerting policies. Log-based metrics count matching log entries or extract numeric values to track trends. By pairing precise log queries with Cloud Monitoring, you can create alerts that notify your team instantly for critical events, helping distinguish between normal system noise and real problems.

Securing Log Pipelines and Implementing Compliance Retention Policies

Securing the log pipeline involves controlling who can see logs and ensuring logs are written securely to their destinations. You use IAM permissions and roles like Logs Viewer to control access. The service account used by a log sink, called its writer identity, must have permission to write to the destination, such as a Cloud Storage bucket or a BigQuery dataset.

For long-term archival, especially for compliance, you configure retention policies on your log destinations. In Cloud Storage, you can apply an Object Retention Lock to prevent logs from being deleted or modified before their retention period ends. In BigQuery, you can use partitioning strategies to manage data and automatically benefit from reduced long-term storage pricing for tables not edited for 90 days.

The system-managed _Required log bucket has a non-configurable 400-day retention for critical audit logs. The _Default bucket and any user-defined buckets have configurable retention. You can lock a user-defined log bucket to prevent its retention policy from being shortened or its data from being deleted, which is crucial for regulatory audits.

To optimize costs, you must monitor log storage usage, as you are charged for data kept beyond default retention periods. You should estimate costs using Cloud Logging pricing and set up log-based alerting policies to warn you when usage approaches a budget threshold. This ensures you maintain the logs you need for compliance without unexpected expenses.