Prepare and test your skills
Prepare and test your skills
Managing the log lifecycle in Cloud Logging (opens in a new tab) means controlling how long log entries are kept, where they are stored, and how much volume is ingested so observability stays within budget. Log data flows from the originating project, folder, billing account, or organization resource into Log Buckets, and it can then be routed to other destinations for long-term analysis. Each stage — ingestion into Cloud Logging, retention in a bucket, and export to another service — carries its own cost driver.
Default storage in Cloud Logging requires no configuration because each Google Cloud project, folder, billing account, and organization resource automatically stores the log data that originates in that resource. For example, log data generated by a Compute Engine instance in a project is automatically stored in that project's buckets. Every stored log entry then follows a retention lifecycle where it is kept for a specified retention period and then deleted, so shortening retention directly reduces storage cost.
Operators can configure which log data are stored, which are discarded, and where log data are stored. Two primary controls reduce Cloud Logging storage cost: first, reduce volume by excluding or filtering unnecessary entries with exclusion filters, and then reduce the period for which entries are retained with custom retention on the bucket. Exclusion stops cost before ingestion and storage, while shorter retention stops ongoing accumulation of low-value history.
A Log Bucket is the Cloud Logging storage container that holds routed log entries and enforces retention, region, and access control with Identity and Access Management (IAM). Every project includes the _Required and _Default sinks that route entries to buckets, and sinks can also be inherited from parent folders or organizations. Because inherited sinks can forward imported or existing logs to unwanted destinations, a dedicated destination project with reviewed sink filters prevents extra charges and accidental export.
A log bucket can be upgraded to use Observability Analytics, formerly Log Analytics, which enables SQL querying directly on stored logs. There is no additional cost to upgrade the bucket or to use Observability Analytics, but after a bucket is upgraded it cannot be downgraded. Upgrading the _Default bucket with gcloud logging buckets update BUCKET_ID --location=LOCATION --enable-analytics keeps short- and medium-term troubleshooting in Cloud Logging without paying for a separate analytics export.
Tiered storage routes, or forwards, log entries from Cloud Logging to a destination that can sit in a different resource than the source, such as routing logs from one project to a bucket in another project. Routing separates hot operational storage in Log Buckets from colder or specialized analysis storage, which lets teams keep brief retention in Cloud Logging while preserving compliance history elsewhere. The choice of destination determines whether cost comes from storage, query computation, or streaming delivery.
Supported routing destinations include:
Exam tip: Keep short retention in a Log Bucket for troubleshooting and route only high-value logs to BigQuery, because BigQuery adds query-computation cost while Cloud Storage with lifecycle downgrade is cheaper for rarely accessed archives.
Log source categories determine how much volume enters the lifecycle and therefore how much filtering and retention tuning matters. Platform logs are written by Google Cloud services for debugging, Component logs are generated by Google-provided software running on customer systems such as GKE components, and Security logs answer who did what, where, and when through Cloud Audit Logs and Access Transparency logs. User-written logs come from custom applications via the Ops Agent, the Cloud Logging API, or client libraries, while Multicloud and Hybrid-cloud logs come from other clouds such as Microsoft Azure or on-premises infrastructure.
High-volume, low-value user-written and platform debug logs are the first candidates for exclusion filters, sampling, and shorter retention, while audit and security logs often require longer retention for compliance despite their cost. For exported data, apply automated time to live (TTL) lifecycle rules, consistent tags and labels for owner, purpose, and retention period, and Organization Policy constraints to enforce tagging so FinOps reports can track spend by dataset. After deployment, use Active Assist to review cost-optimization recommendations and use the Google Cloud Pricing Calculator plus Google Cloud Observability, Cloud Storage, BigQuery, and Pub/Sub pricing pages to model ingestion, storage, and analysis tradeoffs before changing retention or adding an export.
Log sampling and aggregation techniques reduce high-volume log streams to a smaller, cheaper set that still preserves operational trends. In Google Cloud, Cloud Logging aggregates entries through centralized sinks and batched writes, reduces volume through exemptions and exclusion filters, and preserves patterns through log-based metrics and selective long-term storage. The goal is to keep signals needed for troubleshooting and compliance while discarding redundant individual entries.
Aggregated sinks collect matching logs from many projects into one destination for centralized analysis and control. Log data flows from source projects to the sink and then to a central Cloud Logging bucket for impromptu investigation in Observability Analytics, or to a Cloud Storage bucket for long-term retention of sensitive Data Access audit logs. This central bucket sits outside the control of the workload teams that produced the logs, which creates a trust boundary with restricted access for the security team.
Choose centralized aggregation when the security team expects a central record of all audit logs or needs storage isolated from producers. Avoid centralized aggregation when there is no organization-wide requirement for consistent audit logs or when individual project owners retain full responsibility for their own audit logs. A selective alternative is direct ingestion of supported log types to Google Security Operations (opens in a new tab) for real-time SIEM analysis, or export of only required logs to an external SIEM where network egress and external storage costs make exporting every log prohibitively expensive.
Exemptions and exclusions control Data Access audit log volume at different points in the lifecycle, which matters most for services like Bigtable that can generate an extremely high number of DATA_READ and DATA_WRITE logs. Most Bigtable audit activity is generated by service accounts used by applications to call the Data API, so the first step is to identify noisy principals by searching IAM policies for the organization or by viewing principals on the IAM Permissions page. Only after identifying which service accounts are not useful for troubleshooting or auditing should volume controls be applied.
| Control | Where volume is stopped | Cost and operational effect |
|---|---|---|
| Exempt service accounts using audit configuration | Log generation is prevented by the IAM audit policy before entries are created | Recommended approach because neither ingestion nor storage charges are incurred |
| Exclude logs using exclusion filters | Log ingestion into log buckets is blocked after entries are generated | Entries are discarded from storage, but generation has already occurred |
Exempting principals prevents logs from being generated in the first place, while excluding logs discards them from ingestion into log buckets. Use exemptions for high-throughput service accounts that are never needed for investigation, and use exclusion filters for additional criteria such as log name, resource type, or connection attributes when generation-level control is too coarse.
Batch aggregation combines many individual LogEntry structures into fewer API or network operations without changing their content. The import path from Cloud Storage to Cloud Logging uses a Cloud Run job that processes previously exported objects in order:
LogEntry structures and aggregates multiple entries into batches to reduce API quota consumption, with retry handling for quota errors.Batching preserves every selected entry while lowering write-quota pressure and network overhead. The same tradeoff applies when forwarding through Pub/Sub and Dataflow to partners such as Datadog, where configuring the batchCount parameter to its maximum of 1,000 improves network egress efficiency and its associated cost savings.
Log-based metrics and selective storage preserve essential trends while redundant individual entries are discarded or aged out. Imported log entries with timestamps older than the 30-day retention period are not stored in Cloud Logging, and imported logs are not included in log-based metrics because their timestamps are in the past. If a label is used so the timestamp records the import time, the imported logs are included in metric data and can contribute to counts and aggregations.
Control storage cost by matching destination and retention to use:
_Default and _Required buckets.Route short-lived operational analysis to a Logging bucket queried by Observability Analytics or BigQuery, route compliance retention to Cloud Storage, and be selective about which audit, VPC flow, firewall, and serial console logs leave Google Cloud. Cloud Audit Logs (opens in a new tab) can be large, so estimate bills from usage data, configure alerting policies, and exclude logs that are not useful rather than retaining every source.
Exam tip: Exempting a service account prevents Data Access audit logs from being generated, while an exclusion filter discards them from ingestion after generation.
Log filtering and exclusion in Google Cloud is the practice of dropping verbose, low-value, or redundant log entries before they are stored and indexed. Cloud Logging (opens in a new tab) collects entries from Google Cloud resources, passes them through the Log Router, and then applies inclusion and exclusion filters in sinks to decide what is kept, where it goes, and what is discarded. Filtering at the source reduces ingestion volume, storage cost, and search noise while preserving operational insight.
Analyzing log streams means using ingestion metrics to find which log types and sources drive most volume and cost. Cloud Logging ingestion metrics show volume by log type, resource, and project, which reveals candidates such as debug logs, successful health checks, and repetitive request logs. Once candidates are found, filtering criteria are defined as specific fields, values, patterns, or regular expressions that reliably identify entries to drop without losing security or troubleshooting value. This analysis must also account for the ingestion method, because direct collection, agent collection with the Ops Agent, API writes, and routed exports each affect where a filter can be applied.
Filtering before storage creates a direct tradeoff: lower volume means lower cost and faster search and analysis, but an overly broad filter can remove entries needed for auditing or incident response. The decision rule is to exclude only entries with no diagnostic, security, or compliance value, and to verify the filter against historical logs before enforcing it.
Log Router is the component that evaluates every incoming log entry against sinks and forwards matching entries to destinations. A sink contains an inclusion filter that selects which entries to route and one or more exclusion filters that discard matching entries before they are stored or forwarded. When an entry matches an exclusion filter, the Log Router drops it so it consumes no bucket storage and incurs no downstream ingestion cost.
Every project, folder, billing account, and organization has default sinks, including _Required and _Default, and sinks can be inherited from parent folders or organizations. Before adding exclusions, review all sinks and their filters to avoid duplicate routing or accidental export to unwanted destinations, which causes extra charges. For organization-wide control, use aggregated sinks that route entries from resources in folders or organizations through one set of filters.
Routing is forwarding log entries that pass filters to a destination for storage or analysis. The Log Router can route entries to destinations in the same resource or a different project. Data flows from the log source to the Log Router, then through the sink filters to the configured destination.
The Log Router supports these routing destinations:
The choice of destination depends on access needs and cost: log buckets support retention and analytics, BigQuery supports large-scale queries, Cloud Storage supports low-cost retention, and Pub/Sub supports real-time integration. Source category also matters for filtering decisions, because different sources have different value and retention needs:
Exam tip: Exclusion filters discard matching entries so they are never stored, while sink inclusion filters select which surviving entries are routed to each destination.
Exclusion is not the only volume control. For high-request services, request logging can be sampled or filtered, so only a percentage of requests or only errors are recorded. Transform logic modifies entries instead of dropping them, such as stripping fully qualified domain names or deleting verbose fields within events. Redact logic masks or removes sensitive information before storage for compliance and privacy.
The order of operations matters for cost savings:
Upstream filtering provides maximum savings because the Log Router drops the data before bucket storage, downstream delivery through Dataflow or Pub/Sub, and indexing in the final analytics system.
Log storage lifecycle starts with automatic storage in the resource where the log originates, such as a Compute Engine instance storing its logs in its project. Administrators can then configure which entries are stored, which are discarded, and where they are stored. Entries remain in a log bucket for a specified retention period and are then deleted, and custom retention policies can be set per bucket or per sink.
To control cost, combine exclusion of unnecessary entries with a shorter retention period for noisy but short-lived operational logs and longer retention for audit and security logs. Access to log buckets is controlled by Identity and Access Management (IAM) permissions and roles, so only authorized principals can read, modify retention, or change sink filters. This boundary separates log producers, which write entries, from log administrators, who own buckets, sinks, and retention policy.
Exempting a service account in the IAM audit configuration prevents Data Access audit logs from being generated before entries are created, so neither ingestion nor storage charges are incurred. An exclusion filter discards already-generated entries from ingestion into log buckets, so the entries consume no bucket storage but generation has already occurred. Use exemptions for high-throughput service accounts that are never needed for investigation and exclusion filters for additional criteria such as log name, resource type, or connection attributes.
Keep brief retention in Log Buckets for short- and medium-term troubleshooting and operational analysis while preserving compliance history elsewhere. Route only high-value logs to a BigQuery dataset for long-term SQL analysis because cost depends on both stored data and query computation. Use a Cloud Storage bucket for low-cost archival of rarely accessed logs where Object Lifecycle Management can automatically downgrade objects to Nearline, Coldline, or Archive or delete them.
The Log Router evaluates every incoming log entry against sinks and forwards matching entries to destinations. An inclusion filter in a sink selects which entries to route to that destination, while one or more exclusion filters discard matching entries before they are stored or forwarded. When an entry matches an exclusion filter, the Log Router drops it so it consumes no bucket storage and incurs no downstream ingestion cost.
An enterprise web service running on a fleet of Compute Engine Linux VMs generates over 40,000 HTTP access log entries per second per instance during peak traffic. The operations team identifies two major issues:
google-fluentd) is reaching its single-core CPU limit and dropping log entries due to buffer overflow.You must design an observability architecture that achieves the following requirements:
Which solution should you implement?
Keep the legacy Logging agent and assign multiple worker processes on port 24231; apply an exclusion filter of sample(insertId, 0.10) for HTTP 5xx error logs; rely on Logs Analytics SQL queries for request metrics.
Configure the legacy Logging agent with a record_transformer filter to trim payload sizes below 256 KiB; create a Cloud Monitoring uptime check for traffic metrics; route all logs to a Cloud Storage cold archive bucket.
Upgrade instances to the Google Cloud Ops Agent; define log-based metrics on the incoming log stream to track overall request counts; configure an exclusion filter on the _Default log sink with sample(insertId, 0.90) for logs with HTTP status codes below 500.
Modify the logrotate configuration on each VM to use copytruncate with postrotate agent restarts; deploy log-based metrics on the _Default log bucket; route 100% of logs to BigQuery for analysis.
Last updated