Prepare and test your skills

Prepare and test your skills

Log routing and sink implementation in Cloud Logging (opens in a new tab) is the mechanism that directs each log entry to long-term storage or analysis. The Log Router (opens in a new tab) evaluates every incoming log entry against configured log sinks, and each sink that matches sends a copy to its destination. This design separates log generation from log storage, so logs from a project, folder, or organization can be centralized and retained in Cloud Storage, BigQuery, Pub/Sub, or Cloud Logging buckets.
The Log Router is the control point where log flow moves from source to destination. Traffic flows from the monitored resource to Cloud Logging ingestion, then from ingestion to each sink whose filters match, and finally from the sink to its configured destination. A sink depends on a filter that selects entries and a destination that receives them, plus Identity and Access Management (IAM) permission that allows writing to that destination.
A process flow showing log entries from a monitored resource flowing through Cloud Logging ingestion into the Log Router, where each sink applies an inclusion filter, up to 50 exclusion filters, and IAM permissions, then routes a copy to a Cloud Storage bucket, BigQuery dataset, Pub/Sub topic, or Cloud Logging log bucket.
Sinks can be created at the project, folder, organization, and billing account level to support centralized storage. For example, log entries from folders or organizations can be routed by a sink to a log bucket in a central project, which allows retention longer than the 30-day default for folder and organization _Default buckets. Because routing is a copy operation, the same log entry can be sent to multiple destinations by different sinks.
A log sink uses inclusion and exclusion filters to control exactly which log entries are routed. The inclusion filter defines which entries the sink will route, while exclusion filters remove unwanted entries before they are sent to the destination. Both the inclusion filter and each exclusion filter support up to 20,000 characters, and each sink supports up to 50 exclusion filters.
Filters make routing selective and cost efficient by sending only useful logs onward. A filter can capture all cache miss requests, capture all requests for a specific hostname, or keep only a sample of all logs. If more than 50 exclusions are needed, the supported approach is to refine the inclusion filter so fewer unwanted entries match in the first place.
Exam tip: A sink allows a maximum of 50 exclusion filters and this limit cannot be increased — refine the inclusion filter instead.
Routing destinations are the Google Cloud services that receive log copies from sinks for storage, analysis, or forwarding:
backfill_ prefix.The choice of destination controls cost and use. Routing to Cloud Storage, BigQuery, and Pub/Sub incurs charges in those services, and large volumes such as Data Access audit logs (opens in a new tab) can increase storage cost, so excluding non-useful audit logs helps manage bills. When routing to BigQuery is interrupted by a Cloud Logging alert, tables with a backfill_ prefix contain all entries that were to be routed during the alert window and might duplicate successfully routed rows, so merge the backfill data into the original table and then delete the backfill tables.
Log sink capacity is limited to 200 sinks per Google Cloud project, which can be increased to 4,000 on request. This quota also applies separately to billing accounts, folders, and organizations and is not hierarchical, so an organization with many projects can have up to 200 sinks at the organization level plus up to 200 sinks in each project.
Excluded logs can still appear in Logs Explorer when the query scope includes another copy of the same entry:
To fix this, verify the query is running in the correct project, check the filters of all sinks on the Log Router page, and refine the search scope to the source project or bucket. To confirm which identities can route to a bucket, check IAM for principals with the Logs Bucket Writer role and remove any service account that should no longer have write access.
Log Analytics in Cloud Logging (opens in a new tab) is the capability that lets you run SQL-based queries directly on log data stored in log buckets. Logs flow from Google Cloud projects into a log bucket through routing, and when that bucket is linked for advanced analytics, BigQuery can analyze the stored entries with SQL. The amount of history you can analyze depends on the bucket type, its retention period, and whether access controls or links are configured.
A log bucket upgraded for analytics is a Cloud Logging log bucket that has an active link to a linked BigQuery dataset. The link is what enables advanced analytics and SQL queries on the bucket, and the Logs Storage page shows the connection in the Linked dataset in BigQuery column. Because the workflow depends on the link, a bucket with an active link cannot be deleted until the link is deleted first.
A bucket with analytics enabled stays connected to its dataset until you remove that dependency. To list the links on a bucket, run the gcloud logging links list command or call the projects.locations.buckets.links.list API method. To remove the dependency, run the gcloud logging links delete command, which also deletes associated artifacts that can include the linked BigQuery dataset.
SQL-based analysis in BigQuery queries the log entries stored in the linked bucket to identify trends, investigate incidents, and audit past events. For example, logs that were previously exported to Cloud Storage (opens in a new tab) can be brought back into Cloud Logging by a Cloud Run job and then queried with SQL as an optional analysis step. When audit logs are imported this way, the import process modifies the log IDs, so queries on imported audit logs must account for the renamed log IDs.
Query results are only trustworthy when duplicate and delayed data are handled. Tables with a backfill_ prefix contain log entries that were meant to be routed to BigQuery during a Cloud Logging alert, and those tables can include entries that were already routed successfully. To prevent duplicate data in analysis, merge data from the backfill tables into the original tables and then delete the backfill tables.
Field-level access controls and log views control different levels of visibility for analysis. Log views hide the entire LogEntry, while field-level access controls hide individual fields of the LogEntry, such as an email address in the payload used to redact user information. You cannot use Observability Analytics on log buckets that have field-level access controls configured, so do not configure field-level access on a bucket that stores audit logs you plan to analyze.
Exam tip: Log views hide the whole LogEntry, while field-level access hides individual fields — and Observability Analytics does not work on a bucket with field-level access configured.
Retention and timing limits decide how much history SQL can see. Cloud Logging retains logs according to the bucket where they are held, and for project-level _Default and user-defined buckets you can configure retention between 1 day and 3650 days:
| Bucket | Location level | Default retention | Retention configurable |
|---|---|---|---|
| _Required | project, folder, organization | 400 days | No |
| _Default | folder, organization | 30 days | No |
| _Default | project | 30 days | Yes, 1 to 3650 days |
| User-defined | project | 30 days | Yes, 1 to 3650 days |
Imported entries must also fit within time bounds. Imported log entries with timestamps older than 30 days from the import time are not stored, and log entries written with ages older than a bucket's retention period are stored briefly and then purged. Those too-old entries do not add to retention costs but do add to storage costs, so do not write entries older than the bucket's retention period if you want to minimize storage costs.
Log bucket configuration and retention management in Cloud Logging (opens in a new tab) controls where logs are stored and how long they are kept. A log bucket (opens in a new tab) is a regional storage container that holds, indexes, and makes log entries searchable in a specific geographic location. Each bucket has a retention policy that sets how many days Cloud Logging keeps every log in that bucket, which helps meet latency, availability, and compliance needs.
Log buckets are the storage layer that gives operational visibility and supports data retention for compliance. Cloud Logging routes log entries into buckets through log sinks, and the bucket then applies one retention rule to all logs it holds, regardless of log type or whether the logs were copied from another location. Buckets can be customized by storage region, retention period, and encryption, including customer-managed encryption keys (CMEK) for organizations with advanced encryption requirements.
Organizations can also create user-defined buckets to centralize or subdivide storage. For example, filtering logs into a separate bucket allows a different retention policy without changing the policy for all other logs. If the retention policy of the _Default bucket is changed instead, that change affects every log stored in that bucket.
Log bucket types differ by where they can exist, how long they keep logs by default, and whether retention can be changed:
| Bucket | Location | Default retention | Retention configurable |
|---|---|---|---|
_Required | project, folder, organization | 400 days | No |
_Default | folder, organization | 30 days | No |
_Default | project | 30 days | Yes, 1 to 3650 days |
| User-defined | project | 30 days | Yes, 1 to 3650 days |
The _Required bucket holds audit logs that cannot be deactivated, and its 400-day retention period cannot be changed. The _Default bucket in a project and any user-defined bucket in a project start at 30 days but can be configured between 1 day and 3650 days. A bucket that lives in a folder or organization cannot have its retention period extended.
To keep folder-level or organization-level logs longer than 30 days, create a log sink that routes those entries to a log bucket in a project. The project bucket can then use a longer custom retention period, while the original folder or organization bucket keeps its fixed 30-day rule.
Exam tip: _Required is always 400 days and not configurable, while _Default in a project is 30 days by default but configurable from 1 to 3650 days.
A retention policy defines the number of days Cloud Logging stores logs in a log bucket before they are purged. Custom retention for _Default and user-defined buckets in projects can be set between 1 day and 3650 days through the Logs Storage page or with the gcloud logging buckets update command. Extending retention applies going forward and not retroactively, so logs cannot be recovered after their applicable retention period ends.
To update a retention period in the Google Cloud console, follow this order:
If a retention period is shortened, expired logs enter a 7-day grace period. During those 7 days the expired logs cannot be queried or viewed, but full access can be restored by extending the retention period again. Logs held during the grace period still count toward retention costs. Retention enforcement is eventually consistent, so log entries written with a timestamp older than the bucket retention may briefly appear before they are purged and will count toward storage costs but not retention costs.
Locking a log bucket prevents anyone from updating or deleting the bucket and also locks its retention policy. Locking is irreversible and must be done with the gcloud logging buckets update command using the --locked flag, because the console does not support locking. After a bucket is locked, the bucket cannot be deleted until every log entry in it has fulfilled the retention period.
A bucket can be deleted only when it is unlocked, or when it is locked and all entries have fulfilled retention. A bucket cannot be deleted while it has linked datasets, such as a linked BigQuery dataset, so links must be deleted first. Any log sinks that route entries to the bucket should also be deleted or modified, because Logging continues to route logs to the bucket during deletion.
After the delete command, the log bucket transitions to the DELETE_REQUESTED state and stays in that state for 7 days. During that time a new bucket cannot reuse the same name, and the pending bucket still counts toward the project bucket quota (opens in a new tab). A bucket in pending deletion can be restored before the 7 days end, which returns the bucket and its logs to active use.
The inclusion filter defines which log entries the sink will route while exclusion filters remove unwanted entries before they are sent to the destination. Both the inclusion filter and each exclusion filter support up to 20,000 characters and each sink supports up to 50 exclusion filters, so if more than 50 exclusions are needed you should refine the inclusion filter to match fewer unwanted entries in the first place.
The _Required bucket retains logs for 400 days and its retention cannot be changed. The _Default bucket in a folder or organization retains logs for 30 days and cannot be changed, while the _Default bucket in a project and user-defined buckets in a project default to 30 days and can be configured between 1 day and 3650 days.
Excluded logs can still appear when the query is running in the project that generated the entries rather than the project holding the routed copy, when another sink with a broader filter routed a copy to multiple log buckets, or when the user has access to views on the bucket where the entries were sent. To fix this, verify the query is running in the correct project, check the filters of all sinks on the Log Router page, and refine the search scope to the source project or bucket.
A bucket with an active link to a linked BigQuery dataset cannot be deleted until the link is deleted first. After deletion the bucket enters the DELETE_REQUESTED state for 7 days, during which its name cannot be reused and it still counts toward the project bucket quota, but it can be restored before the 7 days end to return the bucket and its logs to active use.
Your team runs multiple containerized applications on Google Cloud and needs to troubleshoot intermittent service errors. You want to aggregate log entries, calculate total storage volume by application, and identify traffic patterns using standard SQL queries directly inside Cloud Logging.
Currently, the application logs are stored in a standard user-defined Cloud Logging bucket where SQL queries cannot be executed.
What should you do to enable SQL-based analysis on your log data?
Last updated