Managing and analyzing logs

Analyzing logs using the Logs Explorer and the Logging query language

Cloud Logging collects log entries from nearly every GCP service, your own applications, and on-premises systems. The Logs Explorer is the primary interface for real-time and historical log analysis, letting you search, filter, and visualize logs without exporting them. You write queries using the Logging query language, which is a structured syntax that combines Boolean operators, field-based filters, and regular expressions. For example, you can filter by resource type (resource.type = "gce_instance"), severity (severity >= WARNING), or a specific text pattern. The query language also supports aggregation functions like count() and avg() over time windows, enabling you to build custom metrics from log data. Logs Explorer retains logs for the default retention period (30 days for most logs, adjustable), and queries run against the currently stored data. If you need to analyze logs older than the retention period, you must first export them to a long-term storage destination.

Exporting and retaining logs (e.g., routing to BigQuery, Pub/Sub, Cloud Storage)

Logs can be routed to external destinations through log sinks, which are configurable pipelines that copy matching log entries from Cloud Logging to a chosen destination. The decision of where to export depends on the use case:

• BigQuery is ideal for SQL-based analytics, ad hoc queries, and joining log data with other datasets. Logs are streamed into BigQuery tables (one per day) and can then be queried with standard SQL. This destination is best when you need to run complex reporting or correlate logs with application data.
• Cloud Storage is the cheapest option for long-term archival and compliance. You define a retention policy on the bucket (e.g., retain for 7 years) and can also apply lifecycle rules to transition logs to colder storage classes like Nearline or Archive. Logs are stored as newline-delimited JSON files, so you cannot query them in place; you must download or process them separately.
• Pub/Sub enables real-time log streaming for event-driven processing, such as feeding logs into a SIEM system, triggering Cloud Functions, or forwarding to an external analytics platform. Pub/Sub is the only destination that preserves near-real-time latency, but you must manage the subscriber and acknowledge messages to avoid backlogs.

You can also create multiple sinks with different filters, sending, for example, all audit logs to a BigQuery dataset for security analysis and all application logs to a Cloud Storage bucket for long-term storage. Logs are retained in Cloud Logging only if you set a custom retention period; otherwise they expire after the default period, so exporting is critical for compliance or extended analysis.

Handling sensitive data (e.g., using log processors to redact personally identifiable information [PII], protected health information [PHI])

Log entries often contain sensitive data like user emails, IP addresses, or medical record numbers. To comply with privacy regulations, you must redact or transform this data before it is stored or exported. Cloud Logging provides two main mechanisms for handling sensitive content:

• Log entry processors are a pipeline stage that runs after a log entry is received but before it is written to Cloud Logging or forwarded to a sink. You can create custom processors using the Log Processor API (currently in preview) that inspect fields, detect patterns (e.g., credit card numbers, email addresses), and replace them with tokens or hashes. The processor executes on the log entry in-memory, so the raw sensitive data is never persisted in the logging backend.
• Sinks with _Default and _Required buckets also support log exclusion filters; however, exclusion filters only drop entire log entries, not individual fields. For field-level redaction, you must use a processor or configure a separate export pipeline that passes logs through a Cloud Dataflow job or Cloud Function that redacts data before writing to the destination.

A common architecture is to route logs to a Pub/Sub topic, subscribe with a Cloud Function that runs a redaction library (e.g., DLP API), and then write the cleaned logs to BigQuery or Cloud Storage. This approach gives full control over the redaction logic but adds latency and cost. The tradeoff is between security compliance and operational overhead; using built-in log processors reduces complexity but may not cover every pattern.

Using Gemini Cloud Assist for AI-powered log analysis

Gemini Cloud Assist brings generative AI capabilities into Cloud Logging, allowing you to interact with your logs using natural language instead of writing complex queries. In the Logs Explorer, you can ask Gemini questions like "Show me all error logs from the last hour with stack traces" or "Summarize the most common error types in the last 24 hours", and it generates the corresponding Logging query language or a summary of findings. Gemini can also explain existing queries, suggest optimizations, and detect anomalies by comparing recent log patterns to historical baselines. It is important to note that Gemini operates on the same access controls and data governance policies as the rest of Cloud Logging; it does not bypass IAM permissions. You still need the logging.logEntry.list permission on the log scope. Gemini is particularly useful for incident response, reducing the time to identify root causes by quickly filtering and correlating log events. However, you must validate its output because the generated queries may need refinement for edge cases. Gemini Cloud Assist is an optional assistant, not a replacement for understanding the underlying query language and log structure.