An enterprise runs microservices on Google Kubernetes Engine (GKE) and utilizes Cloud Bigtable for real-time transactions. The environment generates 25 TB of application logs and a large volume of Data Access audit logs each month. The DevOps team must meet the following observability and budget requirements: Query and join application log data with existing BigQuery business datasets using standard SQL. Retain application logs for 90 days before deletion. Eliminate duplicate storage charges and unnecessary ingestion costs from high-volume Bigtable service accounts. Which strategy should the DevOps team implement to achieve these goals?

Exempt the application service accounts in the project's Data Access audit configuration, upgrade a user-defined log bucket to use Log Analytics with a 90-day retention period, and create a linked BigQuery dataset.: What the solution is This solution combines audit configuration exemptions, custom log bucket retention, and Log Analytics linked datasets to establish a cost-optimized, queryable log lifecycle. How it meets the requirements Eliminating unnecessary log ingestion and processing costs: Configuring service account exemptions under the Data Access audit policy prevents Bigtable DATA_READ and DATA_WRITE logs from being generated at the source, preventing unnecessary data transfer and ingestion fees. Retaining logs for 90 days: Configuring a 90-day custom retention period on the user-defined log bucket satisfies compliance and analytical requirements directly within Cloud Logging. Enabling SQL joins without double storage: Upgrading the bucket to Log Analytics (which has no upgrade fee) and provisioning a linked BigQuery dataset allows BigQuery Studio and SQL queries to read log data in-place. This completely avoids routing duplicate copies of log records into BigQuery native table storage. Key advantages and capabilities Zero data duplication: BigQuery queries query the underlying Cloud Logging storage directly via read-only metadata links. Full SQL capability: Developers and data analysts can run complex analytical SQL queries joining LogEntry records with relational tables in BigQuery. Source-level suppression: Audit exemptions prevent log generation altogether rather than dropping them downstream after resource consumption has already occurred. Centralized retention: Log retention is managed through a single policy on the log bucket rather than synchronizing multiple database partition expiration rules. Why this is optimal Alternative approaches that rely on export sinks duplicate log records into both Cloud Logging buckets and BigQuery storage tables, doubling ingestion and storage expenses. Leveraging Log Analytics linked datasets provides full BigQuery analytical capabilities on a single copy of log data.

StudyCerts

Professional Cloud DevOps Engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Optimize Log Retention Tiering Bigquery Linked Datasets

Section: Optimizing logs (e.g., filtering, sampling, exclusions, cost management, source considerations)

An enterprise runs microservices on Google Kubernetes Engine (GKE) and utilizes Cloud Bigtable for real-time transactions. The environment generates 25 TB of application logs and a large volume of Data Access audit logs each month.

The DevOps team must meet the following observability and budget requirements:

Query and join application log data with existing BigQuery business datasets using standard SQL.
Retain application logs for 90 days before deletion.
Eliminate duplicate storage charges and unnecessary ingestion costs from high-volume Bigtable service accounts.

Which strategy should the DevOps team implement to achieve these goals?

Exempt the application service accounts in the project's Data Access audit configuration, upgrade a user-defined log bucket to use Log Analytics with a 90-day retention period, and create a linked BigQuery dataset.

Create a Log Router sink to stream logs directly to a partitioned BigQuery table with a 90-day partition expiration, set an exclusion filter on the sink for service accounts, and keep the Cloud Logging bucket retention at 30 days.

Route all application logs to a Cloud Storage bucket with Autoclass enabled, and run a scheduled Cloud Run job every 24 hours to import the log batch into BigQuery.

Configure field-level access controls on the log bucket to mask PII, extend the default log bucket retention to 90 days, and configure a Log Analytics linked dataset in BigQuery.

Professional Cloud DevOps Engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Optimize Log Retention Tiering Bigquery Linked Datasets

Section: Optimizing logs (e.g., filtering, sampling, exclusions, cost management, source considerations)

The DevOps team must meet the following observability and budget requirements:

Query and join application log data with existing BigQuery business datasets using standard SQL.
Retain application logs for 90 days before deletion.
Eliminate duplicate storage charges and unnecessary ingestion costs from high-volume Bigtable service accounts.

Which strategy should the DevOps team implement to achieve these goals?

Route all application logs to a Cloud Storage bucket with Autoclass enabled, and run a scheduled Cloud Run job every 24 hours to import the log batch into BigQuery.

Configure field-level access controls on the log bucket to mask PII, extend the default log bucket retention to 90 days, and configure a Log Analytics linked dataset in BigQuery.

Explanations

What the solution is

This solution combines audit configuration exemptions, custom log bucket retention, and Log Analytics linked datasets to establish a cost-optimized, queryable log lifecycle.

How it meets the requirements

Eliminating unnecessary log ingestion and processing costs: Configuring service account exemptions under the Data Access audit policy prevents Bigtable DATA_READ and DATA_WRITE logs from being generated at the source, preventing unnecessary data transfer and ingestion fees.
Retaining logs for 90 days: Configuring a 90-day custom retention period on the user-defined log bucket satisfies compliance and analytical requirements directly within Cloud Logging.
Enabling SQL joins without double storage: Upgrading the bucket to Log Analytics (which has no upgrade fee) and provisioning a linked BigQuery dataset allows BigQuery Studio and SQL queries to read log data in-place. This completely avoids routing duplicate copies of log records into BigQuery native table storage.

Key advantages and capabilities

Zero data duplication: BigQuery queries query the underlying Cloud Logging storage directly via read-only metadata links.
Full SQL capability: Developers and data analysts can run complex analytical SQL queries joining LogEntry records with relational tables in BigQuery.
Source-level suppression: Audit exemptions prevent log generation altogether rather than dropping them downstream after resource consumption has already occurred.
Centralized retention: Log retention is managed through a single policy on the log bucket rather than synchronizing multiple database partition expiration rules.

Why this is optimal

Alternative approaches that rely on export sinks duplicate log records into both Cloud Logging buckets and BigQuery storage tables, doubling ingestion and storage expenses. Leveraging Log Analytics linked datasets provides full BigQuery analytical capabilities on a single copy of log data.

View source documentation

Professional Cloud DevOps Engineer

Optimize Log Retention Tiering Bigquery Linked Datasets

Professional Cloud DevOps Engineer

Optimize Log Retention Tiering Bigquery Linked Datasets

What the solution is

How it meets the requirements

Key advantages and capabilities

Why this is optimal

Related questions

Dataflow Unified Batch and Stream Processing