An enterprise runs microservices on Google Kubernetes Engine (GKE) and utilizes Cloud Bigtable for real-time transactions. The environment generates 25 TB of application logs and a large volume of Data Access audit logs each month. The DevOps team must meet the following observability and budget requirements: Query and join application log data with existing BigQuery business datasets using standard SQL. Retain application logs for 90 days before deletion. Eliminate duplicate storage charges and unnecessary ingestion costs from high-volume Bigtable service accounts. Which strategy should the DevOps team implement to achieve these goals?

Exempt the application service accounts in the project's Data Access audit configuration, upgrade a user-defined log bucket to use Log Analytics with a 90-day retention period, and create a linked BigQuery dataset.: ### What the solution is This solution combines **audit configuration exemptions**, **custom log bucket retention**, and **Log Analytics linked datasets** to establish a cost-optimized, queryable log lifecycle. ### How it meets the requirements - **Eliminating unnecessary log ingestion and processing costs**: Configuring **service account exemptions** under the Data Access audit policy prevents Bigtable `DATA_READ` and `DATA_WRITE` logs from being generated at the source, preventing unnecessary data transfer and ingestion fees. - **Retaining logs for 90 days**: Configuring a **90-day custom retention period** on the user-defined log bucket satisfies compliance and analytical requirements directly within Cloud Logging. - **Enabling SQL joins without double storage**: Upgrading the bucket to **Log Analytics** (which has no upgrade fee) and provisioning a **linked BigQuery dataset** allows BigQuery Studio and SQL queries to read log data in-place. This completely avoids routing duplicate copies of log records into BigQuery native table storage. ### Key advantages and capabilities - **Zero data duplication**: BigQuery queries query the underlying Cloud Logging storage directly via read-only metadata links. - **Full SQL capability**: Developers and data analysts can run complex analytical SQL queries joining `LogEntry` records with relational tables in BigQuery. - **Source-level suppression**: Audit exemptions prevent log generation altogether rather than dropping them downstream after resource consumption has already occurred. - **Centralized retention**: Log retention is managed through a single policy on the log bucket rather than synchronizing multiple database partition expiration rules. ### Why this is optimal Alternative approaches that rely on export sinks duplicate log records into both Cloud Logging buckets and BigQuery storage tables, doubling ingestion and storage expenses. Leveraging Log Analytics linked datasets provides full BigQuery analytical capabilities on a single copy of log data.

Optimize Log Retention Tiering Bigquery Linked Datasets | Professional Cloud DevOps Engineer

Optimize Log Retention Tiering Bigquery Linked Datasets

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise runs microservices on Google Kubernetes Engine (GKE) and utilizes Cloud Bigtable for real-time transactions. The environment generates 25 TB of application logs and a large volume of Data Access audit logs each month.

The DevOps team must meet the following observability and budget requirements:

Query and join application log data with existing BigQuery business datasets using standard SQL.
Retain application logs for 90 days before deletion.
Eliminate duplicate storage charges and unnecessary ingestion costs from high-volume Bigtable service accounts.

Which strategy should the DevOps team implement to achieve these goals?

Exempt the application service accounts in the project's Data Access audit configuration, upgrade a user-defined log bucket to use Log Analytics with a 90-day retention period, and create a linked BigQuery dataset.

Route all application logs to a Cloud Storage bucket with Autoclass enabled, and run a scheduled Cloud Run job every 24 hours to import the log batch into BigQuery.

Create a Log Router sink to stream logs directly to a partitioned BigQuery table with a 90-day partition expiration, set an exclusion filter on the sink for service accounts, and keep the Cloud Logging bucket retention at 30 days.

Configure field-level access controls on the log bucket to mask PII, extend the default log bucket retention to 90 days, and configure a Log Analytics linked dataset in BigQuery.

Professional Cloud DevOps Engineer

Optimize Log Retention Tiering Bigquery Linked Datasets

Related questions