An enterprise organization is experiencing surging Cloud Logging ingestion costs across multiple production Google Cloud projects. An analysis of log volume via Cloud Monitoring Metrics Explorer reveals two major contributors to the high ingestion bill: GKE system overhead: High-frequency container and pod logs originating from infrastructure namespaces such as kube-system and monitoring-system. Dataflow worker logs: Verbose worker step telemetry from batch pipelines where engineers only require pipeline lifecycle, error, and supportability data to troubleshoot job failures. You must implement a logging optimization strategy that eliminates redundant log ingestion and minimizes storage costs while preserving critical GKE workload logs, Kubernetes audit trails, and essential Dataflow troubleshooting capabilities. What should you do to achieve this objective?

Configure exclusion filters on the _Default log sinks targeting GKE system namespaces with resource.type=("k8s_container" OR "k8s_pod") AND resource.labels.namespace_name=("kube-system" OR "monitoring-system"), and exclude Dataflow step logs where labels."dataflow.googleapis.com/log_type"!="system" AND labels."dataflow.googleapis.com/log_type"!="supportability".: ### What Log Exclusion Filters Are **Log exclusion filters** in Google Cloud Logging are rules attached to log sinks (such as the default `_Default` sink or user-defined aggregated sinks) that explicitly discard matching log entries from being routed to and ingested by downstream storage destinations like log buckets. ### How It Meets the Requirements - **Eliminating GKE System Overhead:** By creating an exclusion rule on `resource.type=("k8s_container" OR "k8s_pod")` combined with `resource.labels.namespace_name=("kube-system" OR "monitoring-system")`, high-frequency system pod telemetry is dropped before ingestion into the project `_Default` bucket, while user application workloads running in custom namespaces continue to be ingested normally. - **Preserving GKE Auditability:** Excluding container and pod logs from specific namespaces does not disable Kubernetes Audit Logs written by the API server (routed via Cloud Audit Logs) or critical cluster event logs, maintaining essential security and administrative visibility. - **Optimizing Dataflow Telemetry:** Filtering out `resource.type="dataflow_step"` where `labels."dataflow.googleapis.com/log_type"!="system" AND labels."dataflow.googleapis.com/log_type"!="supportability"` removes noisy, low-value worker thread execution output while guaranteeing that essential system lifecycle messages and supportability records needed by developers and Google support remain available in the **Job details** console. - **Reducing Ingestion Costs:** Excluded log entries are dropped by the Log Router and are not billed for log ingestion or storage. ### Key Advantages - **Precision at Source:** Granular Cloud Logging query expressions allow targeted exclusion of high-volume, non-critical logs without impacting application observability. - **Supportability Retention:** Maintains core operational logs necessary for Google Cloud support cases and developer debugging. - **Zero Code Modification:** Policy enforcement occurs purely at the Cloud Logging control plane with no changes to application containers or pipeline worker scripts. ### Why This Is Optimal This approach directly stops the ingestion of non-critical, high-volume log streams at the Log Router level while precisely retaining all telemetry required for Day-2 cluster operations and pipeline diagnostics.

Cloud Logging Gke Dataflow Exclusion Filters | Professional Cloud DevOps Engineer

Cloud Logging Gke Dataflow Exclusion Filters

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise organization is experiencing surging Cloud Logging ingestion costs across multiple production Google Cloud projects. An analysis of log volume via Cloud Monitoring Metrics Explorer reveals two major contributors to the high ingestion bill:

GKE system overhead: High-frequency container and pod logs originating from infrastructure namespaces such as kube-system and monitoring-system.
Dataflow worker logs: Verbose worker step telemetry from batch pipelines where engineers only require pipeline lifecycle, error, and supportability data to troubleshoot job failures.

You must implement a logging optimization strategy that eliminates redundant log ingestion and minimizes storage costs while preserving critical GKE workload logs, Kubernetes audit trails, and essential Dataflow troubleshooting capabilities.

What should you do to achieve this objective?

Create an organization-level non-intercepting aggregated sink routed to Pub/Sub with an inclusion filter for Dataflow and GKE logs, and disable the _Required sink in each underlying project.

Configure exclusion filters on the _Default log sinks targeting GKE system namespaces with resource.type=("k8s_container" OR "k8s_pod") AND resource.labels.namespace_name=("kube-system" OR "monitoring-system"), and exclude Dataflow step logs where labels."dataflow.googleapis.com/log_type"!="system" AND labels."dataflow.googleapis.com/log_type"!="supportability".

Update the inclusion filter of the project-level _Default sinks to empty string (""), and set a 10% sample() exclusion rate across all resource.type values.

Configure an intercepting aggregated sink at the folder level routed directly to a central Cloud Storage bucket with an exclusion filter matching resource.type=("k8s_cluster" OR "dataflow_job").

Professional Cloud DevOps Engineer

Cloud Logging Gke Dataflow Exclusion Filters

Related questions