Log routing in Google Cloud directs operational and audit event logs from data processing services to designated storage and analysis destinations using the Cloud Logging Log Router. The Log Router evaluates incoming log entries against defined sinks that filter data by service origin, resource type, or severity level. Depending on operational and analytical needs, the Log Router forwards filtered log entries across three primary destination types:
Identity and Access Management (IAM) permissions must be explicitly granted to the Log Router's service account on the destination resources to allow write operations. Administrators also configure exclusion filters within sinks to discard high-volume, low-value logs, preventing unnecessary data transfer and lowering storage costs. For data processing workloads like Cloud Dataflow and Cloud Dataproc, configuring dedicated sinks ensures that system logs and worker diagnostics are captured for operational troubleshooting.
Log retention policies define how long log data persists in a given destination before automatic deletion occurs. Cloud Logging applies default retention windows based on log type, storing Admin Activity audit logs for 400 days while retaining Data Access audit logs for significantly shorter default periods. When logs are routed to Cloud Storage, organizations use object lifecycle management rules to transition older log objects across storage classes or delete them after a set number of days. For logs routed to BigQuery, retention is controlled at the dataset level by configuring partition and table expiration schedules.
Audit logging tracks user operations, data queries, and administrative changes to maintain security compliance. Cloud Audit Logs capture these events across services and must be routed to secure, immutable storage—such as a locked Cloud Storage bucket—to prevent tampering. Teams can also build custom log-based metrics in Cloud Monitoring to track audit patterns, such as unauthorized data access attempts or administrative permission modifications, and trigger proactive alerts.
Exam tip: Cloud Logging retains Admin Activity audit logs for 400 days by default, whereas Data Access audit logs have shorter default retention periods and must be explicitly enabled to capture data read and write events.
BigQuery query performance depends on active slot consumption and query queue management. Administrators monitor queue depths using the Job Concurrency chart in the BigQuery admin panel or by tracking the job count metric in a pending state within Cloud Monitoring. When incoming queries exceed maximum concurrency limits, BigQuery places additional queries into a queue until slots become available. If unstarted queries remain queued past system limits, BigQuery cancels them with automatic timeouts:
Excessive queue lengths signal slot contention and require adjusting compute capacity or optimizing underlying SQL queries to avoid job failures.
BigQuery slot reservations allocate dedicated processing power to isolate critical analytical workloads from shared resource contention. Organizations transition from on-demand pricing to capacity-based pricing by creating reservations, but workload management features vary across pricing models and edition tiers:
| Pricing model / Edition | Maximum concurrency target support | Workload management capability |
|---|---|---|
| On-demand | Not supported | Distributes queries across a shared multi-tenant slot pool without dedicated capacity limits |
| Standard edition | Not supported | Provides baseline dedicated slot reservations without support for custom concurrency targets |
| Enterprise edition | Supported | Provides advanced concurrency controls to optimize slot sharing and prevent workload starvation |
Choosing Enterprise edition allows administrators to define a maximum concurrency target, ensuring high-priority workloads maintain predictable execution latency even during peak reporting hours.
BigQuery cost governance prevents unexpected billing spikes by enforcing query scan limits and tracking slot utilization. Because BigQuery does not apply daily query limits by default, administrators must set custom quotas to cap query usage per day at the project or user level. Spend tracking is managed by exporting Google Cloud billing data into BigQuery, where teams run analytical queries against billing metadata to identify high-cost jobs. Cloud Monitoring alerting policies track quota consumption and warn administrators before daily limits are exhausted, ensuring continuous operations within budget limits.
Exam tip: Unstarted interactive queries in BigQuery time out after 6 hours in the queue, while unstarted batch queries time out after 24 hours.
Observability in data systems requires continuous collection and visualization of telemetry data to diagnose pipeline health. Telemetry data is structured into three primary operational signals:
Cloud Monitoring uses these signals to track the four golden signals: latency, traffic, errors, and saturation. Real-time dashboards visualize infrastructure and job health metrics across Cloud Dataflow, BigQuery, and Google Kubernetes Engine (GKE), giving engineers immediate visibility into pipeline throughput and resource bottlenecks.
Proactive alerting policies notify data engineering teams of pipeline degradation before complete job failures occur. Engineers write Monitoring Query Language (MQL) queries to create multi-condition alerts that correlate resource exhaustion with processing delays. Alert thresholds are tied directly to Service Level Indicators (SLIs) and Service Level Objectives (SLOs) to minimize alert fatigue while enforcing reliability targets. Effective data pipeline monitoring watches specific telemetry indicators:
When alert conditions are met or log-based errors occur, Cloud Monitoring routes notifications to channels such as email, Slack, or Pub/Sub to trigger automated recovery workflows.
Exam tip: Multi-condition alerting policies written in Monitoring Query Language (MQL) correlate multiple operational signals—such as rising Pub/Sub message age alongside high Dataflow worker CPU utilization—to identify true pipeline degradation while reducing false alarms.
professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
The Cloud Logging Log Router forwards filtered log entries to Cloud Storage for long-term archival, BigQuery for analytical SQL querying and security investigations, and Pub/Sub for real-time streaming to external platforms or automation pipelines. Administrators can also apply exclusion filters within sinks to discard high-volume, low-value logs and reduce costs.
Unstarted interactive queries time out automatically after 6 hours in the queue, whereas unstarted batch queries time out after 24 hours. These timeouts are triggered when incoming queries exceed concurrency limits and remain queued due to slot contention.
Only BigQuery Enterprise edition supports configuring a maximum concurrency target for workload management. In contrast, On-demand pricing and Standard edition reservations do not support custom concurrency targets.
Monitoring Query Language (MQL) enables the creation of multi-condition alerting policies that correlate multiple operational signals, such as rising Pub/Sub backlog growth and high Cloud Dataflow worker utilization. This correlation identifies genuine pipeline degradation and resource exhaustion while minimizing false alarms and alert fatigue.
Assign all queries to run as batch query jobs rather than interactive jobs, and use INFORMATION_SCHEMA.RECOMMENDATIONS to verify monthly slot reductions.
Set a non-zero target_job_concurrency on the reservation using ALTER RESERVATION, and monitor autoscaling capacity costs through reservation-level capacity compute metrics rather than aggregating slot-milliseconds from INFORMATION_SCHEMA.JOBS.
Configure ignore_idle_slots to true on the reservation, and calculate compute billing by aggregating total_slot_ms from INFORMATION_SCHEMA.JOBS grouped by reservation_id.
Set default_interactive_query_queue_timeout_ms to -1 in INFORMATION_SCHEMA.EFFECTIVE_PROJECT_OPTIONS, and increase baseline slots to match peak concurrency demand.
An analytics team is running interactive dashboard queries in BigQuery using an Enterprise Edition reservation configured with baseline slots and autoscaling capacity. During peak reporting hours, numerous concurrent queries execute simultaneously, diluting the slot allocation per query and causing unacceptable latency. Additionally, finance reports discrepancies between query-level slot runtime metrics and compute reservation charges.
Which combination of configuration and observability practices should the data engineering team implement to stabilize query performance and correctly analyze compute costs?
StudyCerts
StudyCerts