An enterprise data analytics platform processes real-time events across several Google Cloud projects within an organization folder. The data engineering team needs to design a unified observability and alerting architecture using Google Cloud Observability to track pipeline health, custom processing latency, and resource utilization. The solution must satisfy the following technical requirements: Provide a single centralized dashboard in a designated management project that aggregates metrics and logs across all constituent pipeline projects. Ingest custom workload performance metrics and system logs from pipeline worker instances. Trigger automated alerts that notify on-call engineers whenever processing latency anomalies occur, avoiding silent alert executions. Which combination of steps should the team implement?

Configure the management project's metrics scope to include all worker projects and define a default log scope with the appropriate log views; install the Ops Agent on worker instances to stream workload metrics and logs; build a custom dashboard using dashboard configuration JSON; and configure Cloud Monitoring alerting policies linked to explicit notification channels.: What the centralized observability architecture is This architecture establishes a centralized observability strategy by consolidating telemetry from multiple Google Cloud projects into a designated management project metrics scope and log scope. It relies on the Ops Agent for collecting system and workload telemetry, custom dashboard definitions via configuration files, and Cloud Monitoring alerting policies paired with defined notification channels. How it meets the requirements Aggregates metrics and logs across projects: Configuring a metrics scope in the management project and linking project log views (such as _Default/_AllLogs) into a default log scope allows engineers to visualize time-series telemetry and logs across all pipeline projects from a single pane of glass. Ingests custom workload performance metrics: Deploying the Ops Agent on worker compute resources enables ingestion of system metrics, process logs, and workload-specific metrics (such as latency measurements under workload.googleapis.com/...). Alerts on-call engineers without silent firing: Defining alerting policies with configured notification channels (e.g., PagerDuty, Slack, Email) ensures on-call staff receive proactive incident alerts rather than having policies fire silently without notifying personnel. Centralized dashboard: Deploying dashboards using infrastructure-as-code or gcloud monitoring dashboards create with JSON templates creates unified visualization panels for pipeline health. Key advantages Centralized visibility: Prevents siloed monitoring across multi-project data processing boundaries. Out-of-the-box telemetry: Ops Agent handles logging and metric streaming seamlessly. Operational reliability: Explicit notification channel binding ensures zero missed degradation incidents. Why this is optimal This approach aligns with Google Cloud architecture best practices by decoupling workload execution from centralized monitoring and governance, ensuring complete fidelity without custom log-forwarding infrastructure.

Centralized Pipeline Observability Alerting Monitoring

AdvancedProfessional Cloud Data Engineer

A data engineering team uses Cloud Composer to orchestrate a complex daily ETL pipeline that integrates Cloud Storage, Dataproc Serverless, and BigQuery. During peak batch windows, several tasks fail abruptly with Negsignal.SIGKILL and Kubernetes worker Pod evictions. In addition, the task queue experiences severe head-of-line blocking because Airflow workers remain fully occupied while synchronously polling long-running Dataproc jobs and BigQuery queries. Which design strategy should the team implement to optimize workflow execution and resolve these reliability issues?

Monitoring and orchestration of data pipelines

AdvancedProfessional Cloud Data Engineer

An enterprise is designing a mission-critical streaming data pipeline using Google Cloud Dataflow that ingests high-volume telemetry events and writes them into Google BigQuery. The architecture must satisfy the following operational requirements: Ensure data fidelity and pipeline resilience by preventing malformed records from crashing or blocking the streaming pipeline. Route all failed record insertions along with their corresponding error messages to a secondary dead-letter destination for root-cause analysis and automated reprocessing. Guarantee idempotency across the processing lifecycle so that upstream at-least-once message retries do not introduce duplicate records. Which architectural strategy should the data engineering team implement to achieve these requirements?

Monitoring and orchestration of data pipelines

AdvancedProfessional Cloud Data Engineer

Your team is designing an event-driven data pipeline using Eventarc to trigger a Cloud Run microservice whenever a new file is uploaded to Cloud Storage. The microservice processes the file and updates a stateful database. Because Eventarc guarantees at-least-once delivery, the microservice occasionally receives duplicate events for the same file upload, causing duplicate database entries. You need to implement a solution to prevent data duplication and ensure the database is only updated once per event. What should you do?

Designing system for fault tolerance and managing restarts

AdvancedProfessional Cloud Data Engineer

A data engineering team is deploying automated machine learning pipelines with BigQuery ML and Vertex AI. To satisfy strict governance and reproducibility requirements across iterative model runs, the team must implement an architecture that: Captures immutable, point-in-time versions of training datasets from rapidly changing BigQuery tables Automatically tracks the lineage and linkage across dataset artifacts, hyperparameters, and model evaluations Detects training-serving skew by evaluating production inference data against baseline training statistics Which combination of Google Cloud services and practices should the team implement?

Preparing data for feature engineering, training and serving machine learning models (e.g., BigQueryML)

AdvancedProfessional Cloud Data Engineer

A data engineering team maintains a Cloud SQL for PostgreSQL instance that serves data models for executive reporting dashboards. During morning peak hours, numerous concurrent dashboard sessions execute complex queries involving large ORDER BY, GROUP BY, and multi-table join operations. Database logs reveal recurring Out of Memory: Killed process (postgres) errors and connection terminations (terminating connection due to administrator command). Monitoring shows high active session counts and severe RAM saturation. Which resource configuration adjustment should the data engineer implement to balance query performance and prevent out-of-memory errors?

Troubleshooting poor performing queries

FundamentalAssociate Data Practitioner

ELT vs ETL

Evaluate use cases for ELT and ETL

Professional Cloud Data Engineer

Centralized Pipeline Observability Alerting Monitoring

Related questions

ELT vs ETL

Professional Cloud Data Engineer

Centralized Pipeline Observability Alerting Monitoring

What the centralized observability architecture is

How it meets the requirements

Key advantages

Why this is optimal

Related questions

ELT vs ETL

What the centralized observability architecture is

How it meets the requirements

Key advantages

Why this is optimal