Cloud Composer Etl Deferrable Operators Reliability

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team uses Cloud Composer to orchestrate a complex daily ETL pipeline that integrates Cloud Storage, Dataproc Serverless, and BigQuery. During peak batch windows, several tasks fail abruptly with Negsignal.SIGKILL and Kubernetes worker Pod evictions. In addition, the task queue experiences severe head-of-line blocking because Airflow workers remain fully occupied while synchronously polling long-running Dataproc jobs and BigQuery queries.

Which design strategy should the team implement to optimize workflow execution and resolve these reliability issues?

Use deferrable Airflow operators with triggerers to release worker slots during long-running tasks, process datasets natively in Dataproc and BigQuery without local worker downloads, and configure task-level retries.

Increase [celery]worker_concurrency across all Airflow workers, pass intermediate transform payloads via XCom, and expand local worker ephemeral storage.

Download source objects locally into the Cloud Composer /data directory using gcsfuse and rely on worker autoscaling to absorb the memory and storage spikes.

Migrate the data pipeline orchestration to Cloud Workflows, using inline YAML workflow steps to download, transform, and buffer the multi-gigabyte files in memory.

Professional Cloud Data Engineer

Cloud Composer Etl Deferrable Operators Reliability

Related questions