Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team uses Cloud Composer to orchestrate a complex daily ETL pipeline that integrates Cloud Storage, Dataproc Serverless, and BigQuery. During peak batch windows, several tasks fail abruptly with Negsignal.SIGKILL and Kubernetes worker Pod evictions. In addition, the task queue experiences severe head-of-line blocking because Airflow workers remain fully occupied while synchronously polling long-running Dataproc jobs and BigQuery queries.
Which design strategy should the team implement to optimize workflow execution and resolve these reliability issues?
In Cloud Composer, the /data folder in the environment's Cloud Storage bucket is mounted to worker pods using gcsfuse, allowing file-like read and write access to Cloud Storage objects.
/data folder causes the worker to pull large files onto its local ephemeral disk. When worker storage capacity is exceeded, the underlying Kubernetes control plane evicts the worker Pod, immediately failing all active tasks.Transport endpoint is not connected errors./data folder is not synchronized to the web server in modern Composer environments and introduces severe I/O bottlenecks when used for large data transfers.Tasks only needed to read small static configuration files or shared lookup scripts under a few megabytes.
Cloud Workflows is a fully managed, serverless orchestration platform that coordinates Google Cloud APIs, Cloud Run functions, and external HTTP-based microservices using YAML or JSON definitions.
The objective was to create a low-latency, serverless API orchestration workflow chaining lightweight Cloud Run functions and HTTP endpoints without large data payloads.
Deferrable operators (also known as asynchronous operators) leverage the Airflow triggerer component to yield worker slots while waiting for external jobs—such as BigQuery queries or Dataproc Serverless batches—to complete. Instead of keeping a heavy Airflow worker thread blocked during passive polling, the task is suspended and monitored by an asynchronous event loop in the triggerer until the external service signals completion.
SIGKILL) and Pod Evictions: Ensuring that large data transformation workloads execute directly inside dedicated services (BigQuery and Dataproc) rather than downloading files or holding data frames in the Airflow worker filesystem (/data or local disk) eliminates local memory and storage exhaustion.Cloud Composer workers are designed for lightweight orchestration and workflow logic, not intensive data manipulation. Offloading data processing to backend engines while releasing polling slots via triggerers directly addresses both resource starvation and queuing bottlenecks.
This approach attempts to resolve task concurrency bottlenecks by increasing the [celery]worker_concurrency setting, utilizing Airflow XCom to pass intermediate transformation datasets between DAG tasks, and adding storage to worker nodes.
worker_concurrency forces individual Airflow workers to execute more task instances concurrently. Because cumulative memory consumption increases under concurrent load, this makes out-of-memory errors (SIGKILL) more likely, not less.The workload consisted exclusively of lightweight, sub-second API status checks where tasks transfer micro-metadata (<48 KB) and workers had ample surplus RAM per thread.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated