Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team manages a Cloud Composer environment that runs daily batch ETL pipelines. A newly deployed workflow triggers dozens of resource-intensive, long-running data processing tasks in parallel.
During peak workflow execution, the team observes severe performance degradation and worker resource exhaustion. However, the Cloud Composer autoscaling mechanism fails to provision additional Airflow workers because the Celery task queue remains empty as the existing workers immediately pull and accept all queued tasks.
Which configuration adjustment should the data engineer implement to resolve the resource exhaustion and enable proper autoscaling?
The [core]max_active_runs_per_dag configuration dictates how many concurrent DAG execution instances (DAG runs) can exist in parallel for a single DAG definition. Reducing worker memory limits lowers the amount of RAM allocated to worker containers in the underlying Kubernetes cluster.
Increasing active runs per DAG is suitable when backfilling historical DAG runs sequentially throttles pipeline progression and individual tasks are ultra-lightweight and consume negligible memory.
The [scheduler]dag_dir_list_interval configuration determines how frequently the Airflow scheduler scans the Cloud Storage DAGs bucket for new or modified files. The [core]parallelism parameter sets the global upper boundary on the number of task instances that can run simultaneously across the entire Airflow environment.
This would be the correct path if the Airflow scheduler and metadata database were experiencing severe performance bottlenecks due to excessive Cloud Storage directory scans and too many active task instances overloading the Airflow database backend.
Increasing [celery]worker_concurrency commands each Airflow worker to pull even more task instances concurrently from the broker queue. Allocating higher scheduler CPU enhances DAG parsing frequency, file serialization, and task scheduling loop efficiency.
This approach would be appropriate if the workload consisted of hundreds of lightweight, fast-completing I/O or sensor tasks where workers had very low CPU/memory utilization and scheduler scheduling latency was identified as the primary bottleneck.
In Cloud Composer, [celery]worker_concurrency defines the maximum number of task instances an individual Airflow worker can pull from the Celery queue and process simultaneously. The maximum number of workers parameter establishes the upper boundary for horizontal autoscaling within the underlying Google Kubernetes Engine (GKE) cluster.
[celery]worker_concurrency ensures that excess long-running tasks remain in the queue.For pipelines dominated by long-running, CPU/memory-intensive tasks, high worker concurrency creates a false signal that worker capacity is sufficient while starving container resources. Reducing worker concurrency while raising maximum worker limits aligns queue metrics with real workload demands.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated