A data engineering team manages a Cloud Composer environment that runs daily batch ETL pipelines. A newly deployed workflow triggers dozens of resource-intensive, long-running data processing tasks in parallel. During peak workflow execution, the team observes severe performance degradation and worker resource exhaustion. However, the Cloud Composer autoscaling mechanism fails to provision additional Airflow workers because the Celery task queue remains empty as the existing workers immediately pull and accept all queued tasks. Which configuration adjustment should the data engineer implement to resolve the resource exhaustion and enable proper autoscaling?

Decrease the worker concurrency ([celery]worker_concurrency) and increase the maximum number of workers.: What Worker Concurrency and Worker Scaling Limits Are In Cloud Composer, [celery]worker_concurrency defines the maximum number of task instances an individual Airflow worker can pull from the Celery queue and process simultaneously. The maximum number of workers parameter establishes the upper boundary for horizontal autoscaling within the underlying Google Kubernetes Engine (GKE) cluster. How It Meets the Scenario Requirements Triggers Autoscaling Efficiently: Cloud Composer evaluates queue depth (the number of pending tasks in the Celery broker queue) to decide when to scale out workers. High worker concurrency allows a minimal number of workers to eagerly grab all tasks, preventing a queue backlog from forming. Decreasing [celery]worker_concurrency ensures that excess long-running tasks remain in the queue. Prevents Worker Resource Starvation: By reducing concurrent executions per worker pod, each active task instance gets dedicated CPU and memory resources, preventing out-of-memory errors and pod evictions. Scales Compute Horizontally: Increasing the maximum number of workers provides the required ceiling for Cloud Composer to spin up new pods when task backlog builds up in the queue. Key Advantages Optimal Resource Allocation: Prevents heavy workloads from overloading individual worker nodes. Dynamic Elasticity: Ensures horizontal pod autoscaling behaves predictably based on actual task demand. Increased Task Throughput: Distributes heavy execution across isolated worker pods rather than over-saturating a single container. Why This Is Optimal For pipelines dominated by long-running, CPU/memory-intensive tasks, high worker concurrency creates a false signal that worker capacity is sufficient while starving container resources. Reducing worker concurrency while raising maximum worker limits aligns queue metrics with real workload demands.

Optimize Composer Worker Concurrency Autoscaling | Professional Cloud Data Engineer

StudyCerts

Explanations

What Worker Concurrency and Worker Scaling Limits Are

In Cloud Composer, [celery]worker_concurrency defines the maximum number of task instances an individual Airflow worker can pull from the Celery queue and process simultaneously. The maximum number of workers parameter establishes the upper boundary for horizontal autoscaling within the underlying Google Kubernetes Engine (GKE) cluster.

How It Meets the Scenario Requirements

Triggers Autoscaling Efficiently: Cloud Composer evaluates queue depth (the number of pending tasks in the Celery broker queue) to decide when to scale out workers. High worker concurrency allows a minimal number of workers to eagerly grab all tasks, preventing a queue backlog from forming. Decreasing [celery]worker_concurrency ensures that excess long-running tasks remain in the queue.
Prevents Worker Resource Starvation: By reducing concurrent executions per worker pod, each active task instance gets dedicated CPU and memory resources, preventing out-of-memory errors and pod evictions.
Scales Compute Horizontally: Increasing the maximum number of workers provides the required ceiling for Cloud Composer to spin up new pods when task backlog builds up in the queue.

Key Advantages

Optimal Resource Allocation: Prevents heavy workloads from overloading individual worker nodes.
Dynamic Elasticity: Ensures horizontal pod autoscaling behaves predictably based on actual task demand.
Increased Task Throughput: Distributes heavy execution across isolated worker pods rather than over-saturating a single container.

Why This Is Optimal

For pipelines dominated by long-running, CPU/memory-intensive tasks, high worker concurrency creates a false signal that worker capacity is sufficient while starving container resources. Reducing worker concurrency while raising maximum worker limits aligns queue metrics with real workload demands.

View source documentation

Professional Cloud Data Engineer

Optimize Composer Worker Concurrency Autoscaling