Troubleshoot Dataproc Agent Master Node Oom

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An analytics team runs daily analytics workloads by submitting multiple concurrent Spark jobs to a shared Dataproc cluster. Recently, several newly submitted jobs failed immediately with the error:

No agent on master node(s) found to be active

Other pending jobs reported Task was not acquired. You need to identify the root cause, restore cluster operation, and prevent future job submission failures caused by driver resource saturation.

What should you do?

Execute the df -h command across all worker nodes to inspect HDFS scratch space, and recreate the cluster with larger primary persistent disks.

Connect via SSH or view serial port logs on the master VM to diagnose out-of-memory (OOM) conditions on the Dataproc agent; resolve the bottleneck by configuring driver scheduling parameters or allocating dedicated driver node groups for submitted jobs.

Query Cloud Audit Logs for cluster deletion events, verify the cluster's scheduled auto-delete TTL duration, and recreate the cluster without scheduled deletion.

Modify the Cloud Composer DAG configuration by increasing the task execution timeout and switching the graph view orientation from LR to RL.

Professional Cloud Data Engineer

Troubleshoot Dataproc Agent Master Node Oom

Related questions