professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
An analytics team runs daily analytics workloads by submitting multiple concurrent Spark jobs to a shared Dataproc cluster. Recently, several newly submitted jobs failed immediately with the error:
No agent on master node(s) found to be active
Other pending jobs reported Task was not acquired. You need to identify the root cause, restore cluster operation, and prevent future job submission failures caused by driver resource saturation.
What should you do?
Execute the df -h command across all worker nodes to inspect HDFS scratch space, and recreate the cluster with larger primary persistent disks.
Query Cloud Audit Logs for cluster deletion events, verify the cluster's scheduled auto-delete TTL duration, and recreate the cluster without scheduled deletion.
Modify the Cloud Composer DAG configuration by increasing the task execution timeout and switching the graph view orientation from LR to RL.
Connect via SSH or view serial port logs on the master VM to diagnose out-of-memory (OOM) conditions on the Dataproc agent; resolve the bottleneck by configuring driver scheduling parameters or allocating dedicated driver node groups for submitted jobs.
Execute the df -h command across all worker nodes to inspect HDFS scratch space, and recreate the cluster with larger primary persistent disks.
Query Cloud Audit Logs for cluster deletion events, verify the cluster's scheduled auto-delete TTL duration, and recreate the cluster without scheduled deletion.
Modify the Cloud Composer DAG configuration by increasing the task execution timeout and switching the graph view orientation from LR to RL.
Connect via SSH or view serial port logs on the master VM to diagnose out-of-memory (OOM) conditions on the Dataproc agent; resolve the bottleneck by configuring driver scheduling parameters or allocating dedicated driver node groups for submitted jobs.
This solution uses SSH connection and serial port console logs to inspect the Dataproc cluster's master node status, identifies agent process crashes caused by out-of-memory (OOM) exceptions, and mitigates future occurrences using driver scheduling configurations or driver node groups.
StudyCerts
StudyCertsgoogle.dataproc.agent).The errors No agent on master node(s) found to be active and Task was not acquired typically indicate that the Dataproc agent on the master VM has crashed or become unresponsive due to severe memory exhaustion. Isolating driver resource allocations addresses both the diagnostic and preventative requirements.