Troubleshoot Dataproc Agent Master Node Oom

What the Solution Is

This solution uses SSH connection and serial port console logs to inspect the Dataproc cluster's master node status, identifies agent process crashes caused by out-of-memory (OOM) exceptions, and mitigates future occurrences using driver scheduling configurations or driver node groups.

How It Meets the Requirements

• Identify the root cause: When the Dataproc agent crashes or stops responding to the control plane, inspecting the master VM via SSH or checking the serial port logs provides visibility into kernel OOM killer events and daemon logs (google.dataproc.agent).
• Restore cluster operation: Pinpointing and terminating runaway processes or restarting the VM instance frees up memory resources for the Dataproc control plane agent.
• Prevent future driver resource saturation: Submitting concurrent jobs with explicit DriverSchedulingConfig (specifying memory and vCores) or utilizing dedicated driver node groups offloads memory-intensive driver processes from the primary master VM, preventing resource contention.

Key Advantages and Capabilities

• Accurate root-cause identification: Serial port and system diagnostic logs capture JVM memory exhaustion and OOM kills that occur before application logs are written.
• Protection of the control plane: Offloading driver execution ensures the master node's agent maintains uninterrupted heartbeat communication with Google Cloud's control plane.
• Predictable concurrency: Explicit scheduling parameters enforce resource boundaries across concurrent workloads.

Why This Is Optimal

The errors No agent on master node(s) found to be active and Task was not acquired typically indicate that the Dataproc agent on the master VM has crashed or become unresponsive due to severe memory exhaustion. Isolating driver resource allocations addresses both the diagnostic and preventative requirements.