Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An analytics team runs daily analytics workloads by submitting multiple concurrent Spark jobs to a shared Dataproc cluster. Recently, several newly submitted jobs failed immediately with the error:
No agent on master node(s) found to be active
Other pending jobs reported Task was not acquired. You need to identify the root cause, restore cluster operation, and prevent future job submission failures caused by driver resource saturation.
What should you do?
This solution uses SSH connection and serial port console logs to inspect the Dataproc cluster's master node status, identifies agent process crashes caused by out-of-memory (OOM) exceptions, and mitigates future occurrences using driver scheduling configurations or driver node groups.
google.dataproc.agent).The errors No agent on master node(s) found to be active and Task was not acquired typically indicate that the Dataproc agent on the master VM has crashed or become unresponsive due to severe memory exhaustion. Isolating driver resource allocations addresses both the diagnostic and preventative requirements.
This option modifies workflow execution parameters within Cloud Composer (Apache Airflow), adjusting DAG timeout settings and changing the UI layout of DAG graph representations.
dag_orientation) is purely a visual UI setting for large DAG navigation.The challenge was navigating a massive Airflow DAG graph in the web UI to visually identify failed operator tasks.
When Cloud Composer submits jobs to Dataproc via operators, Dataproc agent failures will surface in Airflow logs as job submission errors, requiring remediation on the Dataproc cluster directly.
This approach investigates cluster lifecycle events in Cloud Audit Logs to identify whether the cluster was prematurely torn down by an automated scheduled deletion policy or user action.
Task not found, indicating that the underlying compute resources were terminated while the YARN application was in a running state.No agent on master node(s) found to be active and Task was not acquired) occur while the cluster is still running, resulting from process failure or network unreachability on the master VM rather than cluster deletion.DeleteCluster API requests initiated by users or the Dataproc service agent.The workload failed with the error Task not found and the cluster had disappeared from the Google Cloud console during active job execution.
Scheduled deletion is used to control costs on ephemeral clusters, but long-running shared clusters should have auto-delete policies configured appropriately or disabled.
df -h command across all worker nodes to inspect HDFS scratch space, and recreate the cluster with larger primary persistent disks.This approach checks disk utilization across cluster nodes using Linux storage commands (df -h) to detect local storage exhaustion and remedies it by provisioning larger primary persistent disks on cluster creation.
No space left on device during shuffle or HDFS write operations, not as agent communication failures.df -h helps analyze HDFS capacity and local disk usage for shuffle files.n1-standard-4) prevent disk exhaustion for large shuffle jobs.The job output showed task failures caused by No space left on device when writing temporary shuffle files or staging HDFS data.
Dataproc master nodes need adequate memory for cluster coordination daemons, including YARN ResourceManager and the Dataproc agent, independent of persistent disk capacity.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated