Your team manages an automated batch analytics pipeline deployed on a Google Cloud Dataproc cluster with autoscaling enabled. During analysis of recent billing alerts, you notice that the cluster consistently scales up during Apache Spark job execution but fails to scale down worker nodes after processing completes, remaining at peak worker capacity and incurring excess compute costs. Observability metrics in Cloud Monitoring show that YARN Pending resource drops to zero and Available resource is high, but Spark executor containers are not being decommissioned because interim data was persisted into memory using DataFrame cache operations. Which corrective action should you implement to resolve this autoscaling failure?

Configure the spark.dynamicAllocation.cachedExecutorIdleTimeout property or uncache datasets when they are no longer required.: ### What this solution is In Apache Spark on Google Cloud Dataproc, **dynamic allocation** allows the Spark engine to dynamically request and release executor containers from YARN based on workload demands. However, by default, Spark protects executors hosting cached or persisted in-memory datasets from eviction to prevent costly data recomputation. ### How it addresses the autoscaling failure - **Releases pinned executors:** Configuring `spark.dynamicAllocation.cachedExecutorIdleTimeout` sets a maximum duration that an executor holding cached data can sit idle before Spark actively evicts it and releases the underlying YARN container. - **Enables cluster downscaling:** Once the cached executor containers are decommissioned, YARN marks the worker nodes as empty and allows the Dataproc **autoscaling policy** to scale down the cluster to the minimum required worker threshold. - **Optimizes memory management:** Explicitly calling `unpersist()` or uncaching datasets when downstream transformations are complete frees cluster memory immediately, removing the block on executor removal. ### Key advantages and capabilities - **Automated resource reclamation:** Prevents zombie worker nodes from persisting indefinitely after batch jobs conclude. - **Cost governance:** Eliminates unnecessary Compute Engine VM hourly charges without interrupting active queries. - **Native integration:** Operates seamlessly with YARN resource management and Dataproc autoscaling algorithms. - **Balanced performance:** Balances query caching performance with elastic infrastructure efficiency. ### Why this is optimal This approach directly remediates the root cause of the scaling constraint without disabling autoscaling or compromising execution efficiency for future workloads.