Dataproc Decouple Storage Metastore Job Clusters

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise currently maintains a large, multi-tenant, 24/7 persistent Google Cloud Dataproc cluster running scheduled Apache Spark and Apache Hive analytics workloads. Table data resides on the cluster's local Hadoop Distributed File System (HDFS), and table metadata is stored in a local MySQL Hive metastore instance residing on the primary master node.

The team faces significant compute overhead costs during idle periods and resource contention during peak processing windows. To resolve these issues, the team wants to transition to an ephemeral, job-scoped cluster model orchestrated by Cloud Composer.

They have the following requirements:

Decouple compute from storage so ephemeral Dataproc clusters can be created dynamically per workload and terminated immediately upon completion without data loss.
Centralize and persist table schemas and metadata across independent, transient cluster executions.
Provide total workload isolation and cost optimization.

Which architecture should the data engineering team implement?

Retain a long-running, 24/7 master-only Dataproc cluster to host the Hive Metastore and NameNode, while launching ephemeral worker-only compute clusters configured with remote Hadoop RPC endpoints.

Store table data in Cloud Storage buckets accessed via the Cloud Storage connector, configure clusters to connect to an external Dataproc Metastore service, and delete each cluster after job completion.

Store table datasets in Cloud Storage, initialize each ephemeral cluster with an embedded local Apache Derby metastore, and run an initialization script that executes MSCK REPAIR TABLE on every job startup.

Maintain table data on worker HDFS local persistent disks, back up HDFS snapshots to Cloud Storage using an initialization action, and host MySQL metastore on the primary master node.

Professional Cloud Data Engineer

Dataproc Decouple Storage Metastore Job Clusters

Related questions