professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
An enterprise currently maintains a large, multi-tenant, 24/7 persistent Google Cloud Dataproc cluster running scheduled Apache Spark and Apache Hive analytics workloads. Table data resides on the cluster's local Hadoop Distributed File System (HDFS), and table metadata is stored in a local MySQL Hive metastore instance residing on the primary master node.
The team faces significant compute overhead costs during idle periods and resource contention during peak processing windows. To resolve these issues, the team wants to transition to an ephemeral, job-scoped cluster model orchestrated by Cloud Composer.
They have the following requirements:
Which architecture should the data engineering team implement?
Store table data in Cloud Storage buckets accessed via the Cloud Storage connector, configure clusters to connect to an external Dataproc Metastore service, and delete each cluster after job completion.
Retain a long-running, 24/7 master-only Dataproc cluster to host the Hive Metastore and NameNode, while launching ephemeral worker-only compute clusters configured with remote Hadoop RPC endpoints.
Maintain table data on worker HDFS local persistent disks, back up HDFS snapshots to Cloud Storage using an initialization action, and host MySQL metastore on the primary master node.
Store table datasets in Cloud Storage, initialize each ephemeral cluster with an embedded local Apache Derby metastore, and run an initialization script that executes MSCK REPAIR TABLE on every job startup.
Store table data in Cloud Storage buckets accessed via the Cloud Storage connector, configure clusters to connect to an external Dataproc Metastore service, and delete each cluster after job completion.
This architecture fully decouples compute, storage, and metadata management for big data processing on Google Cloud. Instead of relying on local Hadoop Distributed File System (HDFS) and an on-cluster database for table metadata, the data files are externalized to Google Cloud Storage (GCS) and metadata schemas are managed centrally by a fully managed Dataproc Metastore service.
StudyCerts
StudyCertsgs:// URI scheme), compute clusters can be provisioned on-demand, scaled, and deleted immediately when jobs finish without risking any data loss.Decoupling compute from storage is the foundational cloud-native best practice for Hadoop and Spark workloads on Google Cloud. Externalizing state to Cloud Storage and Dataproc Metastore allows organizations to leverage short-lived, specialized clusters tailored to specific job resource requirements (e.g., memory-optimized, compute-optimized, or GPU-attached) while maintaining unified data assets and metadata governance.
Retain a long-running, 24/7 master-only Dataproc cluster to host the Hive Metastore and NameNode, while launching ephemeral worker-only compute clusters configured with remote Hadoop RPC endpoints.
Maintain table data on worker HDFS local persistent disks, back up HDFS snapshots to Cloud Storage using an initialization action, and host MySQL metastore on the primary master node.
Store table datasets in Cloud Storage, initialize each ephemeral cluster with an embedded local Apache Derby metastore, and run an initialization script that executes MSCK REPAIR TABLE on every job startup.