Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise currently maintains a large, multi-tenant, 24/7 persistent Google Cloud Dataproc cluster running scheduled Apache Spark and Apache Hive analytics workloads. Table data resides on the cluster's local Hadoop Distributed File System (HDFS), and table metadata is stored in a local MySQL Hive metastore instance residing on the primary master node.
The team faces significant compute overhead costs during idle periods and resource contention during peak processing windows. To resolve these issues, the team wants to transition to an ephemeral, job-scoped cluster model orchestrated by Cloud Composer.
They have the following requirements:
Which architecture should the data engineering team implement?
This approach uses Cloud Storage for file persistence but maintains a default embedded Apache Derby metastore within each newly spun-up Dataproc master node. To discover existing Hive tables and partitions in Cloud Storage, the cluster executes a metadata discovery command like MSCK REPAIR TABLE during its initialization phase.
MSCK REPAIR TABLE scans the entire storage prefix directory tree in Cloud Storage to rebuild partition metadata on every job execution. For large datasets with thousands of partitions, this operation introduces massive startup latency, high Cloud Storage API call volumes, and potential rate-limiting errors.You were developing or unit-testing an isolated, single-step PySpark script on a single-node local machine or developer sandbox where persistent schemas and multi-job metadata sharing were not required.
This design maintains a persistent, 24/7 single-node Dataproc cluster dedicated exclusively to running the Hive Metastore daemon and the HDFS NameNode, while ephemeral worker clusters are created dynamically to execute processing jobs and communicate back to the persistent master over Hadoop Remote Procedure Call (RPC) protocols.
You were deploying an unmanaged, custom Hadoop cluster on Compute Engine instances with custom RPC networking plugins rather than utilizing managed Google Cloud native big data services.
This architecture fully decouples compute, storage, and metadata management for big data processing on Google Cloud. Instead of relying on local Hadoop Distributed File System (HDFS) and an on-cluster database for table metadata, the data files are externalized to Google Cloud Storage (GCS) and metadata schemas are managed centrally by a fully managed Dataproc Metastore service.
gs:// URI scheme), compute clusters can be provisioned on-demand, scaled, and deleted immediately when jobs finish without risking any data loss.Decoupling compute from storage is the foundational cloud-native best practice for Hadoop and Spark workloads on Google Cloud. Externalizing state to Cloud Storage and Dataproc Metastore allows organizations to leverage short-lived, specialized clusters tailored to specific job resource requirements (e.g., memory-optimized, compute-optimized, or GPU-attached) while maintaining unified data assets and metadata governance.
This approach keeps primary datasets on cluster-attached persistent disks managed by HDFS and relies on scheduled HDFS snapshots and scripts (e.g., using distcp or initialization actions) to synchronize data into Cloud Storage, while running an on-master MySQL database for Hive metadata.
You had legacy on-premises applications with strict POSIX compliance requirements or non-standard file system operations (like append-heavy operations) that could not leverage the Cloud Storage connector.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated