Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A financial analytics company is migrating its daily batch data pipeline to Google Cloud. The workload has the following characteristics and constraints:
Which execution framework and architecture should the company select?
Cloud Dataproc is a fully managed service for executing open-source data processing engines, such as Apache Spark, Apache Hadoop, and Hive. An ephemeral cluster pattern creates dedicated compute resources specifically for the duration of a discrete workflow and terminates the cluster immediately upon job completion, persisting data externally in Cloud Storage and BigQuery.
Compared to refactoring pipelines into Apache Beam or BigQuery SQL, ephemeral Dataproc allows the team to migrate legacy Spark applications with minimal operational friction while completely eliminating the continuous cost overhead of persistent servers.
Cloud Dataflow is a fully managed, serverless execution service for data processing pipelines built with the open-source Apache Beam SDK. It automates provisioning, autoscaling, and worker management for both streaming and batch unified data processing workloads.
PTransform, PCollection, and DoFn) introduces significant development, testing, and migration delay.The organization wanted to standardize on a fully serverless data processing engine, was building greenfield pipelines, or was transitioning their batch pipelines into unified real-time streaming architectures using Apache Beam.
While Dataflow is ideal for removing cluster management overhead, re-architecting existing production Spark codebases to Apache Beam is usually cost-prohibitive unless streaming unification or advanced windowing capabilities are required.
BigQuery is an enterprise, serverless data warehouse that supports high-speed SQL analytics and data manipulation (ELT) using its distributed DML, DDL, and stored procedure engines.
The workload consisted purely of SQL queries, data modeling, and relational aggregations, or if the organization was standardizing on an in-warehouse ELT pattern rather than maintaining custom Spark application logic.
While BigQuery SQL is the preferred Google Cloud paradigm for analytical data modeling, using it as a replacement for established multi-step Spark pipelines requires substantial engineering hours to rewrite and validate business logic.
A persistent Cloud Dataproc cluster is a dedicated set of master and worker virtual machines running in Compute Engine that remain continuously active regardless of whether jobs are running.
The cluster had to serve continuous, low-latency ad-hoc queries from interactive users throughout the day (such as multiple data analysts using Jupyter notebooks or Hive/Presto queries) with hundreds of jobs submitted around the clock.
For batch workloads that run periodically on Google Cloud, the recommended architectural best practice is to decouple compute from storage by saving data in Cloud Storage/BigQuery and using ephemeral Dataproc clusters.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated