Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise analytics team manages batch Spark workloads for multiple internal business units. The workloads have the following operational requirements and constraints:
Which architecture and orchestration strategy should the team implement?
Cloud Composer acts as an orchestration engine using Apache Airflow Directed Acyclic Graphs (DAGs) to programmatically manage the lifecycle of ephemeral (job-scoped) Dataproc clusters. In this pattern, the cluster is dynamically provisioned just before job execution and deleted immediately following completion, while all persistent input and output data resides externally in Cloud Storage.
--image-version) do not conflict across different teams.DataprocCreateClusterOperator, DataprocSubmitJobOperator, DataprocDeleteClusterOperator) fully automate workflow execution.For intermittent, scheduled batch workloads with heterogeneous dependencies and multi-tenant billing requirements, ephemeral Dataproc clusters orchestrated via Cloud Composer offer the best balance of cost efficiency, fault tolerance, and security boundary isolation.
Initialization actions on Dataproc are scripts executed during initial cluster creation across master and worker nodes to install software packages and configure environment settings.
Initialization actions were used strictly during the automated creation step of dedicated ephemeral clusters sourced from Cloud Storage.
Cloud Composer workers run inside a Google Kubernetes Engine (GKE) cluster and are designed strictly to execute Airflow orchestration tasks, DAG parsing, operators, sensors, and API call triggers.
The pipeline tasks only performed lightweight metadata verification, basic REST API triggers, or orchestration scheduling without intensive data compute requirements.
A persistent Dataproc cluster is a long-running, centralized cluster shared by multiple users and workloads. YARN's FairScheduler allocates cluster resources among concurrently running applications so that all jobs get an equal share over time.
The workload consisted of continuous, low-latency micro-batches or ad-hoc interactive analytical queries where the 90–120 second cluster startup time of ephemeral clusters is unacceptable.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated