Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise data engineering team uses Cloud Composer to orchestrate batch pipelines that compute daily sales aggregates and precalculate key business metrics. The precalculated datasets are stored in Cloud Storage and must be loaded into an external relational reporting database accessed by visualization dashboards.
To ensure data integrity, the pipeline must support automated task retries and backfilling historical dates without creating duplicate aggregate records or causing inconsistent metric states in the target database.
How should the team configure the Dataproc Serverless for Apache Spark Cloud Storage to JDBC template to achieve idempotent updates?
ErrorIfExists) and manage pipeline retries with exponential backoffErrorIfExists is the default write mode for the Cloud Storage to JDBC Spark template. Under this setting, Spark throws an exception and fails the job immediately if the target table already exists in the destination database.
This option would be appropriate for a one-time initial table creation migration where existing data in the destination indicates an unexpected naming collision or environment misconfiguration.
MODE) to Ignore and disable dynamic allocationThe Ignore write mode checks whether the destination table already exists; if it does, the write operation is silently bypassed, and no data is written to the destination database.
spark.dynamicAllocation.enabled=false) only affects how Spark requests YARN/executor containers, having no effect on data idempotency or write semantics.This option would be suitable for static reference data or lookup tables that need to be created once if missing, but should never be modified by recurring batch runs.
MODE) to Append and rely on standard JDBC transaction commitsThe Append mode appends the output DataFrame's rows to the existing destination JDBC table without inspecting existing keys or clearing previously written data.
Append inserts duplicate records into the target database.SUM and COUNT) consumed by downstream BI dashboards.This option would be suitable for an append-only immutable audit log or clickstream event pipeline where every batch represents distinct, non-overlapping transactional events that should never overwrite historical records.
MODE) to Overwrite when submitting the template executionThe Overwrite write mode in the Dataproc Serverless for Apache Spark Cloud Storage to JDBC template directs the Spark execution engine to replace the existing target table content or target partition data with the newly computed dataset.
Overwrite ensures that executing the pipeline multiple times for the same time window produces the exact same end state without duplicate records.DELETE or TRUNCATE SQL scripts in Cloud Composer DAGs.Using the built-in Overwrite parameter in the Dataproc Serverless template provides declarative, atomic-like replacement semantics at the Spark execution layer, ensuring reliable idempotent operations during automated retries and backfills.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated