Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team is developing an Apache Beam streaming pipeline deployed on Google Cloud Dataflow. The pipeline ingests high-throughput user activity records from Pub/Sub, processes and enriches the records using custom transforms, and writes the output dataset to Cloud Spanner.
During testing and transient worker retries, several pipeline workers fail with com.google.cloud.spanner.SpannerException: ALREADY_EXISTS errors, leading to stalled pipeline execution.
Which pipeline mutation logic should the engineering team implement to resolve these write failures?
DoFn with try/catch blocks around Mutation.newInsertBuilder to catch SpannerException and silently discard duplicatesThis approach attempts to wrap the mutation builder logic in a custom DoFn that catches SpannerException errors at runtime and drops the failed elements without throwing an unhandled exception.
Mutation objects are merely instantiated in memory within the DoFn and emitted to the SpannerIO.write() sink transform. The actual database execution and resulting exceptions occur asynchronously inside the SpannerIO write runner, not inside the transform that builds the mutation object.SpannerIO.write() during RPC commit time.You were executing direct, synchronous client RPC calls inside a non-standard custom client sink and routing failed rows to a dead-letter queue, although this is anti-pattern when standard INSERT_OR_UPDATE connectors exist.
Mutation.newInsertBuilder and wrap the write step in a Dataflow GroupByKey transform to guarantee globally unique primary keys across workersThis approach attempts to use standard INSERT mutations (Mutation.newInsertBuilder) while inserting an intermediate GroupByKey (GBK) transform upstream to group duplicate keys and eliminate duplicates within the pipeline processing graph before emitting to the Spanner sink.
GroupByKey can deduplicate records within a single stream bundle or window, it does not prevent Dataflow from replaying bundles across worker failures. If a write partially succeeds and Dataflow retries the bundle, the Spanner sink still receives an INSERT for a row that was already committed in the previous attempt.ALREADY_EXISTS when newInsertBuilder is executed.GroupByKey.This approach would be helpful if the goal was purely to combine or aggregate in-flight streaming records per key within a fixed window prior to processing, assuming database writes were managed through an external transactional deduplication store.
SpannerIO.Write with .withBatching(false) and keep standard INSERT mutations to force unbatched single-record commitsThis approach disables mutation batching in the Dataflow Spanner connector by setting batching to false, causing every mutation to be dispatched individually to Cloud Spanner rather than grouped into larger atomic bundles.
newInsertBuilder, Spanner will still encounter an existing row with the same primary key and throw ALREADY_EXISTS.INSERT non-idempotency completely unaddressed.Disabling batching is sometimes applied in read operations (such as dynamically executing separate query operations across tables), but it is not a solution for avoiding duplicate insertion errors on writes.
INSERT_OR_UPDATE builder (newInsertOrUpdateBuilder) when generating Spanner mutations in the pipelineINSERT_OR_UPDATE Mutation Logic IsIn Apache Beam and Cloud Dataflow pipelines targeting Cloud Spanner, mutation objects represent individual row modifications. The INSERT_OR_UPDATE mutation (Mutation.newInsertOrUpdateBuilder(...) in the Java SDK or SpannerInsertOrUpdate in Python) is an upsert operation. It inserts a new record if the primary key does not exist or overwrites existing column values if a matching primary key is already present.
ALREADY_EXISTS Failures: Cloud Dataflow operates with at-least-once delivery guarantees across distributed workers during retries or bundle replays. When transient network faults or worker restarts occur, bundles of mutations may be re-sent to Cloud Spanner.INSERT_OR_UPDATE guarantees that replaying a batch of mutations succeeds seamlessly rather than throwing a duplicate key constraint violation (ALREADY_EXISTS).Standard INSERT mutations fail immediately upon encountering an existing key. In a distributed processing framework like Dataflow where worker retries and at-least-once mutations are standard behavior, choosing INSERT_OR_UPDATE provides an idempotent sink pattern that aligns with distributed streaming resilience.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated