professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
A data engineering team is developing an Apache Beam streaming pipeline deployed on Google Cloud Dataflow. The pipeline ingests high-throughput user activity records from Pub/Sub, processes and enriches the records using custom transforms, and writes the output dataset to Cloud Spanner.
During testing and transient worker retries, several pipeline workers fail with com.google.cloud.spanner.SpannerException: ALREADY_EXISTS errors, leading to stalled pipeline execution.
Which pipeline mutation logic should the engineering team implement to resolve these write failures?
Use Mutation.newInsertBuilder and wrap the write step in a Dataflow GroupByKey transform to guarantee globally unique primary keys across workers
Apply a custom DoFn with try/catch blocks around Mutation.newInsertBuilder to catch SpannerException and silently discard duplicates
Configure SpannerIO.Write with .withBatching(false) and keep standard INSERT mutations to force unbatched single-record commits
Construct mutations using the INSERT_OR_UPDATE builder (newInsertOrUpdateBuilder) when generating Spanner mutations in the pipeline
Use Mutation.newInsertBuilder and wrap the write step in a Dataflow GroupByKey transform to guarantee globally unique primary keys across workers
Apply a custom DoFn with try/catch blocks around Mutation.newInsertBuilder to catch SpannerException and silently discard duplicates
Configure SpannerIO.Write with .withBatching(false) and keep standard INSERT mutations to force unbatched single-record commits
Construct mutations using the INSERT_OR_UPDATE builder (newInsertOrUpdateBuilder) when generating Spanner mutations in the pipeline
StudyCerts
StudyCertsINSERT_OR_UPDATE Mutation Logic IsIn Apache Beam and Cloud Dataflow pipelines targeting Cloud Spanner, mutation objects represent individual row modifications. The INSERT_OR_UPDATE mutation (Mutation.newInsertOrUpdateBuilder(...) in the Java SDK or SpannerInsertOrUpdate in Python) is an upsert operation. It inserts a new record if the primary key does not exist or overwrites existing column values if a matching primary key is already present.
ALREADY_EXISTS Failures: Cloud Dataflow operates with at-least-once delivery guarantees across distributed workers during retries or bundle replays. When transient network faults or worker restarts occur, bundles of mutations may be re-sent to Cloud Spanner.INSERT_OR_UPDATE guarantees that replaying a batch of mutations succeeds seamlessly rather than throwing a duplicate key constraint violation (ALREADY_EXISTS).Standard INSERT mutations fail immediately upon encountering an existing key. In a distributed processing framework like Dataflow where worker retries and at-least-once mutations are standard behavior, choosing INSERT_OR_UPDATE provides an idempotent sink pattern that aligns with distributed streaming resilience.