Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An ecommerce company runs an Apache Beam streaming pipeline on Google Cloud Dataflow to track product inventory updates from high-throughput retail transactions. The pipeline performs stateful aggregations using Combine.perKey within fixed temporal windows configured with early and late triggers to handle incremental updates and late-arriving data.
The resulting inventory totals must be written to Cloud Spanner. Because streaming Dataflow pipelines retry failed worker tasks indefinitely, the write mechanism must be fully idempotent, prevent primary key collision errors during retries or multiple trigger firings for the same window key, and maintain exactly-once sink semantics without pipeline stalls.
Which approach should you implement to write the aggregated inventory totals to Cloud Spanner?
INSERT mutations for the aggregated records and commit them using SpannerIO.write().INSERT with SpannerIO.write() DoesAn INSERT mutation instructs Cloud Spanner to write a new row with the specified primary key. If a record with the same primary key already exists in the destination table, Cloud Spanner rejects the operation and raises an ALREADY_EXISTS error.
INSERT mutations, the subsequent retry attempt will resubmit the same primary keys, triggering ALREADY_EXISTS errors and causing the streaming pipeline to stall permanently.INSERT mutation will fail on the second trigger firing because the primary key was written during the first firing.The pipeline was operating in an append-only append log architecture where every emitted event receives a globally unique UUID as its primary key and is never updated or re-emitted by triggers.
INSERT_OR_UPDATE mutations for the aggregated records and commit them using SpannerIO.write().INSERT_OR_UPDATE with SpannerIO.write() IsIn Apache Beam and Google Cloud Dataflow, writing to Cloud Spanner is typically handled via the native SpannerIO.write() transform using mutation objects. The INSERT_OR_UPDATE (or upsert) mutation type instructs Cloud Spanner to insert a new row if the primary key does not exist or update the existing row's column values if the record is already present.
INSERT_OR_UPDATE mutations can be applied multiple times without throwing unique key constraint or duplicate record exceptions, re-executing a bundle following a transient network glitch or worker crash is completely safe.INSERT_OR_UPDATE updates the stored state in Spanner in-place with the latest accumulated values without failing on duplicate keys.ALREADY_EXISTS mutation errors that cause streaming pipelines to crash or stall in infinite retry loops.SpannerIO.write() automatically batches incoming mutations to maximize throughput across Spanner splits.Using INSERT_OR_UPDATE mutations natively within SpannerIO.write() provides the highest write throughput and the required idempotency. It cleanly accommodates the non-deterministic arrival of late data and Dataflow worker retries without requiring complex external locking or manual duplicate checking.
SpannerIO.write() configured with explicit DELETE followed by INSERT mutations in separate sequential pipeline branches.DELETE Followed by INSERT in Separate Branches DoesThis approach splits the write pipeline into two parallel branches: one branch generates and applies DELETE mutations to remove existing window keys, and a secondary branch subsequently generates and applies INSERT mutations to write the latest accumulated values.
DELETE mutation will execute before the INSERT mutation in Cloud Spanner, leading to non-deterministic state and dropped data.SpannerIO.write() commits mutations in independent batches. If a worker fails between the delete and insert stages, the deleted data is lost while the new state is not committed.INSERT branch executes before the DELETE branch or retries independently, duplicate key exceptions will cause the streaming worker to fail.SpannerIO.write() stages cannot share an atomic transaction boundary.You were implementing a batch staging table rotation where an entire table or partition is dropped and reloaded atomically outside of live streaming ingestion.
ParDo transform that executes direct DML INSERT statements using the Cloud Spanner JDBC driver inside transactional boundaries.ParDo DoesThis pattern uses a custom Apache Beam DoFn to open connections via standard JDBC drivers and issue raw SQL DML statements (such as INSERT INTO inventory ...) wrapped inside manually managed database transactions.
ParDo bypasses the highly optimized mutation batching and session pooling provided by the native SpannerIO connector, severely bottlenecking high-throughput streaming pipelines.INSERT queries will fail with primary key constraint violations when processing late trigger updates or handling streaming task retries.ProcessElement method forces Dataflow to retry the bundle indefinitely, halting pipeline progress and causing processing watermark lag.SpannerIO's native split-aware mutation grouping and backpressure handling.You were migrating a legacy batch ETL workflow designed for an external relational database that strictly relies on complex multi-table triggers and stored procedures unsupported by mutation APIs.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated