Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
A financial services organization is designing a real-time event processing architecture using Google Cloud. The architecture continuously ingests transaction streams from Pub/Sub, transforms and enriches the data using Cloud Dataflow, and writes the output to an Apache Kafka cluster for downstream consumer applications.
The system must meet the following requirements:
How should the architect configure the Dataflow streaming pipeline and Kafka integration to meet these requirements?
KafkaIO write transform with withEOS(), configure pipeline drain operations during deployments, and rely on default Kafka broker client timeout settings.This configuration applies the withEOS() method to the KafkaIO write transform but relies on Dataflow pipeline draining for deployments and leaves Kafka broker transaction timeouts at default settings.
withEOS() provides transactional writing during steady state, the deployment and timeout strategy undermines the guarantee.Pipeline updates were never performed while live streaming traffic was active, and Kafka brokers were guaranteed to experience zero network latency variations.
KafkaIO write transform without withEOS(), set transactional.id.expiration.ms to 0, and cancel the running Dataflow job before submitting a new job version.This approach uses default at-least-once Kafka write transformations, immediately expires transactional IDs on the Kafka broker, and tears down jobs abruptly via job cancellation.
KafkaIO transform without withEOS() does not initiate Kafka transactional producers, meaning records will be written multiple times if workers retry failed bundles.transactional.id.expiration.ms to 0 immediately invalidates transactional IDs, preventing producers from completing multi-batch transactions.The pipeline handled stateless, loss-tolerant telemetry logs where message dropping during updates and duplicated records carry zero business impact.
KafkaIO write transform with withEOS(), increase Kafka's transaction.max.timeout.ms to align near the topic retention time, and deploy pipeline modifications using pipeline updates instead of pipeline drains.This architecture configures Cloud Dataflow's KafkaIO sink connector for Exactly-Once Semantics (EOS), optimizes Kafka transaction timeouts, and enforces non-disruptive pipeline deployment practices for streaming data movement.
withEOS() on the Beam KafkaIO transform forces Dataflow to use transactional writes to Kafka, ensuring records are written exactly once downstream.transaction.max.timeout.ms with Kafka topic retention gives ongoing transactions sufficient time to commit after transient network issues or broker reboots, preventing aborted transactions that lead to retried duplicates.gcloud dataflow jobs run ... --update) preserve the execution graph and state without triggering a drain phase, maintaining exactly-once semantics.Default Kafka write connectors only offer at-least-once delivery. Draining a pipeline bypasses commit acknowledgments for late/in-flight offsets, breaking EOS. Combining withEOS(), tuned transaction timeouts, and in-place pipeline updates ensures unbroken end-to-end exactly-once stream processing.
transaction.max.timeout.ms to 1,000 milliseconds, and drain the Dataflow pipeline whenever pipeline updates are deployed.This configuration sets Dataflow to run under at-least-once streaming mode, aggressively lowers Kafka transaction timeouts, and relies on standard pipeline draining during operational updates.
transaction.max.timeout.ms to a very low threshold (1,000 ms) causes transient network hiccups or brief broker failovers to abort in-flight transactions, generating transaction restarts and duplicate records.The project prioritized lowest cost and maximum throughput over message duplication, and the downstream consumers were already designed with idempotent database upserts.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated Validated