Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Ordered Pub/Sub Delivery with Merchant ID Keys
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise e-commerce platform is designing a real-time event ingestion architecture on Google Cloud for order processing and inventory updates. The system must satisfy the following architectural and operational requirements:
- Ordering Guarantees: Ensure strict in-order message delivery per merchant account (
merchant_id). - Operational Overhead: Eliminate cluster provisioning, broker patching, ZooKeeper/KRaft maintenance, and manual partition capacity planning.
- Elastic Scalability: Automatically scale ingestion throughput up or down to handle sudden traffic spikes without manual partition rebalancing.
- Delivery Semantics: Provide built-in retry mechanisms, dead-letter support, and granular message acknowledgment.
Which messaging framework should the enterprise implement?
Explanations
Google Cloud Pub/Sub standard topics without ordering keys, handling event ordering using downstream Dataflow session windows
Google Cloud Pub/Sub standard topics without ordering keys, handling event ordering using downstream Dataflow session windows
What Pub/Sub Without Ordering Keys and Dataflow Windowing Is
This architecture ingests events into a standard Pub/Sub topic where messages are distributed across internal servers without key affinity, leaving the responsibility of sorting and re-sequencing messages to an Apache Beam / Dataflow streaming pipeline using stateful buffering and windowing.
Why It Doesn't Meet the Requirement
- Fails Ingestion Ordering Guarantees: Standard Pub/Sub topics deliver messages out-of-order by default. Offloading ordering to downstream pipelines requires maintaining stateful buffers in Dataflow, which increases end-to-end processing latency and memory footprint.
- Increased Pipeline Complexity: Dataflow must buffer messages within memory/disk state to re-sequence them based on event timestamps, creating operational bottlenecks and failure risks when dealing with late-arriving or stalled records.
- Redundant Complexity: Pub/Sub natively supports ordering keys directly at the broker layer, making custom downstream re-ordering unnecessary.
Key Characteristics and Limitations
- Out-of-Order Delivery: Without ordering keys, subscriber clients receive messages in arbitrary order regardless of publication time.
- Stateful Overhead: Re-ordering streams in Dataflow requires stateful
GroupByKeyor session windowing transforms, adding compute cost.
This Answer Would Be Correct If
This pattern would be suitable if the downstream application only cared about event-time window aggregations (e.g., calculating 5-minute sales totals per merchant) rather than requiring strict sequential per-message transactional processing at ingestion.
A Confluent Cloud managed Kafka cluster with topic replication linked directly to Pub/Sub via Cloud Storage Ingestion
A Confluent Cloud managed Kafka cluster with topic replication linked directly to Pub/Sub via Cloud Storage Ingestion
What Confluent Cloud with Cloud Storage Ingestion Is
This architecture deploys a managed Kafka service (such as Confluent Cloud) and attempts to route messages through intermediary Cloud Storage buckets using Cloud Storage Ingestion into Google Cloud Pub/Sub.
Why It Doesn't Meet the Requirement
- High Architectural Complexity: Introducing multiple intermediary hops (Kafka -> Cloud Storage -> Pub/Sub) introduces significant latency, operational fragility, and dual-billing overhead across two distinct streaming and storage products.
- Ordering Disruption: Staging streaming messages through intermediate Cloud Storage object dumps before Pub/Sub ingestion breaks real-time end-to-end ordering guarantees unless complex batch orchestration is implemented.
- Unnecessary Dual Management: Managing Confluent Cloud topics alongside Pub/Sub topics duplicates schema management, monitoring, and IAM access controls.
Key Characteristics and Limitations
- Increased Latency: Writing events to object storage before ingesting into Pub/Sub transforms a low-latency event stream into a micro-batch file workflow.
- Over-Engineering: Pub/Sub natively supports both direct ingestion and direct Confluent Cloud import topics without needing intermediate object storage staging.
This Answer Would Be Correct If
This pattern would be appropriate if the primary goal was migrating legacy Kafka logs into Cloud Storage for long-term cold analytical archiving, followed by periodic batch replay into Pub/Sub for analytical reprocessing.
A self-managed Apache Kafka cluster deployed on Google Kubernetes Engine (GKE) with topics partitioned by merchant_id
A self-managed Apache Kafka cluster deployed on Google Kubernetes Engine (GKE) with topics partitioned by merchant_id
What Apache Kafka on GKE Is
Apache Kafka on GKE involves deploying distributed Kafka brokers, stateful storage volumes (Persistent Disks), and metadata controllers (KRaft or ZooKeeper) inside a Kubernetes cluster to manage partitioned, append-only log streams.
Why It Doesn't Meet the Requirement
- High Operational Overhead: Self-hosting Kafka requires substantial operational management, including broker capacity sizing, pod autoscaling, persistent volume replication, operating system and broker version patching, and JVM tuning.
- Manual Partition Capacity Planning: Kafka ordering is strictly enforced only within individual partitions. If data volume spikes beyond the partition count, increasing partitions requires manual rebalancing, which can disrupt message ordering during repartitioning.
- Operational Complexity During Spikes: Unlike serverless Pub/Sub, scaling a stateful Kafka cluster horizontally during sudden traffic bursts requires complex broker rebalancing and data redistribution.
Key Characteristics and Limitations
- Static Partitioning Model: Topic throughput is capped by partition count, requiring upfront capacity calculations.
- Stateful Maintenance: Requires specialized operational knowledge to recover from node failures, rebalance topic leader replicas, and manage persistent storage.
- Rebalance Overhead: Consumer group rebalances can introduce processing pauses during dynamic subscriber scaling.
This Answer Would Be Correct If
This approach would be appropriate if the organization required strict sub-millisecond end-to-end latency, long-term immutable log retention on self-managed storage, or compatibility with specialized Kafka Connect ecosystem plugins and custom byte-level protocols.
Google Cloud Pub/Sub with ordering keys configured using the merchant_id
Google Cloud Pub/Sub with ordering keys configured using the merchant_id
What Google Cloud Pub/Sub with Ordering Keys Is
Google Cloud Pub/Sub is a fully managed, serverless, globally distributed messaging service. When configured with ordering keys, Pub/Sub guarantees that messages published with the same key (such as merchant_id) are delivered to subscribers in the exact order they were received by the service, while still scaling horizontally across independent keys without requiring manual partition management.
How It Meets All Requirements
- Ordering Guarantees: Pub/Sub ordering keys ensure strict, FIFO delivery for all messages sharing the same key string (e.g.,
merchant_id). When a subscriber client receives an ordered message, subsequent messages for that key are withheld until the earlier message is acknowledged. - Operational Overhead: Pub/Sub is completely serverless. It eliminates operational tasks such as cluster provisioning, broker sizing, storage rebalancing, replication configuration, and Kafka metadata/KRaft cluster maintenance.
- Elastic Scalability: In standard Apache Kafka architectures, throughput scaling is constrained by the static number of partitions assigned to a topic. Pub/Sub decouples storage and serving dynamically, scaling seamlessly to absorb massive, unpredictable traffic spikes without manual partition re-allocation or partition rebalancing.
- Delivery Semantics: Pub/Sub offers native individual message acknowledgments (
ack), negative acknowledgments (nack), automatic exponential backoff retries, and integrated dead-letter topics.
Key Advantages
- Zero Infrastructure Management: Fully automated infrastructure scaling and automated high-availability across multiple zones.
- Key-Level Concurrency: Allows massive concurrency across thousands of distinct
merchant_idvalues while maintaining strict serialization within each specific merchant stream. - Native Cloud Integration: Seamlessly integrates with Dataflow, Cloud Run, Cloud Functions, and BigQuery without custom connectors.
Why This Is Optimal
Compared to managing Apache Kafka clusters, Pub/Sub delivers enterprise-grade ordering guarantees and elastic scale with zero operational overhead, making it the superior choice for cloud-native ingestion.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
BigQuery Storage Write API Exactly-Once Dead-Letter
Identifying the services (e.g., Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Apache Spark, Hadoop ecosystem, and Apache Kafka)
AdvancedProfessional Data Engineer
Dataproc vs Dataflow vs Data Fusion
Identifying the services (e.g., Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Apache Spark, Hadoop ecosystem, and Apache Kafka)
AdvancedProfessional Data Engineer
BigQuery Nested and Repeated Fields
Designing the data model
AdvancedProfessional Data Engineer
Cloud Spanner INSERT_OR_UPDATE Idempotent Writes
Streaming (e.g., windowing, late arriving data)
AdvancedProfessional Data Engineer
Cloud Storage gRPC Client and Broadcast Hash Join
Processing logic
FundamentalAssociate Data Practitioner
Looker Interactive Dashboard Filters
Create, modify, and share dashboards to answer business questions
Last updated Validated