An e-commerce platform requires a real-time event processing architecture on Google Cloud to detect fraudulent transactions and personalize user experiences. The technical architecture must meet the following specifications: Ingest unpredictable volumes of globally published user transaction and clickstream events that may arrive out of order. Perform low-latency key-value lookups for each incoming event against historical customer profile data without overwhelming the lookup store. Calculate windowed aggregations over sliding intervals with low-latency speculative results, while updating results when late data arrives. Store output data in a high-throughput, sub-10ms latency database optimized for real-time application serving. Which combination of Google Cloud services and Dataflow streaming pipeline configurations should you implement?

Ingest events using Cloud Pub/Sub; use Dataflow with the built-in Apache Beam Enrichment transform and BigTableEnrichmentHandler to query customer profile data; apply sliding time windows with allowed lateness and triggers for early and late firings; and write serving results to Cloud Bigtable using BigtableIO.: ### What the Solution Is This architecture combines **Cloud Pub/Sub**, **Cloud Dataflow**, and **Cloud Bigtable** to construct an end-to-end, horizontally scalable, low-latency streaming pipeline that natively manages out-of-order records, real-time enrichment, and windowed analytics. ### How It Meets the Requirements - **Unpredictable ingestion with out-of-order handling**: **Cloud Pub/Sub** provides serverless, global message ingestion that dynamically scales to workload spikes and handles out-of-order event streams seamlessly. - **Low-latency key-value lookups**: The Apache Beam **`Enrichment` transform** combined with the **`BigTableEnrichmentHandler`** performs point lookups against **Cloud Bigtable**. It includes built-in client-side throttling and exponential backoff to avoid overwhelming the database cluster. - **Sliding aggregations and late data processing**: Apache Beam **sliding windows** paired with **allowed lateness** and **triggers** (emitting early speculative panes and late refinement panes) ensure immediate visibility for time-sensitive analytics while properly reconciling late-arriving event timestamps. - **High-throughput, low-latency serving**: Sinking processed aggregates and state directly to **Cloud Bigtable** via **`BigtableIO`** provides sub-10ms read/write response times required by high-concurrency real-time web applications. ### Key Advantages and Capabilities - **Automated Flow Control**: The `Enrichment` transform natively manages throughput and rate limits against Bigtable. - **Event-Time Processing**: Dataflow uses event-time watermarks, allowed lateness, and custom triggers to guarantee accurate sessionization and aggregations. - **Scalable Real-Time Sinks**: `BigtableIO` enables parallelized batch mutations directly into Bigtable tablets. - **Fully Managed Infrastructure**: Minimizes operational overhead through serverless autoscaling across Pub/Sub and Dataflow. ### Why This Is Optimal This architecture completely satisfies the low-latency serving SLA (via Bigtable) and operational simplicity requirements (via Pub/Sub and Dataflow turnkey transforms) while preserving analytical accuracy across out-of-order distributed event streams.

Real Time Streaming Pipeline Pubsub Dataflow Bigtable | professional-cloud-data-engineer

Real Time Streaming Pipeline Pubsub Dataflow Bigtable

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An e-commerce platform requires a real-time event processing architecture on Google Cloud to detect fraudulent transactions and personalize user experiences. The technical architecture must meet the following specifications:

Ingest unpredictable volumes of globally published user transaction and clickstream events that may arrive out of order.
Perform low-latency key-value lookups for each incoming event against historical customer profile data without overwhelming the lookup store.
Calculate windowed aggregations over sliding intervals with low-latency speculative results, while updating results when late data arrives.
Store output data in a high-throughput, sub-10ms latency database optimized for real-time application serving.

Which combination of Google Cloud services and Dataflow streaming pipeline configurations should you implement?

Ingest events using Cloud Pub/Sub; use Dataflow with the built-in Apache Beam Enrichment transform and BigTableEnrichmentHandler to query customer profile data; apply sliding time windows with allowed lateness and triggers for early and late firings; and write serving results to Cloud Bigtable using BigtableIO.

Ingest events using Cloud Datastream; enrich streaming records using Apache Beam side inputs loaded once at pipeline startup from Cloud Storage; apply session windows with speculative triggers; and write serving state to Cloud SQL.

Ingest events using Google Cloud Managed Service for Apache Kafka; use Dataflow to execute federated BigQuery SQL queries per event to enrich records; apply tumbling fixed windows with default triggers discarding late data; and sink results to BigQuery.

Ingest events using Cloud Pub/Sub; execute unthrottled synchronous HTTP point queries in a custom ParDo against Bigtable; apply global windows with repeated triggers; and output micro-batches to Cloud Storage using BigLake Iceberg tables.

Professional Cloud Data Engineer

Real Time Streaming Pipeline Pubsub Dataflow Bigtable

Related questions