Diagram
A left-to-right process flow showing producers publishing to a Pub/Sub topic, a Dataflow Beam job filtering and mapping, and BigQuery storing rows for SQL analytics.
Analytics pipeline from messaging through processing to warehouse
The analytics pipeline separates durable messaging, in-flight transformation, and SQL analysis into services that hand data forward in order. Producers publish events once to a Pub/Sub topic or to Google Cloud Managed Service for Apache Kafka, and consumers subscribe independently so ingestion keeps running when downstream jobs restart or fall behind. A Dataflow job built with the Apache Beam software development kits (SDKs) for Java, Python, or Go then reads from the topic, applies filtering and mapping, and writes processed rows to BigQuery for near real-time analytics.
Domain 2 of 4
96 practice questions
Practice this DomainLast updated