Diagram
A process flow showing event producers publishing to a Pub/Sub topic, Dataflow processing the events through three stages (source, transformation, sink), and results written to BigQuery and Cloud Storage, with Dataplex providing governance over the pipeline's data.
Data Pipelines with Pub/Sub, Dataflow, Dataproc, and Dataplex
Data pipelines move raw events through ingestion, transformation, and governance before analysis. Pub/Sub ingests streaming events, Dataflow runs unified batch and streaming pipelines, Dataproc runs managed Spark and Hadoop clusters, and Dataplex governs data across systems. The services compose in order: Pub/Sub captures events, Dataflow transforms them, and Dataplex tracks ownership and quality.
A publisher sends a message to a Pub/Sub topic, and each subscription receives its own copy so one slow consumer does not block others. A Dataflow pipeline written with Apache Beam then flows through three stages in order: first source reads from Pub/Sub or Cloud Storage, then transformation filters or aggregates, then sink writes to BigQuery or Cloud Storage. Dataproc fits existing Spark jobs that should move without rewriting, while Dataplex organizes lakes, zones, and assets and applies metadata and quality checks without moving data itself.
Domain 2 of 3
166 practice questions
Practice this DomainLast updated