Diagram
A process flow of a Dataflow Apache Beam pipeline showing a source feeding PCollections through PTransforms (each containing a DoFn) on parallel autoscaled workers, converging via side inputs or joins, and ending at a sink such as BigQuery or Cloud Storage.
Dataflow for Unified Batch and Streaming Pipelines
Dataflow is a fully managed service for executing data processing pipelines written with the open source Apache Beam framework. Apache Beam provides a unified programming model, so the same pipeline code can process bounded batch data and unbounded streaming data. A pipeline is represented as a directed acyclic graph (DAG) where immutable collections called PCollections flow from a source through PTransforms into a sink, and each PTransform contains processing logic in a DoFn that acts on one data element at a time. The Dataflow Runner for Apache Beam distributes that DAG to parallel workers, brings parallel stages back together with side inputs or joins, and autoscales resources as data volume moves through stages.
Domain 2 of 7
24 practice questions
Practice this DomainLast updated