Diagram
A five-step left-to-right process flow from existing warehouse to on-premises staging, Cloud Storage bucket, Dataflow Beam transformation, and load into BigQuery.
Dataflow for Custom Batch and Streaming Transformation
A common extraction and load pattern moves data through an ordered sequence:
The pipeline depends on Beam I/O connectors for sources and destinations, including BigQueryIO, BigtableIO, AvroIO, and ParquetIO. Google-provided templates cover both modes, with streaming templates such as Bigtable change streams to BigQuery and batch templates such as Cloud Storage Avro to Bigtable and Cloud Storage Parquet to Bigtable. Choose Dataflow when data must be trimmed, joined, enriched, or reshaped, especially when the same logic must serve both one-time bulk loads and ongoing arrivals.
Domain 2 of 2
145 practice questions
Practice this DomainLast updated