A retail company needs to build a data pipeline that currently processes historical sales logs in batch from Cloud Storage, transforms the data, and loads it into BigQuery. In the near future, the company plans to ingest and transform live transactional events in real time from Pub/Sub using the exact same transformation logic and programming framework without rewriting application code. Which Google Cloud service should the company use to implement this unified batch and stream data processing pipeline?

Dataflow: ### What Dataflow is **Dataflow** is a fully managed Google Cloud service designed for large-scale, unified **stream and batch data processing**. Built on the open-source **Apache Beam** framework, Dataflow allows developers to write data processing pipelines in languages such as **Java**, **Python**, or **Go** that execute seamlessly across both bounded (batch) and unbounded (real-time streaming) data sources. ### How it meets the requirements - **Unified programming model (✓):** Dataflow uses the **Apache Beam SDK**, which abstracts the underlying execution differences between batch and streaming pipelines. Developers write the business and transformation logic once and apply it to both static batch datasets and real-time streams. - **Batch to stream evolution (✓):** Moving from batch processing of historical Cloud Storage files to real-time streaming from Pub/Sub requires only switching the source connector without having to redesign or rewrite transformation logic. - **Serverless and autoscaling (✓):** Dataflow dynamically provisions, scales, and tears down worker VM instances based on workload volume, ensuring cost efficiency for variable traffic. - **Reliability and accuracy (✓):** By default, Dataflow provides **exactly-once processing** guarantees for every record, ensuring reliable analytical data loading into BigQuery. ### Key advantages and capabilities - **Open-source portability:** Pipelines written with Apache Beam can run on Dataflow or other execution engines such as Apache Flink or Apache Spark. - **Dynamic work rebalancing:** Optimizes execution in real time by redistributing lagging tasks among worker VMs. - **Pre-built templates:** Provides Google-managed templates for common ingestion workflows without needing to write custom pipeline code from scratch. - **Comprehensive observability:** Integrates natively with Cloud Monitoring and Cloud Logging for job-level metrics, throughput tracking, and troubleshooting. ### Why this is optimal Dataflow is the purpose-built service for building complex, scalable ETL and ELT data pipelines in Google Cloud that require unified batch and streaming capabilities, custom data transformations, and automated resource management.

Dataflow Unified Stream Batch Apache Beam | Unlock the power of your data in the cloud! Get hands-on with Google Cloud's core data services like BigQuery and Looker to validate your practical skills in data ingestion, analysis, and management, and earn your Associate Data Practitioner certification!

StudyCerts

Dataflow Unified Stream Batch Apache Beam | Unlock the power of your data in the cloud! Get hands-on with Google Cloud's core data services like BigQuery and Looker to validate your practical skills in data ingestion, analysis, and management, and earn your Associate Data Practitioner certification! | StudyCerts

Associate Data Practitioner

Dataflow Unified Stream Batch Apache Beam

Related questions

Ask Scout