Prepare and test your skills

Prepare and test your skills

A modern pipeline built with Pub/Sub (opens in a new tab) and Dataflow (opens in a new tab) splits data work into two managed stages that connect end to end. Pub/Sub ingests and distributes real-time events from many producers, and Dataflow transforms those events and delivers them to storage and analytics. Publishers send events to Pub/Sub without waiting for processing, and Dataflow reads those events as a source, processes them in stream or batch, and writes results to a destination. This division supports continuous movement from producers to dashboards and warehouses with low operational effort.
What Pub/Sub adds to the combined pipeline is asynchronous, scalable messaging that decouples event producers from event processors. Producers called publishers send events to the Pub/Sub service without regard to how or when the events will be processed, and consumers called subscribers receive them asynchronously. Communication typically happens with latencies on the order of 100 milliseconds, which makes Pub/Sub suitable for streaming analytics and data integration pipelines that must gather events from many clients at once. Pub/Sub increases flexibility and robustness because publishers broadcast events instead of making synchronous remote procedure calls (RPCs) where publishers must wait for subscribers.
Pub/Sub also uses per-message parallelism to maximize throughput and independence. Pub/Sub leases individual messages to subscriber clients and tracks whether each message is successfully processed. This design lets many subscribers work in parallel and lets publishers and subscribers scale and recover without blocking each other.
What Dataflow adds to the combined pipeline is unified stream and batch processing that transforms and delivers data at scale. A Dataflow pipeline reads from one or more sources, transforms the data, and writes the data to a destination using the same programming model for batch and streaming. Dataflow is built on the open source Apache Beam project, which provides windowing and session-analysis primitives and an ecosystem of source and sink connectors, including a connector for BigQuery. By default, Dataflow provides exactly-once processing of every record, while streaming pipelines that can tolerate duplicates can enable at-least-once mode to reduce cost and improve latency.
Dataflow handles fluctuating volumes of real-time data with low latency. Typical uses include data movement across subsystems, extract-transform-load (ETL) into a warehouse such as BigQuery, support for business intelligence (BI) dashboards, real-time machine learning (ML) analysis, and large-scale sensor or log processing.
How data moves end to end from Pub/Sub through Dataflow is a fixed order from producers through ingestion and transformation to an analytics destination. Pub/Sub acts as the entry point that buffers events, and Dataflow acts as the stream processor that connects that entry point to databases and storage:
A four-step process flow showing publishers sending events to a Pub/Sub topic, Dataflow reading and transforming those messages with Apache Beam, and writing results to BigQuery, Bigtable, or Cloud Storage for dashboards, queries, and ML.
Dataflow templates for moving data from Pub/Sub to Cloud Storage, BigQuery, and other products are available in the Pub/Sub and Dataflow UIs in the Google Cloud console. This template path lets teams reuse a predefined pipeline for common ingestion work without writing Apache Beam code.
Why the combined pipeline is more resilient is that ingestion, processing, and storage can scale and recover independently. Pub/Sub decouples publishers from subscribers, so a slow consumer or a failed zone does not force producers to stop or wait. For example, service instances in multiple zones can subscribe to a common topic, and if the service fails in one zone, the others can pick up the load automatically. Dataflow adds processing resilience by distributing work in parallel across multiple worker virtual machines (VMs).
Dataflow can autoscale by provisioning extra worker VMs when volume rises or shutting some down when fewer are needed, and it can dynamically rebalance work among VMs so parallel work completes more efficiently. Together, Pub/Sub buffering plus Dataflow parallel processing and autoscaling let the pipeline absorb spikes, continue delivering events, and maintain reliable processing without manual resizing.
Exam tip: Pub/Sub ingests and distributes messages but does not transform them; Dataflow reads from Pub/Sub and performs the transform and delivery to BigQuery, Bigtable, or Cloud Storage.
How the combined pipeline lowers operational work is that both services are fully managed and serverless for the pipeline owner. When a Dataflow job runs, the Dataflow service allocates a pool of worker VMs to execute the pipeline, and when the job completes or is cancelled, Dataflow automatically deletes the VMs. Performance, scaling, availability, security, and compliance are handled automatically, so teams focus on pipeline logic instead of managing server clusters. Operational visibility comes from the Dataflow monitoring interface in the Google Cloud console, which shows a graphical representation of the pipeline with progress and execution details for each stage.
Development overhead is also lower because Dataflow supports multiple ways to build and run the same kind of pipeline:
The result is unified data movement where ingestion, transformation, and delivery stay connected without custom servers, custom scaling logic, or custom monitoring tooling.
How the combined pipeline supports real-time analytics is by keeping data flowing continuously from producers to queryable storage. Pub/Sub gathers raw or processed events from many sources at once and supports patterns such as real-time event distribution, database change-event replication, and an enterprise event bus. Dataflow then processes that continuous stream with low latency and writes up-to-date results for BI dashboards, ETL into BigQuery, and real-time ML analysis. Because Dataflow uses the same model for stream and batch, a solution can start with simple data movement and grow toward advanced streaming analytics without rebuilding the pipeline.
The two services have distinct jobs and scaling methods in this flow:
| Service | Primary job in the pipeline | How it handles scale |
|---|---|---|
| Pub/Sub | Ingest events from many publishers and deliver them to subscribers | Horizontally scalable messaging with per-message leasing to parallel subscribers |
| Dataflow | Transform, enrich, and move stream and batch data to destinations | Parallel processing across worker VMs with autoscaling and dynamic work rebalancing |
Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that modernizes data ingestion by decoupling event producers from event processors. Publishers send data to the Pub/Sub service without waiting for processing, and the service then delivers those events to subscribers with latency typically on the order of 100 milliseconds. This design lets pipelines gather events from many sources at once and move them reliably into streaming analytics and data integration workflows.
What Pub/Sub is is a global messaging service for service-to-service communication in event-driven architectures. The Pub/Sub service receives events once from producers and delivers them to all services that need to react, rather than requiring direct calls between systems. It is used to load and distribute data for streaming analytics, to integrate services, and to parallelize tasks across workers. Because capacity is managed automatically, pipelines can grow to handle fluctuating real-time volumes without manual provisioning.
How publishers, topics, and subscribers work together is a decoupled flow from producer to Pub/Sub topic to subscriber. A publisher is an event producer that sends a message to a predefined topic in the Pub/Sub service, without regard to how or when the event will be processed. The Pub/Sub service holds the message on the topic and then delivers it to subscriber services that react to that topic. This flow is different from synchronous remote procedure calls (RPCs) where the publisher must wait for the subscriber to receive the data.
In an RPC system, the sending and receiving services must both be available at the same time, so a slow or failed receiver blocks the sender. In Pub/Sub, asynchronous integration means the publisher finishes its work after publishing, and subscribers process messages independently when they are ready. This separation increases flexibility because new subscribers can be added without changing publishers, and it increases robustness because one slow subscriber does not stop ingestion from other sources.
Why decoupling makes Pub/Sub ingestion scalable and reliable is that senders and receivers scale and fail independently. Many publishers can forward user interactions, server events, database changes, and sensor data to the same topic at the same time, and Pub/Sub gathers those events from many clients simultaneously. Multiple subscriber instances can share the load from a common topic, so if instances in one zone fail, instances in other zones can automatically pick up the work. The result is a real-time pipeline entry point that stays available during spikes and partial failures.
Pub/Sub also uses per-message parallelism instead of partition-based messaging to maximize subscriber throughput. Pub/Sub leases individual messages to subscriber clients and tracks whether each message is successfully processed. By contrast, partition-based systems force subscribers to process messages in each partition in order and limit the number of concurrent clients to the number of partitions.
Side-by-side comparison showing Pub/Sub leasing individual messages for parallel independent processing versus partition-based systems requiring ordered processing limited by partition count.
| Approach | Parallelism unit | Processing constraint |
|---|---|---|
| Pub/Sub | Individual message leased to a subscriber client | Messages can be processed in parallel across many subscribers, with success tracked per message |
| Partition-based messaging | Partition assigned to a subscriber | Messages in each partition must be processed in order, and concurrent clients are limited by partition count |
Common patterns for real-time ingestion and distribution all start with events flowing into Pub/Sub and then flowing out to processing, storage, or other applications. The service acts as an enterprise event bus that makes raw or processed events available to multiple teams for real-time use:
How Pub/Sub connects ingestion to processing and storage is through Dataflow as the primary stream-processing path into analytics. Data flows from a Pub/Sub topic into Dataflow for transformation and integration, and then Dataflow writes the results to BigQuery and data lakes on Cloud Storage. Ready-made Dataflow templates for Pub/Sub to Cloud Storage, BigQuery, and other products are available in the Google Cloud console, and integration with Apache Spark and Data Fusion supports additional pipeline styles. This connection turns Pub/Sub from a message receiver into a unified ingestion front end for batch and streaming destinations.
Pub/Sub also connects ingestion to automation, orchestration, and governance through built-in integrations. Messages can be delivered by push as HTTP POST requests to webhooks to trigger Cloud Functions and other serverless products for workflow automation. Pub/Sub can be integrated declaratively into multistep serverless Workflows and into analytic orchestration with Managed Airflow through Pub/Sub triggers. Authentication uses standard OAuth with granular Identity and Access Management (IAM) controls for individual resources, while monitoring, alerting, and logging products observe pipeline health.
Exam tip: Pub/Sub is intended for service-to-service communication, not direct communication with mobile, web, or IoT clients; use Firebase Realtime Database or Firebase Cloud Messaging for client-server messaging and Cloud Tasks for asynchronous service calls.
Dataflow (opens in a new tab) is a fully managed Google Cloud service for unified stream and batch data processing at scale. A Dataflow pipeline reads data from one or more sources, transforms and enriches the data, and writes the results to a destination. The same programming model supports both batch analytics on historical data and streaming analytics with low latency on fluctuating volumes of real-time data.
Unified stream and batch processing with Apache Beam is the foundation that lets Dataflow run one programming model for real-time and historical data. Apache Beam is the open source framework that Dataflow runs at scale to unify batch and streaming work. Apache Beam provides windowing and session-analysis primitives plus an ecosystem of source and sink connectors, including a connector for BigQuery, so developers transform and enrich data in stream and batch modes with equal reliability and expressiveness. Dataflow executes Apache Beam pipelines written with Java, Python, and Go SDKs, including multi-language pipelines, and the same pipeline code can later run on other runners such as Apache Flink or Apache Spark without rewriting.
This portability means a solution can start with simple data movement and grow toward real-time streaming analytics without changing frameworks. Teams can reuse the same transform logic as needs shift from historical analysis to continuous processing.
Automated scaling and managed pipeline operations in Dataflow remove cluster operations through a serverless approach. When a Dataflow job runs, the Dataflow service allocates a pool of worker VMs to execute the pipeline, and when the job completes or is cancelled, Dataflow automatically deletes those VMs. Data is processed in parallel across multiple VMs, and Dataflow can autoscale by provisioning extra worker VMs when demand rises or shutting down VMs when fewer are needed. Dataflow also optimizes execution based on pipeline characteristics, including dynamically rebalancing work among VMs so parallel work completes more efficiently.
Performance, scaling, availability, security, and compliance are handled automatically, so teams focus on programming, and billing is based on the compute resources the job uses. Dataflow pipelines provide different processing guarantees for streaming data:
| Mode | Guarantee | Choose when |
|---|---|---|
| Exactly-once processing | Every record is processed exactly once by default | Correctness matters and duplicates cannot be tolerated |
| At-least-once mode | Records may be processed more than once | The streaming pipeline can tolerate duplicates to reduce cost and improve latency |
Exam tip: Dataflow uses exactly-once processing by default; enable at-least-once mode only when the pipeline can tolerate duplicates.
The common Dataflow pipeline flow is an ordered movement from source through transform to destination:
This same flow supports many large-scale patterns:
Flexible development and observable jobs in Dataflow support both code-first teams and non-programmers. Developers can write code using the Apache Beam SDKs, submit jobs through the command-line interface, or develop and run pipelines iteratively in JupyterLab notebooks. Teams can also deploy a Dataflow template, which is a predefined pipeline that can be run on demand, and Google provides a library of templates for common scenarios that can be deployed without knowing Apache Beam programming. For example, a developer can create a template once and a data scientist can deploy it when needed.
Dataflow jobs are observable through the Dataflow monitoring interface in the Google Cloud console, which shows a graphical representation of the pipeline with progress and execution details for each pipeline stage. That visibility helps teams follow data from source to destination and understand performance stage by stage.
Pub/Sub ingests and distributes real-time events from publishers to a topic without waiting for processing, and Dataflow reads those messages as a streaming source, transforms and enriches them, and writes results to destinations such as BigQuery, Bigtable, and Cloud Storage. The same Dataflow logic supports stream and batch processing with low operational effort because both services are fully managed and scale independently.
Pub/Sub provides asynchronous, scalable messaging that decouples event producers from processors and buffers events with latency typically on the order of 100 milliseconds, but it does not transform data. Dataflow provides unified stream and batch processing that transforms, enriches, and delivers data at scale using the Apache Beam programming model.
Dataflow processes data in parallel across multiple worker virtual machines (VMs), autoscales by adding worker virtual machines when volume rises or shutting some down when fewer are needed, and dynamically rebalances work among them. Dataflow provides exactly-once processing of every record by default, while streaming pipelines that can tolerate duplicates can enable at-least-once mode to reduce cost and improve latency.
A retail organization is modernizing its data analytics architecture on Google Cloud. The data engineering team needs to transform and enrich both historical sales records and real-time clickstream events ingested from Pub/Sub before loading the results into BigQuery. The team has established the following requirements:
How does Dataflow work with Apache Beam to meet these requirements?
It requires developers to manually provision and delete worker VMs while using Apache Beam exclusively to monitor pipeline bottlenecks
It locks Apache Beam pipeline code to Google Cloud infrastructure while requiring predefined templates for all custom stream transformations
It executes unified Apache Beam pipelines for both data types while automatically provisioning, autoscaling, and rebalancing worker VMs
It uses Flexible Resource Scheduling (FlexRS) to guarantee low-latency streaming execution while requiring separate codebases for batch jobs
Last updated