Selecting Dataflow, BigQuery, Pub/Sub and Kafka

Selecting a core batch and stream processing service means matching the execution engine to the shape of the work. Cloud Dataflow (opens in a new tab) is the serverless engine for unified batch and stream pipelines written with Apache Beam (Beam). Cloud Dataproc and Cloud Data Fusion cover the other shapes: Dataproc-related Spark services run legacy cluster and Hadoop-ecosystem workloads, while Data Fusion provides visual data integration. Dataflow handles provisioning, scaling, and fault tolerance so developers focus on pipeline logic.

1.1Unified Batch and Streaming Pipelines in Cloud Dataflow

Cloud Dataflow (opens in a new tab) pipelines use a staged model where data flows from a source through transforms to a destination. A pipeline reads data from a source, applies transforms and aggregations, and writes results to a destination such as BigQuery. The same Beam model handles both bounded batch data and unbounded streaming data with windowing, grouping, and joining for streaming state.

Pipelines cover four common shapes:

• Move data as-is to a destination for basic replication.
• Transform data to make it usable by the target system.
• Aggregate, process, and enrich data for analysis, including machine learning (ML) enrichment and inference.
• Join data with other data for unified views.

1.2Runner Execution and ETL Flow from Pub/Sub to BigQuery

Runner-based execution in Beam separates what to compute from how and where it runs. Apache Beam supports multiple runners, and the Dataflow runner runs a Beam pipeline on the Dataflow service. When the Dataflow runner is selected, the runner uploads executable code and dependencies to a Cloud Storage bucket and creates a Dataflow job, then Dataflow allocates a pool of virtual machines (VMs) to execute the pipeline stages.

A typical extract, transform, load (ETL) and business intelligence (BI) solution shows the flow across services:

1. Ingest: Pub/Sub ingests data from an external system into a topic.
2. Process: Dataflow reads the data from Pub/Sub, transforms or aggregates it, and writes it to BigQuery.
3. Warehouse: BigQuery acts as a data warehouse where analysts run ad hoc queries.
4. Visualize: Looker provides real-time BI insights from the data stored in BigQuery.

For basic movement, a Google-provided template can run the pipeline without custom Beam code, and some templates support JavaScript user-defined functions (UDFs) for light custom logic. For complex logic, the pipeline is built with the Beam software development kits (SDKs) for Java, Python, or Go.

1.3Reusable Templates and Recurring Schedules

Dataflow templates make pipelines reusable and separable from the people who run them. A template lets each job change specific pipeline parameters, lets anyone with permissions deploy the pipeline later, and provides open-source Apache License 2.0 code that can seed new pipelines. Dataflow (opens in a new tab) has no built-in cross-pipeline communication mechanism, so data flows from one pipeline instance to durable storage or a cache and the next pipeline reads it back, using Cloud Storage for durable sharing or App Engine for in-memory caching.

Recurrent execution is handled outside the single job definition:

• Dataflow data pipelines for recurrent job schedules and tracking resource use across executions.
• Cloud Scheduler for time-based triggering.
• Managed Service for Apache Airflow with the Dataflow Operator for workflow orchestration.
• Custom cron processes on Compute Engine for self-managed schedules.

Choose Dataflow data pipelines when the need is repeated Dataflow schedules with multi-run cost visibility, and choose Airflow when Dataflow is one step in a larger directed acyclic graph (DAG) workflow.

1.4Monitoring, Acceleration, and High-Volume Sinks

The Dataflow monitoring interface in the Google Cloud console makes pipeline jobs observable during execution. The interface shows a graphical representation of the pipeline with progress and execution details for each stage, which helps spot bottlenecks and high latency, and job profiling exposes central processing unit (CPU) usage and memory allocation. Beam metrics (opens in a new tab) can be exported as Cloud Monitoring custom metrics, such as counters for malformed records versus empty payloads, to drive dashboards and alerting policies.

Performance and scale depend on matching resources to pipeline steps and destinations. Right fitting lets different pipeline steps mix CPUs and graphics processing units (GPUs) to target costly steps without overprovisioning the whole job, while accelerators including GPUs and tensor processing units (TPUs) speed machine learning and image-processing tasks. After processing, results flow from Dataflow workers to high-capacity sinks that absorb large streams, including Bigtable, BigQuery, and Pub/Sub, where a large Bigtable cluster can scale horizontally to about one million messages per second.

1.5Exactly-Once Processing and Robust Beam Code

Robust Beam code is built around ParDo and DoFn, where ParDo applies parallel processing and DoFn defines the distributed per-element function. A DoFn should behave as a pure function that does not depend on hidden or external state, has no observable side effects, and is deterministic, because many instances run on different machines with no knowledge of each other. Dataflow (opens in a new tab) provides exactly-once processing so each element in an input PCollection is processed exactly once from the user's view, but the service does not guarantee how many times a DoFn is invoked, how elements are grouped, how many DoFn instances are created, or whether retries and backup copies occur.

That fault-tolerant behavior sets strict coding rules. The service serializes element processing per DoFn instance and may retry code after worker issues, so manual side effects such as temporary files with non-unique names can break. Dead-letter queues isolate bad records by routing parse failures to a separate output while valid records continue, and batching with GroupByKey, concurrency limits on external calls, and controlled parallelism for slow or external operations prevent overload and fused-step slowdowns.

BigQuery (opens in a new tab) is the analytical warehouse that stores historic, large-scale data for analytics and decision-making. Processing engines such as Dataflow, Cloud Data Fusion, and Datastream move and transform data into BigQuery tables. Data flows from sources such as Pub/Sub topics, Cloud Storage buckets, or operational databases through batch or streaming pipelines into BigQuery, where SQL-based transformation and query serving deliver data to dashboards, machine learning, and downstream applications.

2.1BigQuery Warehouse Foundations and Location Choice

BigQuery (opens in a new tab) datasets anchor the warehouse design because they combine an analytical database with analytical components and procedures for serving insights. Warehouses commonly grow to terabytes to petabytes, so recovery time and cost depend on how much data must be restored to meet recovery point and recovery time objectives. Many warehouses are append-only with mostly static historic data, which allows a strategy that backs up and restores only the newly appended data instead of the full warehouse.

Location choice defines the failure domain for a BigQuery dataset. A dataset stores data in either a regional location such as Iowa (us-central1) or a multi-region location such as US or EU that spans a large geographic area. A multi-region location does not provide cross-region replication nor regional redundancy because data is still stored in a single region within that geography, so machine, zonal, or regional failures affect availability differently and must be matched to locality rules.

2.2Ingestion Pathways from Batch and Streaming Engines

Ingestion pathways connect processing engines to BigQuery for both batch history and continuous streams. Dataflow (opens in a new tab) is a fully managed service for running Beam pipelines that transform and enrich data in both stream and batch modes with windowing and session-analysis primitives and a connector for BigQuery. Cloud Data Fusion, built on the open source CDAP project, is a fully managed data integration service where pipelines are built graphically as a DAG of sources, transformations, aggregates, sinks, and other nodes, and the planner converts that DAG into parallel computations executed as an Apache Spark job on Managed Service for Apache Spark.

Datastream provides change data capture (CDC) replication from operational sources into Google Cloud destinations. Datastream writes change events directly to a BigQuery dataset through the BigQuery Write API, where changes are merged continuously with existing data as UPSERTs. Alternatively, Datastream writes change files to Cloud Storage as a staging destination, and then Dataflow templates built for Datastream-generated files read data from Cloud Storage and load it into BigQuery, Cloud SQL for PostgreSQL, or Spanner to maintain up-to-date replicated tables.

Batch orchestration often depends on Managed Airflow and ephemeral compute. In a typical batch pattern, Managed Airflow validates that new files arrived in the Cloud Storage ingestion bucket, then manages an ephemeral Managed Service for Apache Spark cluster that runs an ETL job and is deleted when processing ends. The ordered workflow is:

1. Validate arrival: Managed Airflow checks the Cloud Storage ingestion bucket for new files defined in DAG files stored in a separate Cloud Storage bucket.
2. Start cluster: Managed Airflow starts an ephemeral Managed Service for Apache Spark cluster that exists only for the processing stage.
3. Run Spark job: The Apache Spark job performs ETL on the ingested data to produce business metrics and writes processed output to the data lake or warehouse.
4. Delete cluster: Managed Airflow deletes the cluster at the end of the job, so only data and metadata persist.

Because batch pipelines can be recreated and restarted, disaster recovery focuses on recovering the ingested data in Cloud Storage, the processed data in the lake, and metadata such as databases, tables, columns, and partitions stored in Dataproc Metastore. For streaming, Pub/Sub subscriptions buffer delayed data when pipelines stop, and processing resumes from the backlog when pipelines run again, which lowers recovery point objective.

2.3Reading and Writing BigQuery from Dataflow Pipelines

Dataflow pipelines both read from and write to BigQuery, and the chosen method determines quota use and latency. Beam supports BigQuery I/O in Java, Python, and Go, with BigQueryIO in Java exposing two read methods that consume different quotas. Table export is the default read method, where BigQueryIO invokes a BigQuery export request that writes table data to a temporary Cloud Storage location and then BigQueryIO reads data from that temporary location. Direct read instead uses the BigQuery Storage API to read table rows directly from BigQuery over gRPC with high throughput, which removes the export step.

Writing to BigQuery also splits by pipeline type. Batch pipelines that use BigQuery load jobs consume load-job quotas at the table and project level, while streaming pipelines that use streaming inserts consume streaming-insert quotas. Avoid using load jobs to append data thousands of times per day into one table; use a streaming pipeline with streaming inserts or the Storage Write API for near real-time appends. Dataflow (opens in a new tab) parallelizes work across worker VMs and can autoscale and dynamically rebalance work, but excessive calls to external services such as BigQuery or Pub/Sub can overwhelm quotas, so batching requests and limiting concurrent calls protects downstream systems.

2.4SQL Transformation and Query Execution in BigQuery

SQL-based transformation in BigQuery turns stored tables into curated results through query jobs submitted from the console, the bq query command, the REST API, or client libraries. Multi-statement queries run multiple statements in sequence with shared state and support procedural language statements for variables and control flow, which are often used in stored procedures. Saved queries can be private to one user, shared at the project level with specific principals, or public, which supports reuse without copying data.

When BigQuery runs a query, the query is broken into an execution tree of stages that contain steps able to run in parallel. Stages communicate through a fast distributed shuffle tier that stores intermediate data from stage workers and uses high-bandwidth networking and memory to move data quickly to worker nodes. When all planning information is ready, BigQuery generates a query plan that can be viewed graphically with performance insights, and additional workers monitor progress and dynamically adjust the plan as stage results arrive. When the query completes, BigQuery writes results to persistent storage and returns them, which lets the next identical query be served from cached results.

Concurrency and cost controls shape how queries share the warehouse. BigQuery runs many queries in parallel and can queue queries to run when resources are available, then redistributes resources fairly between new and running queries so performance depends on concurrent load rather than submission order. The query plan, INFORMATION_SCHEMA.JOBS views, and the jobs.get method expose stage output sizes that reveal optimization chances, such as filtering earlier when one stage writes far more output than others. BigQuery (opens in a new tab) offers on-demand pricing based on bytes scanned with fixed per-project processing capacity and capacity-based pricing based on purchased dedicated capacity with reservations, plus project-level quotas and custom cost controls.

2.5Storage Strategies with BigLake and Streaming Delivery

Storage strategy decides whether data lives in BigQuery storage or in external Cloud Storage accessed as BigLake tables. Processed lake data is commonly stored in Cloud Storage in columnar formats such as Parquet and ORC because those formats store efficiently and serve analytical workloads faster than JSON or CSV, and formats that support parallelizable reads such as Avro outperform CSV files with embedded newlines or compressed files. A Hive-partitioned layout lets queries prune partitions with partitioning columns, while file count and file size must be designed for preferred query performance.

If BigLake queries cannot meet service-level agreement requirements and cannot be tuned, convert the exposed data to BigQuery storage and redefine authorized views to use the BigQuery tables. This change generally does not disrupt consumers or require query rewrites, while enabling optimizations unavailable to BigLake tables such as materialized views with different partitioning and clustering than base tables and use of BigQuery BI Engine. For direct programmatic access, the BigQuery Storage Read API can read BigQuery tables and BigLake tables with filtering and fine-grained access control for stable analytics or machine learning consumers, while the Cloud Storage JSON or XML APIs expose buckets directly when consumers cannot use SQL or need formats unsupported by the Storage Read API. Direct bucket access lacks filtering and fine-grained control and creates copies that increase storage cost and lineage overhead, so the BigQuery API is preferred for analytical applications whenever possible.

Streaming delivery extends the same storage with live topics. An ingest pipeline reads raw events, enriches them, saves curated data to the BigQuery base table, and at the same time publishes enriched events to a dedicated Pub/Sub topic where each subscriber consumes independently and can filter for relevant events. The pipeline can also aggregate event statistics and publish them to a separate topic for another consumer, which supports continuously updated dashboards, database replication for low-downtime migration and hybrid-cloud configurations, and event-driven architectures that consume event data from BigQuery and Cloud Storage.

Messaging and ingestion frameworks decouple data producers from data consumers so streaming events can be ingested, enriched, and fanned out without direct connections. Google Cloud Pub/Sub (Pub/Sub) (opens in a new tab) provides serverless asynchronous messaging for server events, user interactions, parallel processing, and IoT streaming, while Apache Kafka (Kafka) provides a broker-based log that Cloud Dataflow (Dataflow) reads through the Beam KafkaIO connector. Eventarc is distinct from both because it manages the flow of state changes for event-driven actions such as error remediation and resource labeling rather than high-volume data ingestion.

3.1Pub/Sub Enrich-and-Fan-Out Ingestion Patterns

Pub/Sub ingestion commonly follows an enrich-and-fan-out pattern where a streaming pipeline reads raw events, enriches them, and saves curated rows to a BigQuery base table. The same pipeline publishes the enriched events to a dedicated topic consumed by multiple subscribers that filter for only relevant events, and it publishes aggregate statistics to its own topic for another consumer. Data flows from producers to topics to subscriptions to consumers, so one publish can serve many independent subscribers without the producer knowing each consumer.

Enriched Pub/Sub subscriptions support several streaming consumer patterns:

• Enriched events that combine full context, such as customer profile information with a customer order.
• Close-to-real-time aggregation notifications, such as total order statistics for the last 15 minutes.
• Business-level alerts, such as an alert if order volume dropped by 20% compared with the same period on the previous day.
• Data change notifications similar to change data capture, such as when a particular order changes status.

3.2Message Formats, Schemas, and Versioning in Pub/Sub

Message format in Pub/Sub affects cost and processing, and high-volume streams in a data mesh architecture work well with Avro or Protobuf. Producers using those formats can assign schemas to Pub/Sub topics, and the schema helps ensure that consumers receive well-formed messages. Because streaming structures change constantly, producers and consumers must coordinate versioning of the message interface.

3.3Ordering, Transforms, and Streaming Scale in Pub/Sub

Message ordering in Pub/Sub (opens in a new tab) is a feature that lets a subscriber receive messages in the order they were published. Ordering is not recommended with Dataflow because the distributed processing model breaks the end-to-end guarantee while adding latency and reducing parallelism.

The limitations of ordering with Dataflow are specific:

• The Pub/Sub I/O connector might not preserve message ordering.
• Apache Beam does not define strict guidelines for the order in which elements are processed, so ordering might not be preserved in downstream transforms.
• Using Pub/Sub message ordering with Dataflow can increase latency and decrease performance because it limits the messages available for parallel processing.

Pub/Sub single message transforms (SMTs) manipulate, validate, and filter messages by attributes or data as they stream. Subscriptions that feed Dataflow should not use SMTs that filter out messages because filtered messages make the subscription backlog appear larger than what reaches Dataflow until the SMT processes them, which interferes with autoscaling. Topic SMTs that filter messages do not cause the same autoscaling problem because filtering happens before the subscription backlog is measured.

Scale and reliability for Pub/Sub to BigQuery pipelines depend on acknowledgment, worker capacity, and write-path efficiency. When backlog grows, increase the Pub/Sub acknowledgment deadline to slightly longer than the maximum expected processing time to prevent premature redelivery, increase the number of Dataflow workers to add processing capacity, and enable exponential backoff for transient failures. For sustained performance, migrate streaming writes from standard inserts to the BigQuery Storage Write API over gRPC for better performance and higher quotas, reduce getTable calls by caching table existence in worker memory or batching checks per bundle, optimize work away from millions of active high-cardinality keys, and proactively manage BigQuery and Compute Engine quotas for queries per second, IP addresses, and CPUs.

3.4Kafka Integration with Dataflow Through KafkaIO

Apache Kafka ingestion with Dataflow (opens in a new tab) centers on the Beam KafkaIO connector, which controls how event time and watermarks advance through the streaming pipeline. By default the watermark behavior is tied to processing and event time, and pipelines can call withTimestampPolicyFactory to assign a TimestampPolicy that calculates the watermark from Kafka log append time or message creation time. Correct timestamp policy selection matters because the watermark determines when windowed aggregations consider data complete.

The KafkaIO connector has two read implementations, the older ReadFromKafkaViaUnbounded and the newer ReadFromKafkaViaSDF, and Dataflow automatically chooses the best implementation based on SDK language and job requirements. Pipelines that use withTopic or withTopics query Kafka for available partitions at pipeline construction time, so the machine that constructs the pipeline must have permission to connect to Kafka. Using withTopicPartitions avoids that construction-time connection because partition assignments are supplied explicitly. For production, launch from a consistent environment with Flex templates and reduce verbose KafkaIO logging, for example by overriding SubscriptionState to WARN.

Kafka operations add explicit networking ownership that Pub/Sub does not require. Dataflow launches workers in a Virtual Private Cloud (VPC) network by default, so teams may need to select a different network and subnet and create firewall rules that allow Dataflow worker machines to reach the Kafka brokers. If VPC Service Controls are used, place the Kafka cluster inside the perimeter or extend the perimeter to the authorized VPN or Cloud Interconnect path. When Kafka runs outside Google Cloud, connect Dataflow to the cluster through shared private address space using Dedicated Interconnect or IPsec VPN, or through public addresses using public internet, Direct Peering, or Carrier Peering, with Dedicated Interconnect offering the most predictable performance and reliability at the cost of longer provisioning by third parties.

3.5Operational Tradeoffs Between Pub/Sub and Kafka

The operational difference is that Pub/Sub (opens in a new tab) removes broker, partition, and cluster management while Kafka keeps control of brokers, partitions, and network paths at the cost of configuration. Pub/Sub pipelines scale by adjusting Dataflow workers, acknowledgment deadlines, retry behavior, quotas, and the BigQuery write path, without provisioning brokers or broker connectivity. Kafka pipelines also use Dataflow workers and Flex templates, but they additionally depend on VPC design, firewall rules, construction-time Kafka permissions, watermark timestamp policy, and, for external clusters, Interconnect, VPN, or peering. Choose Pub/Sub when the goal is serverless fan-out to many filtering subscribers with schema enforcement and minimal infrastructure, and retain Kafka when existing Kafka topics, partitions, log times, or ecosystem dependencies must be preserved despite the added networking and lifecycle management.

Key Facts

• Cloud Dataflow runs Apache Beam pipelines with a Dataflow runner that uploads code to Cloud Storage, creates a Dataflow job, and allocates worker VMs, supporting both batch and streaming with exactly-once processing from the user's view.
• BigQuery multi-region locations store data in a single region inside the geography and do not provide cross-region replication.
• Dataflow BigQueryIO export reads consume export quotas via temporary Cloud Storage, while direct reads use the BigQuery Storage API to avoid export quotas and reduce run time.
• Pub/Sub subscription single message transform filtering interferes with Dataflow autoscaling, while topic-level filtering does not.
• Apache Kafka pipelines read through Beam KafkaIO where watermark timestamp policy controls window completion and Dataflow workers require VPC firewall access to Kafka brokers.