Batch and Streaming Data Processing

Design and implement streaming data processing pipelines means building a continuous path where events flow from ingestion through processing to storage without waiting for a batch window to close. On Google Cloud, Pub/Sub (opens in a new tab) ingests and buffers the event stream, Dataflow (opens in a new tab) runs the Apache Beam pipeline that transforms and enriches events in parallel, and Bigtable (opens in a new tab) provides scalable key-value storage for sinks and lookups. The three services divide work by movement, computation, and state, and events typically flow from producers to a Pub/Sub topic, from the subscription to Dataflow workers, and from Dataflow to Bigtable rows.

1.1Pub/Sub as Streaming Ingestion Layer

Pub/Sub is a scalable and durable event ingestion service that delivers messages from publishers to streaming pipelines. Publishers write events to a topic and subscriber pipelines read from subscriptions attached to that topic, so a Dataflow job can consume a continuous unbounded stream. Pub/Sub quotas and limits are designed around project-level usage, and publishers and subscribers in different projects receive independent data-throughput quotas. If multiple pipelines publish or subscribe to a single topic, deploying each pipeline into its own project provides the maximum allowable throughput on that topic.

Bigtable also supports direct ingestion from Pub/Sub subscriptions as a Preview feature. A configured Pub/Sub subscription sends messages to a Bigtable table, and Pub/Sub writes these messages to Bigtable as new rows. This direct path bypasses Dataflow when no transformation is needed, while Dataflow remains required when events need filtering, enrichment, windowing, or joins before storage.

1.2Dataflow as Stream Processing Engine

Dataflow is a cloud service and programming model for big data processing that supports both batch and streaming execution. A Dataflow streaming job runs the stages of an Apache Beam pipeline in parallel across multiple workers using Compute Engine virtual machine (VM) instances, which lets throughput scale with event volume. Apache Beam provides the unified model for defining batch and streaming data-parallel pipelines, so the same transform logic can run on bounded and unbounded PCollection data.

Dataflow connects to Bigtable through the Bigtable Beam connector (BigtableIO), which performs batch and streaming operations on Bigtable data. A pipeline can read rows from Bigtable, process Pub/Sub events, and write results back to Bigtable, and Dataflow templates can export and import Bigtable data as Avro, Parquet, or SequenceFiles. Bigtable can also serve as a key-value lookup that enriches data inside a pipeline, so each streaming element flows from Pub/Sub into Dataflow, is joined against Bigtable state, and then flows onward to its destination.

1.3Bigtable as Streaming Sink and Lookup Store

Bigtable is the Google Cloud NoSQL store used in streaming architectures both as a pipeline sink and as an enrichment source. Dataflow can process data stored in Bigtable or store the output of a Dataflow pipeline in Bigtable, which makes Bigtable sit downstream of Dataflow for persistent state and upstream of Dataflow for lookups. For analytics, BigQuery complements Bigtable by creating an external table to query a Bigtable table and join that data to other BigQuery tables, or by exporting BigQuery data back to Bigtable with reverse ETL (RETL).

The three services divide streaming work by role, and the choice depends on whether the need is movement, computation, or state:

A common flow composes all three services in order: events flow from producers to a Pub/Sub topic, from the subscription to Dataflow workers, and from Dataflow to Bigtable rows, with later reads flowing from Bigtable to BigQuery for analysis.

1.4Planning Sources, Sinks, and External Calls for Scale

Planning sources, sinks, and external calls means accounting for systems outside pipeline code that bound scalability, cost, and latency. Data pipelines read from sources and write to sinks, and they may also call external systems for enrichment, filtering, or business logic during a processing step. Three factors outside Dataflow directly affect whether a streaming pipeline meets its performance targets:

• Scalability of external systems: an external system can form the upper bound of scalability, such as an Apache Kafka topic configured with too few partitions for the required read throughput.
• Choice of data formats: formats that support parallelizable reads are faster, such as Avro, which is usually faster than CSV files with embedded newlines in fields and faster than compressed files.
• Data location and network topology: geographic proximity and networking between the pipeline and its sources and sinks affect latency, so co-locating Dataflow workers with Pub/Sub, Bigtable, and BigQuery resources matters for time-sensitive delivery.

Calling external services from a pipeline adds per-call overhead and quota risk. Native Beam transforms such as BigQueryIO batch multiple elements into single requests, and pipelines should batch requests wherever possible because external services may enforce daily quotas or per-second rate limits. Because Dataflow parallelizes work and autoscaling can add workers to a slow external-call step, added workers can overwhelm the external system, so traffic must be limited to sustainable batch sizes and concurrent calls. Using Google Cloud managed sources and sinks removes much capacity planning, but Dataflow still depends on Compute Engine quota, and insufficient Compute Engine quota can prevent jobs from starting or hinder autoscaling.

1.5Deploying and Updating Streaming Pipelines in Production

Deploying and updating streaming pipelines in production combines continuous integration, continuous delivery, and explicit update strategy. Test automation runs unit and integration tests on each commit, often with Maven for Java Beam pipelines, and code that passes produces a passing build packaged as deployment artifacts. For the Java SDK that artifact is a self-executing JAR file stored in a deployment-environment bucket, while Classic Templates add a JSON template file and an optional metadata template.

Continuous delivery copies those artifacts to preproduction environments for end-to-end tests and then to production. Batch pipelines can launch directly as a new Dataflow job or be launched by Managed Service for Apache Airflow or Cloud Scheduler, but streaming pipelines are more complex because an existing long-running job must be replaced or updated. If an in-place update is not possible or not desired, coordination of multiple Dataflow jobs through parallel pipelines minimizes or prevents business disruption while the new pipeline takes over the Pub/Sub subscription and Bigtable sink.

Optimize and manage data processing workloads means keeping batch and streaming pipelines fast, reliable, and cost-efficient from development through production. On Google Cloud, Cloud Dataflow (opens in a new tab) runs parallel processing across workers, Cloud Monitoring and Cloud Logging expose performance and health signals, Knowledge Catalog governs quality and lineage, and Managed Service for Apache Airflow orchestrates jobs into larger workflows. Together these services cover how data flows from sources to sinks, where bottlenecks form, and how lifecycle changes are delivered safely.

2.1Analyze Pipeline Performance and Scalability

Pipeline performance analysis starts with the fact that Cloud Dataflow runs pipeline stages in parallel across multiple workers, but outside factors can set the upper bound on throughput. Three outside factors shape scalability and must be checked alongside pipeline code and the Dataflow service itself:

• Scalability of external systems: an external system such as an Apache Kafka topic with too few partitions limits read throughput no matter how many Dataflow workers are added.
• Choice of data formats: formats that support parallelizable reads such as Avro are usually faster than CSV files with embedded newlines in fields and faster than compressed files.
• Data location and network topology: geographic distance and networking between the pipeline, its sources, and its sinks affect latency and throughput, which makes regional placement a design decision.

Calls to external services for enrichment, filtering, or business logic add per-call overhead that lowers efficiency. Many native Apache Beam I/O transforms reduce this overhead automatically by batching elements, including BigQueryIO for streaming inserts, so multiple elements share one request where possible. Because Dataflow parallelizes work, unchecked traffic can overwhelm an external service or exhaust daily quotas and per-second rate limits, and autoscaling can make it worse by adding workers to a slow external-call step and increasing pressure further.

2.2Optimize Resources and Control Costs

Resource optimization for Dataflow means using Google Cloud managed sources and sinks to remove manual capacity planning while still respecting quotas and limits. Cloud Pub/Sub provides scalable and durable event ingestion for streaming pipelines, and BigQuery provides scalable storage and analysis for outputs, but Dataflow itself imposes quotas and limits that can be raised through Google Cloud support. Dataflow jobs run on Compute Engine virtual machine (VM) instances, so sufficient Compute Engine quota is required; insufficient quota can prevent jobs from starting or block autoscaling when load grows.

Cost and quota control also depends on how Pub/Sub throughput is isolated by project. Pub/Sub quotas and limits are designed around project-level usage, where publishers and subscribers in different projects receive independent data-throughput quotas. If multiple pipelines publish or subscribe to a single topic, deploying each pipeline into its own project yields the maximum allowable throughput on that topic. Quota consumption can be viewed in the Google Cloud console, and a quota increase should be requested when any single pipeline exceeds per-project quotas and limits.

2.3Ensure Data Quality and Lineage Tracking

Data quality and lineage tracking ensure that transformations remain trustworthy as data moves from lakes and streams into analytics and machine learning. Knowledge Catalog is the organization-wide data governance platform that centrally discovers, manages, monitors, and governs data and AI artifacts across data platforms. Knowledge Catalog provides trusted access plus a universal catalog where metadata from supported Google Cloud sources is ingested automatically, while outside sources require custom entries enriched with business metadata through aspects.

Knowledge Catalog supports governance work that directly protects pipeline outputs:

• Manage data lineage: track how data flows and transforms across systems, with column-level lineage also available in BigQuery.
• Manage data quality checks and data profiles: validate formats, ranges, and distributions before training, including automatic data quality capabilities that help detect anomalies such as data poisoning.
• Manage data discovery and processing: search and explore assets across different data marts so teams reuse curated data instead of rebuilding pipelines.
• Manage feature and model metadata: centralize features for reuse and govern model artifacts across projects.
• Create a business glossary: standardize definitions and vocabulary so quality rules mean the same thing across the organization.

Operational visibility comes from capturing real-time logs and metrics with Cloud Logging and Cloud Monitoring, plus Cloud Audit Logs for audit trails of API calls. These signals support monitoring of data collection, storage, and transformation, detection of inconsistent behavior or unexpected access, and auditing of who accessed or changed data. Logs must not contain personally identifiable information (PII) or confidential data, and sensitive-data inspection results from BigQuery can be sent into Knowledge Catalog for governance.

2.4Manage Pipeline Lifecycle and Orchestration

Pipeline lifecycle management covers continuous integration (CI), continuous delivery (CD), and orchestrated execution in preproduction and production environments. CI merges code frequently into a shared repository and runs unit and integration tests on each commit, often with a build tool such as Maven, and code that passes becomes a passing build packaged into deployment artifacts. For the Apache Beam Java software development kit (SDK), the artifact can be a self-executing Java Archive (JAR) file stored in a Cloud Storage bucket in the deployment environment, while Classic Templates add a JavaScript Object Notation (JSON) template file plus an optional metadata template alongside the JAR.

Continuous delivery copies those artifacts to deployment environments for manual launch after end-to-end tests, while continuous deployment can launch them automatically with different patterns for batch and streaming. Batch and streaming launches differ in automation and risk:

Streaming deployment is more complex because a long-running job may need to be replaced or updated in place without business disruption. When an in-place update is not possible or not desired, coordination of multiple Dataflow jobs in parallel allows the new pipeline to take over while the old pipeline drains. Proper identities and roles must be granted so the CI/CD pipeline can access the source repository, run integration tests with Application Default Credentials (ADC), and deploy artifacts across projects.

2.5Standardize Collection and Transformation for Reuse

Standardized collection and transformation reduce duplicated engineering and lower technology and personnel costs across the machine learning lifecycle. Cloud Dataflow handles batch and stream processing from varied sources with machine learning component integration, and combining Dataflow with Eventarc supports event-driven processing; GPU and right-fitting capabilities improve efficiency for machine learning jobs. Dataform automates Structured Query Language (SQL) transformations such as feature scaling, encoding categorical variables, and creating new features, while Feature Store centralizes features in BigQuery for reuse across models and projects. Data lakes built on Cloud Storage with BigLake and streaming platforms such as Pub/Sub, Dataflow, or Apache Kafka feed the same governed path, so curated datasets become reusable data products discovered through Knowledge Catalog and BigQuery sharing.

Design and implement batch data processing pipelines means transforming, enriching, and analyzing large volumes of data at rest for business intelligence and machine learning. Data flows from sources such as Cloud Storage objects or JDBC databases through transformation logic into sinks such as BigQuery (opens in a new tab) tables. Three managed services divide the transformation work by engine: Dataflow (opens in a new tab) runs Apache Beam pipelines, Cloud Data Fusion provides visual integration with plugins, and Managed Service for Apache Spark runs Spark batch workloads.

3.1Transformation Patterns for Batch Loads

Transformation patterns for batch loads define where transformation happens relative to loading into BigQuery. In extract, transform, and load (ETL), data is extracted from the source, transformed in an intermediate engine, and then loaded into BigQuery. In extract, load, and transform (ELT), data is loaded first into a BigQuery staging table and then transformed inside BigQuery. Choose ETL when the source data needs cleaning, joining, or reshaping before it lands, and choose ELT when BigQuery SQL and slots can perform the transformation more simply and scalably after loading.

The ELT workflow for BigQuery moves data through an ordered sequence:

1. Load to staging: load raw data into a staging table using an extract and load option.
2. Transform with SQL: run SQL queries against the staging table to clean, join, and reshape the data.
3. Write to production: write the query output into the final production table.

This sequence keeps the raw landing separate from the curated output, so the staging table can be re-queried and the production table stays consistent for downstream consumers.

3.2Building Transforms with Dataflow, Cloud Data Fusion, and Managed Service for Apache Spark

Batch transforms on Google Cloud are implemented in the processing engine that sits between the source and the BigQuery sink. Dataflow clones the JDBC to BigQuery template code and adds Apache Beam transforms to reshape records before the BigQueryIO write. Cloud Data Fusion transforms data using CDAP plugins in a managed integration pipeline. Managed Service for Apache Spark transforms data using Spark SQL or custom code in Scala, Java, Python, or R.

The following table compares the implementation options by mechanism and selection criteria:

A pipeline depends on its chosen engine for compute and on BigQuery for scalable storage and SQL compute, so moving logic between engines changes cost and operational ownership without changing the source-to-sink flow.

3.3Designing Scalable Sources, Sinks, and Data Formats

Scalable batch design accounts for systems outside the pipeline code because Dataflow runs pipeline stages in parallel across multiple workers. Data pipelines read data from sources and write data to sinks, and they may also call external systems for enrichment or business logic during processing. Three factors outside the pipeline constrain how far that parallel execution can scale:

• Scalability of external systems: an external system forms the upper bound of throughput, such as an Apache Kafka topic with too few partitions for the required read throughput.
• Choice of data formats: formats that support parallelizable reads are faster, such as Avro, which is usually faster than CSV files with embedded newlines in fields and faster than compressed files.
• Data location and network topology: geographic proximity and networking between the pipeline workers and the sources and sinks affect performance.

Calling external services from a pipeline adds per-call overhead that reduces efficiency, and many native Beam I/O transforms such as BigQueryIO already batch multiple elements into single requests. Because parallel workers can overwhelm an external service or exhaust daily quotas and per-second rate limits, batch requests where possible and limit concurrent calls to sustainable levels. Using Google Cloud managed sources and sinks removes much of the capacity planning, but Dataflow, Compute Engine, and destination quotas and limits still apply.

3.4Deploying and Operating Batch Pipelines for Cost and Performance

Deploying and operating batch pipelines for cost and performance means separating storage from compute and automating delivery of tested artifacts. BigQuery treats storage, compute, memory, and slots separately, where slots are the unit of computational capacity for queries and Google manages the underlying infrastructure. Slot quantity can be adjusted for workloads and purchased through yearly or three-year commitments with BigQuery reservations. This separation lets a data lake grow without resizing clusters or copying data between nodes.

Continuous integration and continuous delivery automate build, testing, and deployment of pipeline code into different environments. Test automation runs unit and integration tests on each commit, often with a build tool such as Maven, and code that passes produces a passing build packaged as deployment artifacts. For batch pipelines, continuous deployment launches the artifacts directly as a new Dataflow job, or other systems launch the job when required, such as Managed Service for Apache Airflow running the batch job inside a workflow or Cloud Scheduler scheduling the batch job. Storing artifacts such as a self-executing JAR file or template files in a Cloud Storage bucket per environment keeps preproduction and production launches repeatable and isolated.

Key Facts

• Pub/Sub provides durable event ingestion where publishers write to a topic and Dataflow jobs read from subscriptions, and deploying pipelines that share one topic into separate projects gives each pipeline independent throughput quota.
• Dataflow runs Apache Beam pipeline stages in parallel across Compute Engine virtual machine workers and connects to Bigtable through BigtableIO for reads, writes, and enrichment lookups.
• Bigtable serves as a streaming sink for Dataflow output rows and as a lookup source for enrichment, with BigQuery external tables and reverse ETL linking Bigtable to analytics.
• Avro supports parallelizable reads and is usually faster than CSV files with embedded newlines and faster than compressed files, while an external system such as a Kafka topic with too few partitions caps throughput.
• Insufficient Compute Engine quota can prevent Dataflow jobs from starting or block autoscaling, and BigQuery slots are the unit of query compute capacity purchasable through yearly or three-year reservations.