Building Dataflow Pipelines for Batch and Streaming

Data cleansing is the process of inspecting, validating, and correcting datasets in Google Cloud before analytical use. Knowledge Catalog (opens in a new tab) centralizes discovery, monitoring, and governance of data and AI artifacts, while Cloud Dataflow filters records in flight and BigQuery standardizes data after load. Data flows from collection through storage and transformation into analytical consumption, with governance monitoring flows, transformations, and access alongside that path.

1.1Visual Profiling and Governance with Knowledge Catalog

Visual profiling with Knowledge Catalog is the creation of data profiles that describe formats, ranges, distributions, and completeness before data is trusted. A profile reveals missing values, outliers, skewed distributions, and unexpected formats, and BigQuery adds column-level lineage to show which source columns produced each field. Knowledge Catalog also manages automatic data quality checks that verify formats, ranges, and distributions before training, plus business glossary vocabulary, aspects for business context, unified governance for BigLake and Iceberg and Delta Lake tables, and data mesh support for domain ownership. Continuous monitoring uses Cloud Logging for logs, Cloud Monitoring for metrics and alerts, and Cloud Audit Logs for API audit trails, with alerts routed to owners when quality fails or access deviates.

1.2Stream and Batch Filtering in Cloud Dataflow

Filtering in Cloud Dataflow (opens in a new tab) is a transform between managed sources and managed sinks in the ordered flow from ingestion to storage. The pipeline pulls events from Pub/Sub, applies parsing, validation, and enrichment with Apache Beam logic, and writes trusted output to BigQuery, Cloud Storage, or Bigtable. Valid records flow to the primary analytical sink while malformed records flow to a dead-letter sink such as a Pub/Sub topic, Cloud Storage prefix, or BigQuery quarantine table for auditing and reprocessing. Dataflow provides exactly-once processing by default and an at-least-once mode for lower cost and latency when duplicates can be tolerated, and Pub/Sub subscriptions buffer unacknowledged messages for up to 7 days during outages.

1.3Post-Ingestion Standardization in BigQuery

Post-ingestion standardization in BigQuery loads raw data into a staging table and then uses SQL to clean and write trusted output to a production table. SQL stored procedures and SQL user-defined functions run inside BigQuery, while logic in other languages runs through Cloud Run functions or client-side code. Performance controls include partitioning and clustering, fine-grained data manipulation language for small writes, and materialized views for refreshed outputs. Access to curated tables is controlled with Identity and Access Management roles plus row-level security, column-level security, authorized views, and policy tags.

Selecting a processing service means matching the engine to batch, streaming, warehousing, or messaging work. Cloud Dataflow runs Apache Beam pipelines for unified batch and streaming, Spark-based services run cluster and visual integration work, BigQuery stores and transforms warehouse data, and Pub/Sub and Kafka provide ingestion. Dataflow handles provisioning, autoscaling, and fault tolerance so teams focus on pipeline logic.

2.1Unified Batch and Streaming with Cloud Dataflow

Cloud Dataflow pipelines read from a source, apply transforms and aggregations, and write to a destination such as BigQuery. The same Beam model handles bounded batch data and unbounded streaming data with windowing, grouping, and joining. The Beam runner model separates what to compute from where it runs, so the Dataflow runner uploads code to Cloud Storage, creates a Dataflow job, and allocates worker virtual machines. Google-provided templates support basic movement with JavaScript user-defined functions, while Java, Python, and Go software development kits support complex joins and machine learning enrichment.

2.2Analytical Warehousing with BigQuery

BigQuery is the analytical warehouse that stores terabyte to petabyte datasets and serves SQL transformation and queries. A dataset location is regional such as Iowa or multi-region such as US or EU, but multi-region does not replicate data across regions because data still rests in a single region. Ingestion arrives through Dataflow batch and streaming pipelines, Cloud Data Fusion Spark jobs, and Datastream change data capture that merges UPSERTs through the BigQuery Write API. Beam BigQueryIO offers EXPORT reads through temporary Cloud Storage and DIRECT_READ reads through the BigQuery Storage API, while batch writes use load jobs and streaming writes use streaming inserts or the Storage Write API.

2.3Messaging with Pub/Sub and Kafka

Messaging decouples producers from consumers for streaming ingestion and fan-out. Pub/Sub provides serverless topics and subscriptions where one publish serves many filtering subscribers, with Avro or Protobuf schemas and per-project throughput quotas. Apache Kafka provides a broker log that Dataflow reads through the Beam KafkaIO connector, where timestamp policy controls watermarks and workers need Virtual Private Cloud firewall access to brokers. Subscription single message transform filtering interferes with Dataflow autoscaling, while topic-level filtering does not.

Transformations reshape batch and streaming datasets through windowing, distributed SQL, and resilient error handling. Apache Beam on Cloud Dataflow groups events by event time and enriches them, BigQuery runs serverless SQL stages, and Cloud Data Fusion compiles visual designs into Spark jobs. All paths split clean output from malformed records so failures can be inspected and replayed.

3.1Windowing, Triggers, and State for Streaming Pipelines

Windowing assigns each event in an unbounded PCollection to finite intervals for aggregation. Apache Beam (opens in a new tab) supports tumbling windows that place each event in one fixed interval, sliding windows that place each event in multiple overlapping intervals, and session windows that merge events with the same key until an inactivity gap expires. Triggers control when a window emits relative to the watermark, with discarding mode emitting only new elements and accumulating mode emitting the full result plus allowed lateness for corrections. Stateful ParDo transforms keep per-key state and timers, side inputs broadcast small reference data, and the Bigtable enrichment handler performs managed key lookups.

3.2Scale-Out SQL and Spark Execution

Scale-out transformation runs cleaning and restructuring in parallel across workers. BigQuery breaks each query into an execution tree of stages that exchange intermediate data through a distributed shuffle tier, then generates a query plan, dynamically adjusts it, and writes results to persistent storage. Cloud Data Fusion builds a directed acyclic graph of sources, transforms, and sinks in the user interface, then the planner converts that graph into an Apache Spark job on Managed Service for Apache Spark. Dataflow runs the same Beam code for batch and streaming with autoscaled workers, exactly-once processing by default, and regional controls for worker region and temporary files.

3.3Schema Evolution and Dead-Letter Isolation

Schema evolution handling keeps pipelines running when source fields are added, renamed, or retyped. Reusable Beam transform libraries, continuous integration tests, and BigQuery staging tables isolate change so only the staging-to-production SQL needs updating. The dead-letter pattern routes invalid records to a separate Pub/Sub topic, Cloud Storage bucket, or BigQuery quarantine table while valid records continue to the primary sink. Validation checks formats, ranges, and distributions, with Knowledge Catalog quality checks, BigQuery anomaly detection, and de-identification controls such as masking and column-level security.

Batch processing transforms large historical datasets with restartable jobs and cost-effective ephemeral capacity. Cloud Dataflow serves Beam batch logic serverlessly, Managed Service for Apache Spark serves Spark logic on temporary clusters, and BigQuery serves SQL transforms in the warehouse. Orchestration, idempotency, shuffle, and quotas determine whether jobs finish on time without excess cost.

4.1Batch Execution Frameworks

Cloud Dataflow is a fully managed serverless service for Apache Beam batch and streaming with the same programming model. Data flows from sources through Beam transforms to destinations, Google allocates Compute Engine workers at job start and deletes them at completion, and the service autoscales, rebalances work, and corrects stragglers. Cloud Data Fusion builds batch pipelines visually on the Cloud Data Appliance for Hadoop project and executes them as Spark jobs, including Java Database Connectivity ingestion into BigQuery. BigQuery (opens in a new tab) stores history and runs SQL batch transforms, where DIRECT_READ avoids export quotas and streaming inserts avoid repeated load jobs.

4.2Orchestration, Error Handling, and Idempotent Design

Robust batch orchestration defines order, dependencies, and retry behavior so reruns are safe. Cloud Composer runs Airflow directed acyclic graphs with the Dataflow Operator, Google Cloud Workflows sequences loosely coupled HTTP services in YAML or JSON, and Dataflow data pipelines schedule recurrent jobs with multi-run cost tracking. There is no cross-pipeline communication in Dataflow, so one pipeline writes to Cloud Storage or App Engine cache and the next pipeline reads it back. Dead-letter branches isolate parse failures, acknowledgment deadlines prevent premature Pub/Sub redelivery, and pure deterministic DoFn functions with deterministic keys and replay-tolerant sinks make reruns idempotent.

4.3Shuffle Performance and Batch Quotas

Dataflow shuffle partitions and groups data in the service backend instead of on worker virtual machines. The shuffle operation backs GroupByKey, CoGroupByKey, and Combine, which lowers worker central processing unit, memory, and disk use and improves autoscaling and fault tolerance. Batch jobs use Dataflow shuffle by default with a 25 gigabyte boot disk, require the region option without a zone override, and carry a separate service charge. Limits include 25 concurrent jobs per project by default, 2,000 workers per job or 4,000 for shuffle batch, 10-day cancellation, and regional shuffle slots such as 640 slots in us-central1.

Streaming pipelines aggregate unbounded events by event time while handling late and out-of-order arrivals. Dataflow (opens in a new tab) applies Window.into after reading from Pub/Sub and before per-key aggregation, maintains keyed state with combines, and writes idempotent results to BigQuery or Iceberg. Watermarks, allowed lateness, and side outputs together balance completeness, latency, and state cost.

5.1Temporal Aggregation with Fixed, Sliding, and Session Windows

Windowing divides an unbounded stream into finite intervals so each interval aggregates separately. Fixed windows assign events to equal non-overlapping intervals such as every 30 seconds with FixedWindows.of, then map user ID to click count and apply Sum.longsPerKey for one total per user per interval. Sliding windows use overlapping size and period for moving averages, while session windows group activity bursts separated by inactivity gaps. Fixed windows keep state lowest, sliding windows duplicate events across windows, and session windows create variable windows that merge and split under bursty traffic.

5.2State Management and Timely Emission

Stateful processing holds intermediate values by key and window while combines such as Sum, Combine.PerKey, and Count.PerKey merge them. Dataflow runs stages in parallel, so a hot key concentrates state on one worker and shows growing backlog bytes and backlog elements. Mitigations include rekeying with ParDo, withFanout for partial combines, and withHotKeyFanout for skewed per-key counts. Timely emission uses source polling such as poll interval seconds and sink triggering frequency such as every 20 seconds, plus windowed file writes with withWindowedWrites and a window-based filename policy.

5.3Watermarks, Allowed Lateness, and Excessively Late Data

Watermarks estimate event-time completeness from source timestamps and heuristic progress, and a window whose end is below the watermark emits its on-time result. Allowed lateness keeps window state alive after the watermark passes so late events can update results before garbage collection. The lifecycle runs in order: first assigned and open, then watermark passes window end, then late arrivals update retained state, then state is discarded. Excessively late events arriving after discard must be dropped and counted or routed to a side output for audit and batch repair.

Processing logic converts raw batch and streaming inputs into enriched and aggregated outputs at scale. Cloud Dataflow (opens in a new tab) runs custom Beam transforms, BigQuery runs declarative SQL analytics, and Managed Service for Apache Spark runs ephemeral Spark extract-transform-load jobs. Managed sources such as Pub/Sub and sinks such as BigQuery provide elasticity beneath all three patterns.

6.1Custom Transforms with Beam and Dataflow

Apache Beam defines portable batch and streaming transforms in Java, Python, and Go with multi-language pipelines and runners including Flink and Spark. Cloud Dataflow executes those pipelines on managed Compute Engine workers with autoscaling, dynamic rebalancing, geographic redundancy, and exactly-once processing by default. Transforms clean, join, and aggregate inputs before writing to BigQuery or Cloud Storage, with Avro preferred over CSV with embedded newlines for parallel reads. External calls need batching and concurrency limits because autoscaling can overwhelm quotas, and separate Pub/Sub subscriptions let production, replica, and test pipelines pull independently with up to 7 days of buffered messages.

6.2Declarative Analytical Logic in BigQuery

Declarative logic in BigQuery expresses large-scale transforms as SQL without managing clusters. Entry points include the console, bq query command, jobs.query and jobs.insert application programming interface methods, and client libraries, all submitting to the same execution service. Multi-statement queries run ordered statements with shared state and procedural variables for stored procedures, while BigQuery Studio adds SQL editing, Python notebooks, and versioned assets. Execution follows an ordered lifecycle:

1. Execution tree: BigQuery breaks the query into parallel stages with steps.
2. Shuffle tier: Stages exchange intermediate data through fast distributed storage and networking.
3. Query plan: BigQuery generates a plan once planning information is ready.
4. Dynamic planning: Workers monitor progress and adjust the plan as results arrive.
5. Results: BigQuery writes results to persistent storage and returns them for caching.

On-demand pricing charges for bytes scanned with fixed per-project capacity, while capacity-based pricing charges for reserved slots.

6.3Ephemeral Spark Jobs on Managed Service for Apache Spark

Managed Service for Apache Spark is the serverless Spark runtime where distributed extract-transform-load runs only while the processing stage runs. Managed Airflow stores directed acyclic graph files in Cloud Storage and orchestrates validation, cluster creation, Spark execution, and deletion in order. Cloud Data Fusion compiles its visual graph into the same Spark execution. Processed output flows to Cloud Storage in Parquet or Optimized Row Columnar formats, with database and partition metadata in Dataproc Metastore.

AI enrichment adds predictions, labels, and protected features to batch and streaming pipelines. BigQuery ML scores data where it lives, Vertex AI hosts and orchestrates models, Dataflow calls pre-trained application programming interfaces inline, and Sensitive Data Protection plus Knowledge Catalog govern sensitive fields. The choice between warehouse scoring and inline scoring trades operational simplicity against latency and cost.

7.1In-Warehouse Inference with BigQuery ML and Vertex AI

In-warehouse inference with BigQuery ML (opens in a new tab) trains and scores with SQL instead of moving data to Python systems. Models are stored in BigQuery datasets and created with CREATE MODEL, then inference functions score rows and write results back to BigQuery tables. Materialized tables and BigQuery Storage API reads improve throughput, and supported models include linear and logistic regression, k-means, matrix factorization, principal component analysis, and time-series models including ARIMA_PLUS. Remote models point to Vertex AI endpoints or Cloud AI application programming interfaces, with features centralized in Vertex AI Feature Store and orchestration through Vertex AI Pipelines directed acyclic graphs, multi-statement queries, or Dataform versioned workflows.

7.2Pre-Trained APIs in Stream and Batch Pipelines

Inline enrichment in Dataflow treats a pre-trained API as an external system inside a processing step. Data flows from Cloud Storage or Pub/Sub into Dataflow workers, then to the external API, then enriched results flow onward to BigQuery. Each per-element call adds overhead, so batching multiple elements into fewer requests is preferred and native Beam transforms such as BigQueryIO already batch. Quotas limit daily totals and requests per second, and Dataflow parallelism multiplies traffic because many workers call the same endpoint concurrently. Adding workers to a quota-blocked step increases concurrent calls instead of fixing throughput, so pipelines must limit batch sizes and concurrent calls.

7.3Securing Enriched Data with Protection and Governance

Securing enriched data removes sensitive information before AI processing and tracks assets afterward. Sensitive Data Protection discovers and classifies data with infoTypes, then applies pseudonymization, de-identification, masking, tokenization, and bucketing. The inline order is fixed: first raw source enters, then client service orchestrates, then data flows to the Cloud Data Loss Prevention API for redaction, then only protected data flows to Cloud Storage, BigQuery, or training. Knowledge Catalog governs lineage, quality checks, profiles, discovery, and business glossary across BigLake, Iceberg, and Delta Lake, with Cloud Logging, Monitoring, and Audit Logs providing auditability.

Data acquisition moves batch files and streaming events from on-premises, multi-cloud, and Google Cloud sources into analytical services. Pub/Sub buffers continuous events, Cloud Storage stages files, Transfer Appliance ships offline bulk data, and BigQuery Data Transfer Service loads warehouse tables. Encryption, identity, network perimeters, and logging protect every boundary from source to warehouse.

8.1Streaming Ingestion with Pub/Sub

Cloud Pub/Sub (opens in a new tab) is the scalable ingestion backbone that decouples publishers from Dataflow processors. Data flows from publishers to a topic, then from the topic through subscriptions to pipelines, so ingestion continues when processing lags. Throughput quotas are independent per project, so pipelines sharing a topic gain maximum throughput when each runs in its own project. Publisher flow control limits publish rates during spikes, retry settings handle failed publishes, and message storage policies restrict residency while customer-managed encryption keys control application-layer encryption.

8.2Securing and Monitoring Ingestion Pipelines

Secure ingestion protects data with encryption, least-privilege access, network boundaries, and audit trails. Cloud Storage encrypts by default and Pub/Sub encrypts at rest and in transit, with customer-managed keys in Cloud Key Management Service and optional external keys for compliance. Identity uses dedicated service accounts, dataset roles, policy tags for confidential columns, and row and column-level security in BigQuery. Virtual Private Cloud Service Controls defines perimeters, Private Service Connect uses private addresses, and Cloud VPN or Interconnect protects external movement. Cloud Logging, Cloud Monitoring, Admin Activity logs, and Data Access logs record who accessed data, with Knowledge Catalog managing lineage and quality and Security Command Center detecting misconfigurations.

8.3Bulk and Batch Transfer Mechanisms

Bulk transfer moves large existing datasets without continuous streaming. Transfer Appliance captures 300 terabytes in under 25 days without outbound bandwidth, with data available in Cloud Storage within about 10 business days after shipment. The ordered workflow is capture locally, then ship to Google, then upload to Cloud Storage, then access from BigQuery or Dataflow. BigQuery Data Transfer Service loads warehouse sources such as Snowflake on scheduled or incremental transfers that move only changed rows, while Cloud Storage staging holds raw files for validation before BigQuery load.

Integrating a new source maps its volume, velocity, variety, and schema to a batch, streaming, or change-data-capture pattern. Pub/Sub fronts streaming events, Cloud Storage stages batch files, Cloud Data Fusion integrates databases visually, Dataflow transforms both modes, and Datastream replicates database changes. Validation, observability, and safe promotion keep new integrations reliable.

9.1Assessing Volume, Velocity, Variety, and Schema

Assessment determines service choice, cost, and performance before the first pipeline is built. Volume sets throughput and storage, velocity decides between batch files and continuous streams, variety determines parsing, and schema determines tolerance for evolving fields. Avro is usually faster than CSV with embedded newlines, geographic proximity affects latency, and external calls need batching because Dataflow parallelism and autoscaling can overwhelm quotas. Migrate data pipelines (opens in a new tab) guidance maps streaming bursts to Pub/Sub, hourly files to Cloud Storage, visual database work to Cloud Data Fusion, and unified transforms to Beam on Dataflow.

9.2Ingestion Patterns and Implementation Services

Batch ingestion moves bounded extracts on a schedule, streaming ingestion moves unbounded Pub/Sub events with windowing for low latency, and change data capture replicates inserts, updates, and deletes with UPSERT semantics. Publishers send to Pub/Sub topics through functions or APIs, Dataflow runs Beam pipelines with exactly-once processing by default, Cloud Data Fusion converts its directed acyclic graph into Spark computations with Java Database Connectivity drivers, and Datastream writes to BigQuery through the Write API or to Cloud Storage for Dataflow templates that load BigQuery, Cloud SQL, or Spanner. A Pub/Sub subscription stores unacknowledged messages for up to 7 days for buffering and replay, and throughput quotas remain per project.

9.3Reliability and Operational Integrity

Reliability combines validation, controlled external calls, resilient sinks, and tested promotion. Cloud Monitoring tracks signals, Cloud Logging retains audit evidence, Knowledge Catalog verifies quality, and BigQuery anomaly detection flags poisoning. Native Beam input/output batches requests, publisher flow control absorbs spikes, and BigQuery stores data synchronously in two zones while regional Cloud Storage replicates across zones. Continuous integration builds passing artifacts such as self-executing Java archives in Cloud Storage, promotes them through preproduction end-to-end tests, and updates streaming jobs by coordinating parallel Dataflow jobs when in-place update is unsuitable.

Key Facts

• Cloud Dataflow runs Apache Beam pipelines with exactly-once processing by default, autoscaling workers, and dead-letter branches that send malformed records to a separate sink while valid records continue.
• Pub/Sub subscriptions buffer unacknowledged messages for up to 7 days and throughput quotas are independent per project, so separate projects and subscriptions isolate production, test, and replica pipelines.
• BigQuery multi-region locations store data in a single region within the geography and do not provide cross-region replication, while BigQueryIO DIRECT_READ avoids export quotas and partitioning and clustering improve cleansing performance.
• Windowing controls temporal aggregation with tumbling, sliding, and session windows, watermarks track event-time progress, and allowed lateness plus side outputs decide whether late data updates results or flows to audit and repair.
• AI enrichment scores either in-warehouse with BigQuery ML and Vertex AI endpoints or inline in Dataflow calls to pre-trained APIs, with sensitive data de-identified through the Cloud Data Loss Prevention API before storage or training.