Prepare and test your skills
Prepare and test your skills
Evaluating real-time and batch processing needs means deciding how fast data from Pub/Sub to BigQuery must be available for analysis. Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that decouples publishers from subscribers, and BigQuery (opens in a new tab) is the analytics warehouse where the events land for SQL queries and dashboards. Real-time ingestion keeps data continuously fresh, while batch processing collects data on a schedule for complete and validated results.
Real-time data ingestion with Pub/Sub and BigQuery is the continuous movement of events from producers into BigQuery tables for near-immediate analysis. Publishers send events to the Pub/Sub service without waiting for subscribers, which increases flexibility and robustness compared to synchronous calls. Typical Pub/Sub latency is on the order of 100 milliseconds, so dashboards and queries can reflect current activity.
Data flows from event sources to BigQuery in a fixed order:
The Pub/Sub subscription also buffers data if the downstream pipeline stops. Backed-up events are processed when the pipeline runs again, so data loss stays low and freshness recovers.
Batch data processing is the scheduled collection, transformation, and loading of accumulated data to meet a completeness or deadline requirement. A common example is a business-critical daily report that must be completed by a fixed deadline to be sent to a regulator. In that case, running two separate instances of the entire pipeline in different regions can be worth the cost to protect against extended regional unavailability.
Batch ingestion often combines streaming capture with scheduled transformation. Raw user events flow from the customer source system to Pub/Sub topics, then a Dataflow streaming pipeline writes successful events into BigQuery raw event tables and failures into separate raw failed tables. Then Workflows triggers an hourly scheduled Dataflow pipeline that reads the raw events, transforms them into the required format, writes them to BigQuery curated tables, and creates or updates a view for validation and import. Cloud Storage provides temporary storage for Dataflow during processing and archiving for failed event logs.
Event-driven architecture with Pub/Sub is a design where publishers broadcast events once and multiple subscribers react to them independently. The Pub/Sub service delivers each event to all services that react to it, so the same raw or processed event can feed BigQuery, Bigtable, Cloud Storage, and other applications at the same time.
The advantages for data workflows include:
The limitations are cost and complexity tradeoffs. Continuous streaming requires always-on ingestion and processing capacity, and adding full cross-region redundancy for real-time analytics is likely not worth the cost when Pub/Sub and Dataflow already make the path resilient. Batch processing avoids some of that cost by recreating pipelines and restarting jobs after disruption, with the core focus on recovering the actual data.
Choosing between real-time and batch depends on freshness needs, analytic use, and tolerance for delay. Real-time fits when results must reflect what is happening now, while batch fits when results must be complete and validated by a deadline.
| Option | Data freshness | Choose when |
|---|---|---|
| Real-time ingestion | Continuous, near-immediate | Results must reflect current activity, such as live dashboards, trend detection, or event-sequence investigation |
| Batch processing | Scheduled, such as hourly or daily | Results must be complete and validated by a deadline, such as a regulator report or curated hourly import |
Exam tip: Choose real-time Pub/Sub to BigQuery flow when latency is critical and data must be queryable right away; choose batch when a fixed deadline and validation matter more than immediacy.
Event-driven architecture is a design where producers emit events as they happen and consumers react to those events without direct coordination. Pub/Sub (opens in a new tab) captures and distributes events in real time and BigQuery (opens in a new tab) stores those events for immediate SQL analysis, so fresh data is available for dashboards and queries instead of waiting for the next batch load.
How Pub/Sub decouples publishers and subscribers is through asynchronous messaging with shared topics and independent subscriptions. Event producers, called publishers, send events to the Pub/Sub service without knowing how or when the events will be processed. Event consumers, called subscribers, receive events from subscriptions and process them independently, with typical latencies on the order of 100 milliseconds.
This asynchronous model increases flexibility and robustness compared to synchronous remote procedure calls (RPCs). In RPC systems, a publisher must wait for each subscriber to receive the data, so a slow subscriber slows the publisher. In Pub/Sub, data flows from publishers to a shared topic and then from the topic to each subscriber, so publishers and subscribers can scale and fail independently.
Real-time ingestion from Pub/Sub to BigQuery is required when decisions, alerts, or user experiences depend on data that is seconds old rather than hours old. Traditional batch processing collects data and loads it on a schedule, which delays visibility until the job runs. Choose the event-driven path when many sources produce continuous events, multiple applications need the same events, and waiting for a batch window would miss the business need.
Common real-time scenarios include:
The ordered pipeline from Pub/Sub to BigQuery moves data from event sources to analytics storage through dependent stages. Log events from endpoints flow first into Pub/Sub, then through stream processing, then into BigQuery tables where the data can be queried and visualized.
A five-step process flow showing event sources publishing to a Pub/Sub topic, delivery through a subscription that buffers messages, processing by a Dataflow pipeline, writing to BigQuery via the Storage Write API, and ad hoc SQL queries and dashboards in BigQuery.
The pipeline depends on each prior stage: BigQuery depends on Dataflow for transformed rows, Dataflow depends on the Pub/Sub subscription for delivery of buffered events, and the subscription depends on the topic for published events.
Why event data scales better than command messages is that events describe what happened, while commands tell the receiver what to do. A command message might say <SetInventoryLevel> to 3001, which tells the consumer to set state to a precalculated value. An event message says { "inventory": -1 }, which records that inventory decreased by one and leaves each subscriber to decide how to apply that fact.
This difference changes how order affects correctness. In a command model, receiving messages out of order produces the wrong final state because the last value wins. In an event model, each change is additive, so the final total is correct even when messages arrive late or out of order:
| Model | What message contains | Result when messages arrive out of order |
|---|---|---|
| Message-queue commands | Precalculated target state, such as setInventory(5) then setInventory(2) | Final inventory is wrong, because the delayed setInventory(5) overwrites the newer value |
| Pub/Sub events | Change that occurred, such as "inventory": +1 and "inventory": -3 | Final inventory stays correct at 2 from a starting value of 4, because +1 and -3 sum the same in any order |
Because event messages are not imperative commands, strict first-in-first-out (FIFO) ordering becomes less important. Publishers only need to understand their own actions, subscribers manage their own load, and new subscribers can attach to a subscription without changing the originating system.
Advantages over traditional batch processing come from continuous availability and parallel delivery instead of periodic loads. Events flow from publishers to BigQuery within about 100 milliseconds, while batch jobs delay data until the next scheduled run. Pub/Sub also gathers events from many clients at once and uses per-message parallelism, where individual messages are leased to subscriber clients in parallel.
Per-message parallelism maximizes subscriber throughput compared to partition-based systems that force ordered processing within each partition and limit concurrency to the number of partitions. Additional operational advantages include independent scaling of publishers and subscribers, the ability to reuse the same topic for new consumers without duplicating data, and built-in buffering that protects freshness when processing pauses.
Exam tip: Choose Pub/Sub to BigQuery when the requirement names streaming analytics, logging, monitoring, or IoT with near-real-time dashboards; Pub/Sub provides per-message parallelism, not partition-limited parallelism.
BigQuery subscriptions (opens in a new tab) are a type of Google Cloud Pub/Sub subscription that writes messages from a topic directly into a BigQuery table without an intermediate processing service. Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that decouples event producers from event consumers, with latencies typically on the order of 100 milliseconds. This direct path makes streaming data available for SQL queries, dashboards, and real-time analytics almost as soon as publishers send it.
How direct streaming works is through a fixed flow from publisher to topic to BigQuery subscription to BigQuery table. Publishers send events to the Pub/Sub service without waiting to learn how or when the events will be processed, and Pub/Sub then delivers those events to the BigQuery subscription that reacts to them. Because publishers and subscribers communicate asynchronously rather than through synchronous remote procedure calls (RPCs), publishers do not wait for BigQuery to receive the data, which increases flexibility and robustness.
Data flows from the publisher to BigQuery in a fixed order:
The BigQuery subscription depends on two configurations: the Pub/Sub topic that collects the events and the destination BigQuery table that stores them. Message fields must align with the destination table so rows can be written and queried cleanly.
When to use direct delivery versus Dataflow depends on whether events need processing before storage. A BigQuery subscription is the simplest path from Pub/Sub to BigQuery because no Dataflow pipeline must be built or managed. Dataflow is the stream-processing option that reads from Pub/Sub, transforms or joins data, and then loads results into BigQuery, Bigtable, or Cloud Storage.
| Path | Processing | Choose when |
|---|---|---|
| BigQuery subscription | Writes topic messages directly to a BigQuery table | Low latency and low operational overhead matter most and little or no transformation is needed |
| Dataflow pipeline | Reads from Pub/Sub, processes data, then loads to BigQuery and other storage | Events need transformation, SQL joins, filtering, or delivery to multiple systems |
Choose direct delivery when events can be stored as received, and choose Dataflow when events need cleaning, enrichment, or routing to multiple destinations.
Exam tip: Use a BigQuery subscription for direct Pub/Sub to BigQuery streaming, and use Dataflow when the flow requires processing before BigQuery.
Use cases for direct streaming to BigQuery are workloads where many sources generate continuous events that must be gathered at once and analyzed in near real time. Pub/Sub gathers events from many clients simultaneously, so spikes in traffic do not require publishers to coordinate with each other or with BigQuery. In each case, data flows from many publishers to one topic to one BigQuery table, so analysts query a single, fresh dataset instead of waiting for a batch load.
Common patterns for this direct streaming model include:
Use real-time ingestion from Pub/Sub to BigQuery when results must reflect current activity, such as live dashboards, trend detection, or event-sequence investigation. Use batch processing when results must be complete and validated by a deadline, such as an hourly curated import or a daily regulator report.
Use a BigQuery subscription when low latency and low operational overhead matter most and little or no transformation is needed, because it writes topic messages directly to a BigQuery table without an intermediate processing service. Use a Dataflow pipeline when events need transformation, SQL joins, filtering, enrichment, or delivery to multiple systems such as BigQuery, Bigtable, and Cloud Storage.
Event data in Pub/Sub records the change that occurred, such as inventory plus one and minus three, so the final total stays correct because the changes sum the same in any order. Command messages in a message queue contain a precalculated target state, such as setInventory(5) then setInventory(2), so a delayed message overwrites the newer value and produces the wrong final state.
A retail company is designing a data ingestion pipeline to process customer purchases and inventory adjustments. The architecture must satisfy the following requirements:
Which architectural approach best addresses these requirements?
Implement an hourly cron job that exports CSV dumps from local servers to Cloud Storage and triggers BigQuery load jobs.
Implement a scheduled batch extraction pipeline that executes daily SQL queries against transactional databases and loads the delta into BigQuery.
Implement an event-driven architecture using Pub/Sub to publish event messages asynchronously and route them to BigQuery subscriptions.
Implement a centralized message-queue architecture that sends synchronous state-change commands directly to target database endpoints.
Last updated