Pub/Sub Ingestion and Bulk Data Transfer

Designing streaming ingestion architectures means using Cloud Pub/Sub (Pub/Sub) as a scalable and durable event ingestion service that feeds real-time pipelines built on Cloud Dataflow (Dataflow) (opens in a new tab). In this pattern, publishers send event data into Pub/Sub topics, Dataflow pipelines subscribe to those topics, and Dataflow workers process data in parallel before writing results to sinks. Because Pub/Sub handles ingestion scaling, pipeline design focuses on quotas, flow control, encryption, and residency.

1.1Pub/Sub as the Streaming Ingestion Backbone

Cloud Pub/Sub is the managed backbone that decouples event producers from streaming processors. Data flows from publishers to a Pub/Sub topic, then from the topic to subscriber pipelines such as Dataflow, so ingestion can continue even when processing speed or downstream systems vary. Using Google Cloud managed sources and sinks removes much capacity-planning work because Pub/Sub provides consistent performance within quotas and limits that fit most requirements. You still must monitor quota consumption in the Google Cloud console and request a quota increase when a single pipeline is expected to exceed per-project quotas.

Scalability of a streaming pipeline also depends on factors outside Pub/Sub and Dataflow code:

• Scalability of external systems: an external system can limit throughput, such as an Apache Kafka topic with too few partitions for the required read rate.
• Choice of data formats: formats that support parallel reads, such as Avro, are usually faster than CSV files with embedded newlines or compressed files.
• Data location and network topology: geographic proximity and networking between sources, the pipeline, and sinks affect performance.

1.2Scaling Throughput with Project-Level Quotas

Project-level quotas control how much streaming throughput is available to publishers and subscribers. Publishers and subscribers in different Google Cloud projects receive independent data-throughput quotas, so throughput used by one pipeline does not reduce throughput for a pipeline in another project. If multiple pipelines publish or subscribe to a single topic, deploying each pipeline into its own project yields the maximum allowable throughput on that topic. When a single pipeline is expected to exceed per-project quotas, request a quota increase before launch and track consumption in the console.

1.3Reliability for Spikes and Failed Publishes

Reliability for spikes and failed publishes depends on controlling publish rate and retrying transient failures between publishers and Pub/Sub. To manage transient spikes in message traffic, configure flow control in the publisher settings, which limits the rate of publish requests and avoids errors during bursts. To handle failed publishes, adjust the retry-request variables so transient failures are retried rather than lost. Dataflow itself runs on Compute Engine virtual machine instances, so insufficient Compute Engine quota can also prevent jobs from starting or hinder autoscaling when load increases.

1.4Security and Data Residency for Messages

Security and data residency for Pub/Sub messages combine default encryption with customer-controlled keys and storage controls. Cloud Pub/Sub encrypts all messages by default both at rest and in transit using Google-owned and Google-managed encryption keys. For application-layer control, Pub/Sub supports customer-managed encryption keys (CMEKs), and the service depends on Cloud Key Management Service when you need to control the keys. If residency rules require message data to stay in specific locations, configure message storage policies to restrict where Pub/Sub stores message data.

1.5Delivery from Subscriptions to Processing and Storage

Delivery from subscriptions carries data from a Pub/Sub topic through a subscription to the processing engine or storage sink. Dataflow pipelines read data from Pub/Sub sources and write data to sinks, and they may also call external systems for enrichment, filtering, or business logic within a processing step. As a direct sink example, a Bigtable subscription sends messages from a Pub/Sub subscription to a Cloud Bigtable (Bigtable) table, where Pub/Sub writes each message as a new row. Because Dataflow parallelizes work across workers, batch multiple elements into single requests when calling external services and ensure those services can sustain the load, since autoscaling can otherwise add pressure and exhaust quotas.

Securing and monitoring ingestion pipelines means protecting data as it moves from on-premises, multi-cloud, or Google Cloud sources into services such as BigQuery (opens in a new tab), Cloud Storage, and Pub/Sub, while making every access auditable. Data flows from the source through an ingestion path using Cloud Storage or Pub/Sub, then through a Dataflow pipeline for de-identification, and finally into separate BigQuery warehouses for confidential and non-confidential data. Security controls apply at each boundary, with encryption protecting content, Identity and Access Management (IAM) controlling who can act, network perimeters controlling where data can move, and logging recording what happened.

2.1Encryption and Key Management for Ingestion

Encryption and key management for ingestion combine Google-managed default encryption with customer-managed encryption keys (CMEKs) managed in Cloud Key Management Service (Cloud KMS) with Cloud HSM. By default, Cloud Storage encrypts data with Google-owned keys and Pub/Sub encrypts all messages both at rest and in transit, while Pub/Sub and BigQuery also support CMEKs for application-layer control. When CMEKs are used, one CMEK for ingestion is shared by the Dataflow pipeline and Pub/Sub, plus separate CMEKs protect the BigQuery warehouse in the non-confidential project and the warehouse in the confidential project. The CMEK location determines where the key is stored and must be in the same location as the protected resource, and by default the CMEK is rotated every 30 days.

For external ingestion from on-premises or another cloud, client-side encryption adds protection before data leaves the source network. The flow uses envelope encryption with the Tink library, which is compatible with BigQuery Authenticated Encryption with Associated Data (AEAD):

1. Generate key-encrypting key: Create a symmetric CMEK keyset as the key-encrypting key (KEK) in Cloud HSM, where it is managed with IAM roles.
2. Generate and wrap data-encryption key: Generate a data-encryption key (DEK) locally or from Cloud HSM, then wrap the DEK with the KEK.
3. Encrypt and upload: Encrypt sensitive columns locally with the DEK, then upload the encrypted data and the wrapped DEK to BigQuery.
4. Store separately: BigQuery stores the encrypted data in one table and the wrapped DEK in a second table.
5. Decrypt on demand: When an authorized analyst queries the data, BigQuery unwraps the DEK with the KEK and decrypts the column with AEAD decryption.

If compliance requires keys to live outside Google Cloud, Cloud External Key Manager (Cloud EKM) can be enabled, but the customer then owns key management activities including rotation. The choice between default encryption, CMEK control, and application-layer tokenization depends on compliance and operational cost:

2.2Identity and Access Control with Least Privilege

Identity and access control with least privilege means users, groups, and service accounts receive only the no-access, read-only, or write permissions needed for their role. This limit reduces data theft, preserves model and pipeline integrity, and shrinks the surface for accidents or attacks when sensitive data sits in lakes, warehouses, and pipeline staging. Granularity is implemented by mapping IAM roles to identities by project need, access pattern, or resource tags, and by adding IAM Conditions for attribute-based control, such as allowing a service account to trigger a pipeline only when the request originates from a trusted Cloud Build trigger.

Pipeline identities depend on dedicated, least-privilege service accounts rather than broad default accounts. A training or ingestion pipeline service account, for example, has permission to read from only a specific Cloud Storage bucket and to write model artifacts or warehouse tables to a designated destination. Further limits come from principal access boundary (PAB) policies that cap what a principal can access, and from Privileged Access Manager for just-in-time temporary elevation whose activity is later visible in audit logs. Organization policies reinforce this posture by disabling service account creation, disabling service account key creation, and disabling automatic role grants to default service accounts in ingestion projects.

Fine-grained data access in BigQuery depends on policy tags, row-level security, and column-level security layered on dataset IAM roles. Policy tags define a hierarchy where access to a higher tag implies access to lower tags but not the reverse: the 3_Confidential tag covers very sensitive columns such as credit card numbers and also grants 2_Private and 1_Sensitive; the 2_Private tag covers PII such as first name and also grants 1_Sensitive but not 3_Confidential; the 1_Sensitive tag covers non-public data such as credit limit with no access to higher tags. Columns without tags remain available to all users with warehouse access, and analysts need the Fine-Grained Reader role to read protected columns even after re-identification. In Cloud Storage, IAM at bucket and project level is preferred over access control lists (ACLs), with IAM policies also supported on buckets and managed folders.

2.3Network Perimeters and Secure Data Movement

Network perimeters and secure data movement create a security boundary around Virtual Private Cloud (VPC) resources so data cannot be exfiltrated to the internet or unauthorized networks. VPC Service Controls defines a service perimeter that includes AI, ML, and warehouse resources and sensitive data, with ingress and egress rules controlling data flow across the perimeter. Private Service Connect lets pipelines communicate with Google Cloud APIs using private IP addresses at the API and IP layers, while VPC networks with private IP addresses remove internet exposure and firewall rules deny all traffic by default and explicitly allow only matching criteria.

Ingestion from external networks depends on an encrypted connection plus allowlisted access before any data moves. Cloud VPN or Cloud Interconnect protects all traffic between the external environment and Google Cloud, with Dedicated Interconnect recommended for high-throughput streaming because it provides a direct connection. Access from the external environment is permitted only from allowlisted IP addresses defined in access levels policy rules. Once inside the perimeter, sharing across privacy boundaries uses governed paths such as BigQuery sharing with Analytics Hub and BigQuery data clean rooms rather than open export.

2.4Monitoring, Logging, and Governance for Auditability

Monitoring, logging, and governance for auditability combine real-time pipeline telemetry with audit trails that record who accessed or changed data and when. Cloud Logging provides visibility into system activity and live tailing of pipeline logs, while Cloud Monitoring tracks key signals and events that can reveal suspicious activity or performance degradation. Admin Activity audit logs record API calls that modify configuration or metadata, and Data Access audit logs track reads of configuration or user-provided data; both can be centralized with log sinks into Observability Analytics for investigation or into Cloud Storage for long-term retention, and log data can be regionalized to meet residency requirements. Sensitive or PII data must not be written into experiment logs or scattered log servers.

Governance and integrity checks depend on Knowledge Catalog (formerly Dataplex) as the organization-wide platform for discovery, lineage, quality, and access. Knowledge Catalog manages table and column-level lineage, data quality checks and data profiles, business glossary vocabulary, and unified governance across BigLake and open formats such as Iceberg and Delta Lake. During ingestion, Sensitive Data Protection discovers, classifies, and de-identifies data using record transformations for structured fields such as credit card numbers and PINs and information types (infoTypes) for unstructured content, with a Dataflow pipeline tokenizing confidential data from Cloud Storage into BigQuery. Security Command Center then detects misconfigurations and malicious activity, with IAM Recommender suggesting least-privilege reductions and Firewall Insights flagging overly permissive rules.

2.5Reliability Controls for Ingestion Sources

Reliability controls for ingestion sources depend on the storage and messaging guarantees of the source services plus explicit handling of traffic spikes and failed publishes. Cloud Storage buckets can be regional, dual-region, or multi-region, where regional data is replicated synchronously across zones and dual-region or multi-region data is replicated asynchronously across regions for higher availability. BigQuery stores loaded data synchronously in two zones within the specified region so a zone outage does not lose data. Pub/Sub manages transient spikes by configuring flow control in publisher settings to limit publish request rates, and failed publishes are handled by adjusting retry-request variables.

Data residency and message durability further constrain where ingestion data is kept. Cloud Storage stores or replicates data only within the specified regions, Pub/Sub message storage policies restrict message data to specific locations, and Cloud Logging regionalization keeps log data in the specified region. Together these controls let a pipeline owner place compute, keys, data, and logs in the same compliant geography while retaining the redundancy needed for continuous batch and streaming ingestion.

Evaluating bulk and batch data transfer mechanisms means choosing how to move large, existing datasets from on-premises, vendor, or other-cloud sources into Google Cloud without relying on continuous streaming. Transfer Appliance (opens in a new tab) provides offline physical transfer to Cloud Storage and BigQuery, while the BigQuery Data Transfer Service (BQ DTS) provides managed batch transfer from data warehouses directly into BigQuery. Cloud Storage acts as the staging layer for file-based loads into BigQuery when raw files must be uploaded, validated, and then loaded.

3.1Offline Bulk Transfer with Transfer Appliance

Transfer Appliance is an offline, high-capacity device for capturing very large datasets locally and shipping the device to Google for upload. A single appliance can capture 300 terabytes of data in under 25 days, and data can typically be accessed in Cloud Storage within another 10 business days, all without consuming any outbound network bandwidth. Rehydration time depends on data quality, and transfers with very small files take a long time to rehydrate. Choose Transfer Appliance when network bandwidth is limited, outbound costs are high, or data volume makes online transfer impractical.

Data flows from the source environment to the destination through an ordered physical workflow:

1. Capture: Data is copied from researchers, vendors, on-premises systems, or other sites directly to the appliance on the local network.
2. Ship: The completed appliance is shipped back to Google for upload.
3. Upload: Google uploads the appliance contents to a Cloud Storage bucket in the region you specify.
4. Access: Data flows from Cloud Storage to BigQuery, Dataflow, or machine learning services for replication, migration, or analysis.

This workflow supports three related patterns. Data collection moves distributed inputs into one Cloud Storage or BigQuery location for processing. Data replication copies on-premises data to Google Cloud to test hybrid architectures while existing operations continue. Data migration moves backup images and archives to low-cost durable classes such as Archive Storage for long-term retention and analysis.

3.2Batch Warehouse Transfer with BigQuery Data Transfer Service

The BigQuery Data Transfer Service (BQ DTS) is a managed service that loads batch data from external warehouse and application sources into BigQuery on a one-time or recurring schedule. The service includes a Snowflake connector that can transfer selected tables, automatically detect table schema and data types during transfer, and configure incremental transfer that moves only data changed since the last transfer instead of reloading the full dataset. The broader BigQuery Migration Service extends the same transfer approach to Amazon Redshift, Hadoop, Hive, Apache Spark, IBM Netezza, Oracle and Oracle Exadata, Snowflake, and Teradata, combined with assessment and SQL translation from source dialects to GoogleSQL. Choose BQ DTS when the source is a structured warehouse and the destination is BigQuery tables that must stay current through scheduled batch loads.

For file-based warehouse migration, BigQuery relies on Cloud Storage for staging raw data files. Data flows from the source system to Cloud Storage for upload and download of raw files, and then from Cloud Storage into BigQuery for loading. BigQuery also supports export back to Cloud Storage, so the same bucket path can be used to validate round-trip movement and retain source files.

3.3Selecting by Bandwidth, Volume, and Source Location

Selecting by bandwidth, volume, and source location determines whether offline shipment or network-based batch transfer fits the move. Transfer Appliance depends on physical shipment and regional upload location, while BQ DTS depends on network connectivity to the source warehouse and BigQuery destination region. For customers in the European Union, appliances are shipped from Belgium and returned to Belgium for upload, and if the destination Cloud Storage region is within the EU, data never leaves the boundaries of the European Union during any part of the transfer.

3.4Preserving Data Integrity During Transfer

Preserving data integrity during transfer means the destination tables in BigQuery match the source in schema, types, and row content after a bulk or batch move. The open-source data validation tool compares source and destination tables after migration to confirm that they match, and the Snowflake connector reduces schema drift by auto-detecting schema and data types at transfer time. For SQL and schema migration, translated queries should be checked with a wide range of test cases across scenarios and data, then run in both Snowflake and BigQuery to compare results and fix differences before cutover. Together, automated validation, schema detection, and result comparison ensure that large offline and batch transfers remain complete and query-ready.

Key Facts

• Cloud Pub/Sub throughput quotas are independent per Google Cloud project, so pipelines that share a topic get maximum throughput when each pipeline runs in its own project.
• Pub/Sub publisher flow control limits publish request rates during traffic spikes, while retry-request settings control how failed publishes are retried.
• A single Transfer Appliance can capture 300 terabytes in under 25 days without using outbound network bandwidth, with data typically available in Cloud Storage within another 10 business days.
• The BigQuery Data Transfer Service loads warehouse tables into BigQuery on a scheduled basis and supports incremental transfer of only data changed since the last transfer.