Google Cloud Data Sources and Sinks

Securing and structuring data access and network pathing is the practice of controlling who can reach data, which network paths ingestion and egress can use, and how data stays protected in transit and at rest. In Google Cloud, Identity and Access Management (IAM) (opens in a new tab) controls which identities can act, VPC Service Controls (opens in a new tab) controls whether data can move across a service perimeter, and encryption protects content even if storage or network layers are compromised. Together these controls let pipelines ingest from internal or external sources and share results without exposing confidential data to the internet or to overly broad identities.

1.1Role-Based Access With Least Privilege

Role-based access with least privilege is the IAM model where each user, group, or service account receives only the minimum permissions needed for its work. Permissions start from no access and expand only to required read or write actions, which limits theft, accidents, and blast radius for sensitive data in lakes, warehouses, and pipelines. Dedicated service accounts extend this pattern to automation, such as a training pipeline account that can read from only one bucket and write only to Model Registry.

Granular enforcement adds context and time limits without creating broad custom roles. IAM Conditions add attribute-based rules to a grant, principal access boundary policies cap what an identity can access even if other policies grant more, and Privileged Access Manager provides just-in-time temporary elevation that appears in audit logs. IAM Recommender analyzes actual usage and suggests reductions to preserve least privilege over time:

• Map product IAM roles to a user, group, or service account by project need, access pattern, or resource tags for fine-grained scoping.
• Apply IAM Conditions to data, models, and configurations such as code, resource settings, and hyperparameters.
• Enforce principal access boundary policies to bound maximum permissions and use Privileged Access Manager for audited temporary elevation.
• Extend Identity-Aware Proxy with context-aware access to restrict by IP address and end-user device attributes.

Access evaluated at authentication time adds device and network context to identity. Access Context Manager defines access levels, such as allowlisted IP addresses, and Chrome Enterprise Premium evaluates those levels each time a user authenticates to a Workbench instance. The request therefore passes both identity checks in IAM and context checks before reaching data or pipeline resources.

1.2Structured Access to Buckets, Datasets, Rows, and Columns

Structured access to buckets, datasets, rows, and columns is the layered model that places sensitive data in its own governance boundary and then narrows visibility inside that boundary. Cloud Storage (opens in a new tab) enforces IAM policies on buckets and managed folders, while BigQuery (opens in a new tab) enforces IAM roles on datasets and resources within datasets. BigQuery then adds row-level security and column-level security, including policy tags and dynamic data masking, so analysts can query a table without seeing confidential columns or unauthorized rows.

Separation of duties maps these controls to personas such as data engineers, plaintext readers, encrypted-data viewers, and data analysts. An encrypted-data viewer can see ciphertext and metadata, while only a plaintext reader with key and column access can decrypt and view confidential values. Organization Policy reinforces the model by disabling service account creation, disabling service account key creation, disabling automatic grants to default service accounts, and requiring OS Login where applicable.

1.3Encryption at Rest and in Transit

Encryption at rest and in transit in Google Cloud protects stored data by default and adds customer-controlled keys when compliance requires it. Data at rest is encrypted by default with Google-owned and Google-managed keys, with customer control added through customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) (opens in a new tab). CMEK-integrated services including BigQuery, Cloud Storage, Cloud SQL, Spanner, Pub/Sub, Cloud Run, and Vertex AI encrypt stored data with keys the customer can revoke or rotate. An organization policy such as gcp.restrictNonCmekServices set to bigquery.googleapis.com can require CMEK protection so noncompliant datasets cannot be created.

Data in transit is protected with HTTPS for API calls to AI, ML, and data services, enforced at the edge with HTTPS load balancers. Pub/Sub encrypts messages both at rest and in transit by default and supports CMEKs for application-layer message encryption. Cloud Logging and Cloud Storage support regionalized storage so encrypted data and logs remain in the specified region to meet residency requirements.

For external ingestion, client-side envelope encryption protects sensitive columns before data ever reaches Google Cloud. The Tink library encrypts data locally with a data encryption key (DEK) and wraps that DEK with a key encryption key (KEK) that is a symmetric CMEK stored in Cloud HSM and governed by IAM. Tink is compatible with BigQuery Authenticated Encryption with Associated Data (AEAD) functions, so BigQuery can later unwrap the DEK with the KEK and decrypt the column on demand:

1. Generate KEK: Create a symmetric CMEK keyset in Cloud HSM where IAM controls who can use the key.
2. Generate DEK: Generate a DEK locally or from Cloud HSM for the sensitive column or dataset.
3. Encrypt and wrap: Encrypt the column data with the DEK, then wrap the DEK with the KEK using Tink.
4. Store separately: Load encrypted data into one BigQuery table and the wrapped DEK into a separate table.
5. Decrypt on demand: Call BigQuery AEAD decryption to unwrap the DEK with the KEK and reveal plaintext only to authorized readers.

1.4Network Isolation and Service Perimeters

Network isolation and service perimeters keep pipeline traffic off the internet and inside explicit boundaries, with each control operating at a different layer. A Virtual Private Cloud (VPC) network with private IP addresses removes internet-facing exposure for compute and data resources. Private Service Connect lets resources communicate with Google Cloud APIs using private IP addresses, so ingestion and processing do not traverse the public internet. Firewall rules then restrict inbound and outbound traffic, starting from a deny-all default and explicitly allowing only required ports, protocols, and sources.

VPC Service Controls creates a service perimeter around sensitive resources, such as BigQuery datasets, storage buckets, and ML resources, to prevent data exfiltration from the VPC. Data flows freely among member projects and services inside the perimeter, but any flow crossing the perimeter boundary must match an explicit ingress or egress rule. Access levels define which source networks or identities may cross, such as allowlisted IP addresses for an on-premises environment, while Organization Policy can constrain ingress further, such as Cloud Run functions set to ALLOW_INTERNAL_AND_GCLB.

The following comparison summarizes how the three network controls differ in what they bound and how traffic crosses them:

The controls compose rather than replace each other. IAM determines whether an identity may invoke an operation, the firewall determines whether packets may reach the resource, and VPC Service Controls determines whether data may leave the perimeter even when both IAM and network checks pass.

1.5Secure Ingestion Paths From External Sources

Secure ingestion paths from external sources combine a private encrypted connection with client-side encryption and allowlisted access. Cloud VPN protects smaller flows, while Dedicated Interconnect provides a direct private connection with high throughput for large streaming or batch loads into BigQuery. The perimeter access level allowlists the on-premises IP addresses, so only defined network sources can reach the ingestion projects and services.

The ingestion workflow depends on keys, network, and identity in order. First the team provisions Dedicated Interconnect and creates Cloud KMS keys in Cloud HSM, then it installs Tink and generates encrypted keysets with Application Default Credentials. Data is encrypted locally with deterministic or randomized encryption, uploaded by streaming or batch into BigQuery tables inside the confidential perimeter, and decrypted later with AEAD only by holders of both BigQuery access and KMS decrypt permission. This order ensures that plaintext never traverses the external network and that compromise of storage alone does not expose sensitive columns.

1.6Controlled Egress and Sharing Across Privacy Boundaries

Controlled egress and sharing across privacy boundaries is the governed way to share results without opening broad export paths from the perimeter. Analytics Hub in BigQuery shares datasets across projects and organizations under a security and privacy framework, and BigQuery data clean rooms allow analysis across privacy boundaries without copying raw confidential rows to consumers. Looker Action Hub sends dashboard results directly into built-in destinations from a secure cloud environment instead of requiring ad hoc downloads.

Auditability closes the loop on both ingestion and egress. Admin Activity audit logs are enabled by default and record configuration changes, while Data Access audit logs can be enabled to track reads and writes of user-provided data and resource metadata. Regionalized Cloud Logging keeps those logs in the required region, so compliance teams can verify who accessed or modified data, when access occurred, and whether any flow attempted to cross a perimeter boundary.

Evaluating operational characteristics of streaming and batch sources and sinks means matching pipeline entry and exit points to throughput, latency, and scalability needs. In Google Cloud, the data storage layer holds data as files or managed records, while the data computation layer loads data from storage, processes it, and saves results to a target system that can be the same source layer. Keeping these layers on separate platforms in the same zone and region reduces communication latency, outbound transfer cost, and I/O operations while preserving flexibility to change compute engines and to stop compute when it is not running.

2.1Separated Storage and Computation Layers

Separated storage and computation layers let the same durable data serve multiple short-lived processing engines without coupling their lifecycles. Data flows from the storage service to the computation service for processing, then data flows from the computation service back to a storage service as the sink, which may be the original source. To preserve performance, storage and compute should be deployed on separate platforms but in the same zone and region, for example by moving storage from Hadoop Distributed File System (HDFS) to Cloud Storage (opens in a new tab) while running computation on a Managed Service for Apache Spark cluster.

Common Google Cloud storage layer options include:

• Cloud Storage for object files managed as bytes with gs:// URIs
• BigQuery storage for analytical warehouse tables
• Cloud SQL and Spanner for relational operational data
• Bigtable for wide-column serving workloads

Common computation layer options include Dataflow, Managed Service for Apache Spark (formerly Cloud Dataproc), BigQuery, custom workloads on Google Kubernetes Engine (GKE) or Compute Engine, and notebooks on Gemini Enterprise Agent Platform or Colaboratory.

2.2Cloud Storage as a Batch Source and Sink

Cloud Storage as a batch source and sink is the primary landing point for raw files and bulk processing. A pipeline lands raw data into a bucket in the same region as the processing cluster, then a Spark job reads the data through the Cloud Storage connector and writes curated results back to Cloud Storage or onward to a warehouse. This pattern supports batch file uploads, scheduled recurring loads from Cloud Storage to BigQuery, event-driven Cloud Storage triggers, orchestration with Managed Airflow, or a pull model that retrieves files with protocols such as SFTP.

Operationally, Cloud Storage scales for high-throughput bulk reads and writes but with higher access latency than messaging, so bucket placement and class control cost and reliability. Regional buckets synchronously replicate data across multiple zones within a region, while dual-region and multi-region buckets asynchronously replicate data across regions for higher availability. Access is controlled with Identity and Access Management (IAM) at bucket and project levels in most cases, with access control lists (ACLs) as an alternative, and Sensitive Data Protection can discover, classify, and de-identify sensitive data loaded through ingestion. Cloud Storage stores or replicates data within the specified region to meet data residency requirements, and storage class selection should follow retention and access-frequency needs.

2.3Pub/Sub as a Streaming Source and Sink

Pub/Sub (opens in a new tab) as a streaming source and sink is the entry and exit point for continuous, high-volume event streams such as IoT sensor data, application logs, and database change events. Data flows from publishers to a Pub/Sub topic, then data flows from the topic to streaming subscribers such as Dataflow or a Managed Service for Apache Spark job, which processes the stream and lands results into the bronze tier in Cloud Storage. For database synchronization, a Change Data Capture (CDC) flow publishes row changes to a Pub/Sub topic, the Spark job subscribes to that topic, and the job applies the processed changes to the raw datastore in Cloud Storage.

Pub/Sub is built for low-latency delivery and elastic scale for many concurrent publishers and subscribers, in contrast to the bulk-oriented profile of Cloud Storage. By default, Pub/Sub encrypts all messages at rest and in transit with Google-owned and Google-managed keys and supports customer-managed encryption keys (CMEKs) for message encryption at the application layer. When residency matters, message storage policies constrain where message data is stored to specific locations.

2.4BigQuery and Warehouse Sinks

BigQuery (opens in a new tab) and warehouse sinks are the analytical endpoints for batch and ELT pipelines that load data first and transform it later with SQL. Data flows from Cloud Storage to BigQuery through scheduled Cloud Storage transfers, from software-as-a-service (SaaS) apps and sources such as Teradata or Amazon Redshift through BigQuery Data Transfer Service (DTS) on a scheduled managed basis, or directly from Hadoop and Spark through the BigQuery connector. The BigQuery connector is a Java library that lets Hadoop and Spark write directly to BigQuery using abstracted Apache Hadoop InputFormat and OutputFormat classes.

ELT migration separates the extract-and-load behavior from the SQL transform behavior, so DTS can replace the load portion when the source is supported directly or through third-party integrations. Managed Service for Apache Spark (opens in a new tab) favors many ephemeral clusters over one monolithic cluster, which allows different configurations per job, scaling per job, payment only while jobs run, fresh configuration on each run, and reuse of the same definitions for development, testing, and production. Incremental migration isolates each legacy Hadoop, Spark, or Flink job, evaluates its dependencies against supported Managed Service for Apache Spark versions, and validates outputs when versions or configurations change.

2.5Selecting Entry and Exit Points

Selecting entry and exit points is the decision process that matches arrival pattern, downstream access, and operational constraints to the right source or sink. Batch pipelines enter through Cloud Storage files and exit to Cloud Storage or BigQuery, while real-time pipelines enter through Pub/Sub or Apache Kafka topics and exit to streaming processors before landing. Assessment requires inventorying storage layers, computation layers, datasets, network latency between source and target environments, custom configurations and versions, and cluster sizes including node count, CPU, and memory.

The following comparison summarizes supported entry and exit behavior for the main Google Cloud options:

Network latency testing determines copy time and client-perceived delay when sources, sinks, or external systems span environments. Automated deployment preserves framework versions and configurations across migration, and profiling after deployment accounts for processor or regional changes that affect serialization, computation, and output.

Architecting schema evolution and data serialization across pipelines is the practice of choosing file formats that preserve schema from sources, staging them in an intermediate sink, and enforcing that schema when loading into a structured sink. Data flows from diverse sources such as on-premises warehouses, Amazon Redshift, Teradata, or Amazon S3 through Cloud Storage (opens in a new tab) into BigQuery, with Dataflow, Cloud Data Fusion, and Managed Service for Apache Spark available to transform data in flight. Keeping the initial schema unchanged reduces pipeline rework while later evolution uses staging tables and views to decouple producers from consumers.

3.1Serialization Formats That Carry Schema With Data

Serialization formats that carry schema with data differ in whether the sink can interpret fields without external definitions. Apache Avro, Apache Parquet, and Apache ORC embed schema information with the data, which reduces transfer complexity because the sink can interpret fields without external definitions. Comma-separated values (CSV) and other delimited formats do not embed schema, so the schema must be specified separately at load time or inferred by auto-detection of file contents.

The following comparison shows how the main extract formats handle schema and where each format fits in a pipeline:

Choose Avro, Parquet, or ORC when moving data from diverse sources into structured sinks to preserve types and avoid manual schema mapping. Parquet and ORC are preferred when the Cloud Storage sink also serves as a data lake for analytics, because columnar layout lowers scan cost and speeds queries whose metadata is tracked in Dataproc Metastore.

3.2Staging Extracts in Cloud Storage as an Intermediate Sink

Staging extracts in Cloud Storage as an intermediate sink is the step that decouples extraction in the source environment from loading into BigQuery. The ordered flow is first, extract schema and data into files in staging storage in the existing environment; then, upload those files to a Cloud Storage bucket; only after upload, load from the bucket into BigQuery. Locating the bucket close to the source reduces network latency, and co-locating the bucket with the BigQuery dataset simplifies the final load.

The upload path depends on source location and network constraints. Data flows from another cloud through Storage Transfer Service (opens in a new tab), from on-premises with good bandwidth through the gcloud CLI with multi-threaded parallel uploads that resume after errors and encrypt traffic in transit, and without sufficient bandwidth through Transfer Appliance for offline transfer. Higher rates of change and large numbers of small files increase transfer time, so concurrent jobs and compression or bandwidth increases are used to meet recovery and freshness goals.

3.3Loading and Enforcing Schema in BigQuery

Loading and enforcing schema in BigQuery is the load-time decision between applying an explicit schema or auto-detecting schema from staged files. Explicit schemas provide strict control for CSV and for production tables, while auto-detection is useful for initial exploration but can misinterpret types. The choice matters because CSV has no embedded schema, so production loads need explicit definitions to avoid type errors.

Supported load paths from Cloud Storage into BigQuery include:

• BigQuery Data Transfer Service (opens in a new tab) for automated and recurring transfers, including direct Redshift and Teradata migration with Cloud Storage used for staging
• BigQuery Storage Write API (gRPC) for programmatic streaming and batch writes
• Batch load for one-time loads from Cloud Storage files

When transformation is required before the sink, the pipeline depends on an extract, transform, and load (ETL) engine between Cloud Storage and BigQuery. Data flows from the staging bucket through Dataflow with Apache Beam transforms and the BigQuery I/O connector, through Managed Service for Apache Spark with Spark SQL or custom code and the Hadoop BigQuery Connector, or through Cloud Data Fusion with preconfigured connectors to the BigQuery sink. An alternative is extract, load, and transform (ELT), where data flows from Cloud Storage into a BigQuery staging table and then SQL queries write transformed output into the final production table.

3.4Evolving Schema Without Breaking Producers and Consumers

Evolving schema without breaking producers and consumers is the strategy that lets existing writers keep writing while readers migrate to a new model. The strategy starts by transferring the schema without changes, so feeding pipelines need no adjustment, staff need no retraining, and automated tools can copy schema and data directly. Proofs of concept can then run against BigQuery in parallel while migration continues.

When the model must change, facade views in BigQuery decouple upstream writes from downstream reads. First, downstream processes are pointed at views while upstream tables remain on the source schema. Then, new tables are created on the evolved schema and view definitions are modified to query source tables, new tables, or a SQL UNION of both. Because views are read-only, upstream jobs write to tables while downstream jobs read from views, so each side can evolve independently. New tables can use nested and repeated fields to denormalize the model and match BigQuery columnar storage, and once all writers and readers use the evolved tables, source tables and then unneeded views are retired from the data flow.

3.5Validating Data After Schema Transfer

Validating data after schema transfer confirms that serialization, staging, and schema enforcement preserved meaning from source to sink. The Data Validation Tool (DVT) (opens in a new tab) is an open source Python CLI that compares source data to the BigQuery target using aggregations such as count, sum, and average on selected columns. Data flows from the source system and from the BigQuery target into DVT for comparison, and mismatches trigger iteration by expanding the transferred footprint or rerouting upstream feeds to BigQuery. Subsequent iterations repeat extraction, upload, and load until the structured sink fully replaces the legacy warehouse.

Key Facts

• Identity and Access Management (IAM) grants who can call an API, VPC Service Controls governs whether data can move across a service perimeter, and BigQuery row-level and column-level security governs which rows and columns a permitted query can see.
• Tink envelope encryption encrypts column data with a data encryption key (DEK) and wraps that DEK with a symmetric customer-managed encryption key in Cloud HSM, so BigQuery can decrypt with AEAD functions only for authorized readers.
• Cloud Storage is the high-throughput batch source and sink for bulk files, Pub/Sub is the low-latency streaming source for events and Change Data Capture changes, and BigQuery is the analytical sink for ELT and transfers.
• Apache Avro, Apache Parquet, and Apache ORC embed schema with the data, while CSV requires an explicit schema at load time or auto-detection.
• Facade views in BigQuery let upstream jobs write to tables while downstream jobs read from views, so each side of a schema change can evolve independently.