Planning Secure Data Pipelines on Google Cloud

Defining data sources and sinks is the work of choosing where pipeline data comes from, where results land, and how both ends stay secure, fast, and correctly typed. A source can be an internal bucket, a streaming topic, or an external on-premises database, while a sink can be a lake, warehouse, or messaging topic that feeds the next stage. The design must control identity, network path, encryption, throughput, and schema together.

1.1Securing Access, Network Paths, and Encryption for Ingestion and Sharing

Securing sources and sinks means controlling who can act, which network paths data can use, and how data stays protected at rest and in transit. Cloud Storage (opens in a new tab) enforces Identity and Access Management (IAM) policies on buckets and managed folders, while BigQuery enforces IAM roles on datasets plus row-level security and column-level security with policy tags and dynamic data masking. VPC Service Controls creates a service perimeter around projects, buckets, datasets, and machine learning resources, so data flows freely among members inside the perimeter but flows across the boundary only when an explicit ingress or egress rule permits it.

Access follows least privilege from no access upward. Each user, group, or service account receives only required read or write actions, often scoped by project, access pattern, or resource tags. IAM Conditions add attribute-based rules, principal access boundary policies cap maximum permissions, and Privileged Access Manager grants audited just-in-time elevation. Separation of duties places sensitive data in its own boundary, where an encrypted-data viewer sees only ciphertext and metadata while only a plaintext reader with key and column access can decrypt values.

Encryption and private connectivity complete the boundary. Data at rest is encrypted by default with Google-managed keys, with customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) when revocation, rotation, or residency control is required. Data in transit uses HTTPS to APIs, Transport Layer Security terminated at load balancers, and Pub/Sub default encryption at rest and in transit. A Virtual Private Cloud (VPC) network with private IP addresses removes internet exposure, Private Service Connect carries traffic from VPC resources to Google APIs over private addresses, and firewall rules deny all traffic by default and allow only required ports, protocols, and sources. For external ingestion, Dedicated Interconnect provides high-throughput private transport, the perimeter allowlists on-premises IP addresses, and Tink envelope encryption can encrypt sensitive columns locally before upload so plaintext never crosses the external network.

1.2Matching Batch and Streaming Entry and Exit Points to Throughput and Latency

Matching entry and exit points means aligning arrival pattern and downstream access with the operational profile of the storage or messaging service. The storage layer holds data as files or managed records, while the computation layer loads data from storage, processes it, and saves results back to a target that can be the same storage layer. To reduce latency, transfer cost, and I/O overhead, storage and compute should run on separate platforms in the same zone and region, for example by moving storage from Hadoop Distributed File System (HDFS) to Cloud Storage while running computation on Managed Service for Apache Spark.

Cloud Storage is the high-throughput batch source and sink for raw files and bulk processing. A pipeline lands raw data into a regional bucket near the processing cluster, then a Spark job reads through the Cloud Storage connector and writes curated results back to Cloud Storage or onward to a warehouse. Regional buckets replicate synchronously across zones in a region, while dual-region and multi-region buckets replicate asynchronously across regions for higher availability. Pub/Sub is the low-latency streaming source and sink for IoT events, logs, and Change Data Capture (CDC) changes, where data flows from publishers to a topic and then from the topic to subscribers such as Dataflow, which processes continuously and lands results to Cloud Storage bronze storage.

BigQuery is the analytical sink for batch and extract, load, and transform (ELT) pipelines that load first and transform later with SQL. Data flows from Cloud Storage through scheduled transfers, from software-as-a-service apps and sources such as Teradata or Amazon Redshift (opens in a new tab) through the BigQuery Data Transfer Service (opens in a new tab) on a scheduled managed basis, or directly from Hadoop and Spark through the BigQuery connector. Batch pipelines therefore enter through Cloud Storage files and exit to Cloud Storage or BigQuery, while real-time pipelines enter through Pub/Sub or Kafka topics and exit to streaming processors before landing.

1.3Carrying Schema with Data, Staging Extracts, and Evolving Without Breakage

Carrying schema with data is the practice of choosing file formats that preserve types, staging extracts in an intermediate sink, and enforcing schema at load. Apache Avro, Apache Parquet, and Apache ORC embed schema with the data so the sink can interpret fields without external definitions, while comma-separated values (CSV) carries no embedded schema and requires an explicit schema at load time or auto-detection. Parquet and ORC use columnar layout, which lowers scan cost and speeds analytical queries when Cloud Storage also serves as a data lake.

Staging decouples extraction from loading. First, schema and data are extracted to files in the existing environment; then, those files are uploaded to a Cloud Storage bucket; only after upload, data is loaded from the bucket into BigQuery. Data flows from another cloud through Storage Transfer Service, from on-premises with good bandwidth through the gcloud CLI with multi-threaded parallel uploads that resume after errors, and without sufficient bandwidth through Transfer Appliance for offline transfer. Locating the bucket close to the source reduces network latency, and co-locating the bucket with the BigQuery dataset simplifies the final load.

Loading and evolution protect producers and consumers during change. Supported load paths from Cloud Storage to BigQuery include BigQuery Data Transfer Service for automated recurring transfers, the Storage Write API for programmatic streaming and batch writes, and one-time batch loads. When transformation is needed before the sink, data flows from the staging bucket through Dataflow with Beam transforms, through Spark with Spark SQL and the Hadoop BigQuery Connector, or through Cloud Data Fusion with preconfigured connectors, or data flows into a BigQuery staging table and SQL writes transformed output to the final table. For evolution, downstream processes first point at facade views while upstream tables keep the source schema, then new tables are created on the evolved schema and view definitions are changed to query source tables, new tables, or a SQL UNION of both. Because views are read-only, upstream jobs write to tables while downstream jobs read from views, so each side evolves independently until source tables and unneeded views retire.

Defining data transformation and orchestration logic is the work of deciding where data is reshaped and when dependent tasks run. Transformation chooses between programmatic engines that reshape data before warehouse load and declarative SQL that reshapes data after load, while orchestration sequences those tasks across services. Together they control reliability, latency, cost, and replayability.

2.1Coordinating Multi-Step Pipelines with Composer, Workflows, and Connectors

Coordinating multi-step pipelines depends on workload shape, with managed Airflow for heavy data dependencies and serverless sequencing for lightweight service calls. Cloud Composer is the fully managed service for Apache Airflow that authors, schedules, and monitors long-running data pipelines as directed acyclic graphs (DAGs). Workflows (opens in a new tab) is the serverless orchestration service that runs loosely coupled HTTP-based services in a defined order with less code and explicit observable dependencies.

In Composer, an author defines a DAG, tasks call operators, operators invoke target systems, and downstream tasks run only after dependencies complete. The same extract, transform, load pattern that used a Hive operator can use the BigQuery operator to run transformations directly in BigQuery, while the Dataflow operator launches and manages a Dataflow job as one step. Composer also supports Oozie-to-Airflow conversion, Dataproc Workflow Templates for Spark-based extract, transform, load (ETL), and recurring Dataflow execution through Dataflow data pipelines, Cloud Scheduler, the Airflow Dataflow operator, or custom cron on Compute Engine.

In Workflows, a workflow is a series of steps in YAML or JSON where data flows from one step to the next using standard library functions. A representative event-driven flow runs in order: first, a file upload to a Cloud Storage bucket causes a Cloud Run function to trigger Workflows; then, the workflow sends text to the Cloud Natural Language API and images and video to Vision and Video Intelligence APIs; finally, the workflow writes resulting tags to Firestore. Connectors reduce code by formatting requests to Google Cloud APIs, such as translating files in batch and saving results to Cloud Storage or running BigQuery query jobs serially, and a connector to Application Integration lets a workflow trigger an integration, wait for heuristic checks or human approval, and then grant a refund or restore content based on the result.

2.2Unifying Batch and Streaming Transforms with Beam on Dataflow

Unifying batch and streaming transforms with Apache Beam lets one codebase handle bounded batch data and unbounded streaming data on Cloud Dataflow. A developer authors pipelines of PCollections and transforms, and Dataflow distributes stages in parallel across Compute Engine worker virtual machines in a specified region. Dataflow provides exactly-once processing by default, while streaming pipelines that tolerate duplicates can enable at-least-once mode to reduce cost and improve latency.

Scalability depends on code plus external factors. An external Kafka topic with too few partitions caps read throughput, formats that support parallel reads such as Avro are usually faster than CSV with embedded newlines or compressed files, and geographic proximity and networking between workers, sources, and sinks affect performance. Using managed sources and sinks such as Pub/Sub and BigQuery removes much capacity planning because those services scale natively.

Complex logic requires custom pipelines rather than templates or basic connectors. Built-in Dataflow templates are predefined Beam pipelines for code-free deployment, such as Kafka export, but a custom pipeline is needed for unsupported sources or sinks or for record normalization before delivery. For Kafka migration, Kafka Connect with MirrorMaker 2.0 fits transfer, backup, and disaster recovery with only per-message changes, while Dataflow with Beam fits high-volume migration needing cleaning, enrichment, aggregation, and joins. Windowing groups elements by time so related records from different unbounded streams can be aligned and joined, which Kafka Connect cannot do.

Hot-key skew and external-call overhead need explicit design. When many elements share one key, the worker for that key overloads while others idle, so rekey with a ParDo to spread load or add fanout so partial aggregation runs in parallel before the final combine. In Java use Combine.PerKey.withHotKeyFanout instead of Count.PerKey and Combine.Globally.withFanout instead of Combine.Globally, and in Python use with_hot_key_fanout on CombinePerKey. For enrichment, native I/O transforms such as BigQueryIO batch elements into fewer requests, but the pipeline must still limit batch size and concurrency because Dataflow autoscaling can add workers that further overload an already stressed external service or exhaust quotas.

2.3Placing Transform Compute with ETL, ELT, and Scheduling Choices

Placing transform compute is the choice between transforming before warehouse load and loading raw first and transforming inside the warehouse. In extract-transform-load (ETL), data flows from sources through Beam or Spark transforms to a destination sink before warehouse load. In extract-load-transform (ELT), raw data flows into BigQuery first and then BigQuery compute performs filtering, joining, aggregation, and modeling with SQL.

Cloud Dataflow is the serverless Beam runner for batch and streaming ETL with windowing, session analysis, and BigQuery connectors for low-latency ingestion of fluctuating volumes. Dataflow allocates worker virtual machines, autoscales them, rebalances work, corrects stragglers, and deletes workers when the job completes. Cloud Data Fusion is the fully managed visual integration service built on CDAP where a developer connects sources, transforms, aggregates, sinks, and error collectors into a DAG that the planner executes as a Spark job on Managed Service for Apache Spark. Data Fusion connects through prebuilt plugins and Java Database Connectivity (JDBC) drivers, supports batch and streaming, and typically stores processed output in Cloud Storage as Parquet or ORC with metadata in Dataproc Metastore. Declarative ELT in BigQuery uses GoogleSQL with data definition and manipulation statements, Dataform (opens in a new tab) for scheduled complex SQL transformations, and scheduled queries for simple time-driven chains.

Scheduling decides when work starts and in what order. Event-driven scheduling starts on a state change such as a file landing or Pub/Sub message arrival, while time-driven scheduling starts on a clock such as hourly or daily. A typical Spark batch orchestration runs in order:

1. Validate arrival: Managed Airflow validates that new files reached the Cloud Storage ingestion bucket.
2. Create compute: Managed Airflow starts an ephemeral Managed Service for Apache Spark cluster.
3. Run transformation: Managed Airflow starts the Spark job that performs ETL and produces business metrics.
4. Clean up: Managed Airflow deletes the cluster, so the cluster exists only while processing runs.

Choose Dataflow ETL for complex event-time logic, per-record enrichment, unified batch and streaming, or highly parallel fan-out. Choose Data Fusion and Spark for no-code integration, broad JDBC connectivity, or heavy batch reshaping into lake files. Choose BigQuery ELT for large set-based SQL modeling, raw replay, and analyst-owned pipelines with minimal cluster code.

Networking fundamentals for data pipelines cover how traffic is distributed to healthy backends, how isolated networks are built and linked to on-premises sources, and how movement and performance are secured and diagnosed. Traffic flows from clients to load balancer frontends and then from frontends to backends, while VPC networks, hybrid links, perimeters, and logs determine where pipeline data can flow and why it slows or costs more.

3.1Distributing and Securing Traffic with Load Balancing and Private Access

Distributing traffic with Cloud Load Balancing sends inbound requests from a frontend to healthy backends selected by routing rules and health checks. A global external Application Load Balancer can select across Google Kubernetes Engine (GKE) (opens in a new tab) clusters in distinct regions, while a Service Mesh-managed ingress gateway then handles cluster-specific routing and ingress security. GKE configures this through the Kubernetes Gateway API, where traffic flows from the internet client to a Gateway resource that defines listeners, then to HTTPRoute rules that route by path, header, or split, with Policy resources controlling underlying infrastructure.

External access to GKE follows a fixed order: first, the client sends a request to a Service of type LoadBalancer or Gateway and reaches the Google Cloud load balancer; then, the load balancer forwards to a healthy node based on health checks; finally, the node forwards to the destination Pod on the same or different node using kube-proxy or eBPF with Dataplane V2. Setting externalTrafficPolicy to Local avoids the extra hop by sending only to nodes with healthy Pods, at the cost of uneven distribution if Pods are unevenly spread.

The load balancer type depends on which layer must route and whether client identity matters. Application Load Balancer routes Layer 7 HTTP attributes and combines with API gateways, Cloud CDN, and Cloud Armor for content-based routing and global distribution. Proxy Network Load Balancer terminates TCP or TLS at a proxy for multi-region TCP distribution with TLS offload. Passthrough Network Load Balancer preserves client source IP without proxy overhead and supports UDP, ESP, and ICMP. Hybrid load balancing extends the frontend to on-premises or other-cloud endpoints through hybrid connectivity network endpoint groups, where traffic flows from the Google Cloud frontend through Interconnect or VPN to external endpoints for phased migration.

Private and outbound paths keep pipeline access off the internet. Private Service Connect enables private unidirectional access using internal IP addresses, where the producer side depends on an internal load balancer plus service attachment and the consumer side creates an endpoint in its own VPC that routes to that attachment. Cloud NAT is the managed regional gateway that translates source addresses so private Pods and VMs can initiate outbound connections for updates or APIs without receiving inbound internet connections, with Cloud Router providing control configuration. Cloud DNS resolves names to frontend addresses with geolocation policies that steer clients by location and split-horizon responses that keep private endpoints private internally.

3.2Designing VPC Networks, Firewall Policy, Routing, and Hybrid Links

Designing VPC networks in custom mode means creating subnets explicitly per region instead of automatically in every region. Landing zones separate hub, transit, and workload spoke VPC projects, each enabling the Compute Engine API before network creation. Each workload subnet is created where pipeline resources such as Dataflow workers and GKE nodes run, with Private Google Access enabled so VMs with only internal addresses can still reach Google APIs, and Private Service Connect configured so API traffic stays private. Centralized outbound access uses a Cloud NAT gateway in each region where egress is needed, scoped to specific subnets and logged at ERRORS_ONLY minimum or ALL for audit.

Firewall enforcement combines distributed stateful rules with hierarchical policies under least-privilege microsegmentation. Design blocks all traffic by default and allows only required protocols and ports, with firewall rules logging enabled for visibility. Priority controls order with lower numbers evaluated first, so specific rules must sit above general rules:

Hierarchical policies enforce organization-wide exceptions, such as permitting Identity-Aware Proxy TCP forwarding from 35.235.240.0/20 for ports 22 and 3389 and permitting load balancer health checks from well-known ranges for ports 80 and 443.

Routing and inspection decide how routes are learned and forced through checks. Cloud Router is a Border Gateway Protocol speaker in the control plane that creates routes for HA VPN and Interconnect but carries no data-plane traffic and never re-advertises learned prefixes. Regional dynamic routing advertises only same-region subnets, while global routing advertises all VPC subnets but penalizes non-local routes to preserve symmetry. Hub-and-spoke sharing uses VPC Network Peering where the hub exports custom routes and spokes import them, and inspection forces egress through appliances by deleting the 0.0.0.0/0 route to the default internet gateway and creating a new 0.0.0.0/0 route whose next hop is the appliance load balancer forwarding rule.

Hybrid connectivity choice depends on transport, performance, and encryption. Cloud VPN uses IPsec over the internet for cost-sensitive or non-mission-critical ingestion, Dedicated Interconnect uses direct physical connections for low-latency high-bandwidth private performance, Partner Interconnect uses partner networks without colocation, and Cross-Cloud Interconnect links VPC networks to supported clouds privately. Provisioning runs in order: first, provision Dedicated Interconnect ports; then, create two VLAN attachments per region with Cloud Routers and Border Gateway Protocol sessions; finally, configure peer routers and advertise hub and workload ranges back on-premises.

3.3Controlling Perimeters, Firewall Enforcement, and Flow Visibility

Controlling perimeters and firewall enforcement decides what can leave a data boundary and what can reach a virtual machine. VPC Service Controls groups projects and an on-premises network into a service perimeter around storage and analytics services, so traffic that tries to read or copy sensitive data from inside to outside through a managed service is denied even when the identity has broad inside access. Perimeters cannot contain projects from different organizations, so cross-organization pipeline access uses a perimeter bridge that explicitly allows projects and services in different perimeters to communicate.

External access is limited by giving pipeline resources only internal IP addresses unless internet access is explicitly required. Resources in a VPC communicate over internal addresses if firewall rules permit and can still reach Google APIs through Private Service Connect or Private Google Access. Organizational policies further restrict which resources may receive external IP addresses, and controlled egress uses Secure Web Proxy for HTTP(S) downloads with identity controls or Cloud NAT otherwise.

Cloud Next Generation Firewall enforces stateful decisions at each virtual machine rather than a single chokepoint, scaling horizontally with pipeline traffic. Traffic can still be steered to third-party appliances by routes, where same-priority routes share traffic with 5-tuple hash Equal-Cost Multipath and different priorities create redundancy. Designs should prefer network firewall policies with Tags over network tags for Compute Engine because of IAM control and scaling, but use CIDR-based rules for hierarchical policies or Network Connectivity Center (opens in a new tab) peered networks where Tags are unsupported.

Visibility ties security to performance and cost. VPC Flow Logs are enabled at the subnet level, collect from all VMs in that subnet, aggregate by connection at 5-second intervals, and export in real time through Cloud Logging. Longer aggregation to 15 minutes reduces records for long-lived connections, sampling preserves shape while reducing volume for cost analysis such as top talkers and cross-region estimates, and removing metadata reduces Logging consumption for forensics on who talked to whom. Network Intelligence Center combines Topology to trace paths, Connectivity Tests to verify expected packet behavior, Performance Dashboard to separate physical network limits from application delay, Firewall Insights to find shadowed or overly broad rules, Network Analyzer to detect misconfigurations, and Flow Analyzer to study traffic patterns.

Data encryption in Google Cloud protects pipeline data at rest, in transit, and in use through default protection plus customer-controlled keys and perimeters. At-rest design chooses who owns keys and where key material lives, while in-transit design chooses when default network encryption is enough and when Transport Layer Security or hybrid link encryption must be added. Together they create cryptographic boundaries the data owner controls from creation through destruction.

4.1Choosing Key Ownership with Default Encryption, CMEK, and CSEK

Choosing key ownership defines who generates the key, where the key lives, and who can revoke access. Default encryption (opens in a new tab) uses Google-owned and Google-managed keys at no extra cost with automatic rotation, covering stored customer data without customer action. Customer-managed encryption keys (CMEKs) move ownership to the customer in Cloud Key Management Service (Cloud KMS) while Google services still perform encryption, and customer-supplied encryption keys (CSEK) have the customer supply raw key material per request, historically for Compute Engine disks.

CMEK uses envelope encryption where the Cloud KMS key acts as a key-encryption key (KEK) that wraps per-chunk data-encryption keys (DEKs) that directly encrypt storage chunks with AES-256. When a service reads or writes CMEK-protected data, the service calls from its service account to Cloud KMS to unwrap the DEK and then uses the DEK on the chunk, with each use recorded in audit logs. Disabling, destroying, or revoking access to the CMEK makes protected data undecryptable even though encrypted bytes remain. A CMEK-compliant service differs by holding only short-lived ephemeral keys in memory that are flushed when processing ends, so outputs must be persisted in a CMEK-integrated service such as Cloud Storage or BigQuery to retain control.

CMEK integrates across pipeline assets including BigQuery datasets and tables, Cloud Storage buckets, Pub/Sub topics, Dataflow pipeline state, Dataproc clusters and Metastore, Dataform repositories, Cloud SQL, Spanner, Firestore, Bigtable, and Compute Engine disks, snapshots, and images. Granular control comes from granting the service identity permission to use a specific key version per dataset, bucket, topic, or cluster, supporting separation by environment or data domain. Central enforcement uses organization policy such as gcp.restrictNonCmekServices to block non-compliant creation, while Cloud External Key Manager is preferred over CSEK when external control is required because it supports more services with higher availability.

4.2Governing Key Rings, Protection Levels, Rotation, and Destruction

Governing keys in Cloud KMS covers where keys live, what hardware protects them, how often they rotate, and how destruction is guarded and monitored. A key ring is a container that groups keys in a specific project and location, and keys can only protect resources allowed by location and policy. The recommended pattern is one centralized key project per environment with a key ring in each location where buckets, datasets, or disks reside, avoiding creation of keys in the same project as the data they protect so key administrators and data administrators remain separate. Centralized governance assigns all lifecycle work to a security team for regulated workloads, while delegated governance lets a central team set guardrails and lets application owners manage keys to avoid bottlenecks.

Protection level determines where key material lives. Software protection stores keys in a Google software backend with FIPS 140-2 Level 1 validated operations. Multi-tenant Cloud Hardware Security Module (HSM) protection stores keys in shared Google HSM clusters, and single-tenant Cloud HSM stores keys in dedicated HSM partitions with FIPS 140-2 Level 3 certified operations. Cloud External Key Manager with EXTERNAL level keeps material in a third-party system outside Google infrastructure for sovereignty. At creation, material is either generated inside Cloud KMS, which keeps raw material hidden and enables automatic rotation, or imported as bring-your-own-key, which requires automation, escrow, leakage controls, and planning for re-import risk.

Rotation creates a new primary version on a schedule, ideally at least yearly and more often for sensitive pipelines, while older versions remain to decrypt existing DEKs. Rotation does not re-encrypt all old data uniformly: in Spanner the new version re-encrypts the DEK, in Cloud Storage only newly written objects use the new version unless old objects are rewritten, and in BigQuery an existing table keeps the version assigned at creation until the table is recreated or reconfigured. Cloud KMS Autokey automates provisioning by delegating creation to the service agent, generating HSM-backed AES-256 GCM keys with yearly rotation, co-located with the protected resource and scoped per resource type while preserving separation of duties.

The key-version lifecycle moves through enabled, disabled, scheduled for destruction, and destroyed, with restore possible only during the soft-delete window. The default scheduled-for-destruction period is 30 days and can be set as short as 24 hours at creation, but the duration cannot be changed later and the key cannot perform operations while scheduled for destruction. Preventive guardrails include constraints/cloudkms.minimumDestroyScheduledDuration for a minimum window, constraints/cloudkms.allowedProtectionLevels for approved backends, project liens against deletion of key projects, and least privilege separating key managers from key users. Detective controls include centralized Admin Activity logs, optional Data Access logs for encrypt and decrypt volume, inventory and usage dashboards that flag data made inaccessible by disabled keys, Monitoring alerts on destruction scheduling, and Security Command Center findings on overly permissive roles.

4.3Encrypting Traffic Inside Google Networks, to Apps, and Across Hybrid Links

Encrypting traffic in transit protects data while it moves between users, services, and environments, with default protection inside Google networks and added controls at the edge and across hybrid links. Traffic between virtual machines within a VPC and between peered VPCs inside Google production networks is encrypted, integrity-protected, and authenticated without customer configuration. Traffic from the internet or a VPC to Google services terminates at the Google Front End (GFE), which provides distributed denial-of-service countermeasures and routing, and from the GFE onward Google infrastructure (opens in a new tab) uses Application Layer Transport Security (ALTS) for authentication, integrity, and encryption from the GFE to a service and from one service to another, such as a worker calling BigQuery, Cloud Storage, or Pub/Sub.

Application-layer encryption adds Transport Layer Security on top of default protection. Data flows from the end user to the load balancer to the backend virtual machine or container, with each hop needing its own decision. An external Application Load Balancer or external proxy Network Load Balancer encrypts the user-to-load-balancer hop with TLS when the client supports it, terminating at the GFE with Google-managed or self-managed certificates and BoringSSL negotiation. Cloud Load Balancing with HTTPS load balancers enforces HTTPS for apps and APIs, mutual TLS through Cloud Service Mesh authenticates GKE and Compute Engine service-to-service calls, and Apigee configures TLS and mutual TLS for API traffic. Encryption from the load balancer to backends must be enabled separately.

Hybrid links leave Google networks and need explicit encryption. Cloud Interconnect provides high-throughput direct physical connections for large ingestion volumes but is not encrypted by default, and traffic to VMs over external IP addresses also leaves Google networks. Use MACsec for Cloud Interconnect to encrypt interconnect traffic and Cloud VPN with IPsec to encrypt virtual private network traffic, after which internal VM-to-VM and GFE-to-service protections apply inside the VPC. VPC Service Controls adds a service perimeter around projects and services such as Cloud SQL, Spanner, BigQuery, and Cloud Storage, where requests pass identity authentication, IAM checks, and perimeter checks before backend invocation and movement across the boundary is denied unless an ingress or egress rule allows it.

Key Facts

• Cloud Storage is the high-throughput batch source and sink, Pub/Sub is the low-latency streaming source for events and Change Data Capture changes, and BigQuery is the analytical sink for extract, load, and transform pipelines.
• Apache Avro, Apache Parquet, and Apache ORC embed schema with the data, while comma-separated values requires an explicit schema at load time or auto-detection.
• Cloud Composer orchestrates scheduled multi-step pipelines as Airflow DAGs with operators, while Workflows orchestrates lightweight event-driven sequences of HTTP-based services.
• A Virtual Private Cloud Service Controls perimeter cannot contain projects from different organizations, so cross-perimeter pipeline access requires a perimeter bridge.
• Customer-managed encryption keys act as key-encryption keys that wrap data-encryption keys, and Cloud Storage applies rotation only to newly written objects while an existing BigQuery table keeps its original key version.