Multi-Cloud Portability and Data Residency

Leveraging open-source standards for data pipeline portability means writing transformation logic once against an open API and running it on different engines and clouds without refactoring. In Google Cloud, Apache Beam (opens in a new tab) on Cloud Dataflow and Managed Service for Apache Spark (opens in a new tab) (Dataproc) separate pipeline code from execution. Portable file formats, open connectors, and consistent handover points let the same pipeline move data from on-premises or AWS and Azure into Cloud Storage and BigQuery while respecting regional controls.

1.1Apache Beam Pipelines on Cloud Dataflow

Apache Beam pipelines on Cloud Dataflow separate pipeline definition from pipeline execution for both batch and streaming. A developer writes the pipeline once with the Apache Beam software development kit (SDK), which defines sources, transforms, and sinks, and then runs that program on the Cloud Dataflow managed service for serverless execution. Data flows from source systems through Beam transforms to the destination sink, with temporary files and pipeline state artifacts created along the way. If sources, sinks, or temporary files sit outside the job's region, data can move across regions during execution.

Portability comes from the Beam runner model, where the same Beam program can execute on different runners. A primary pipeline can run on Cloud Dataflow while a backup pipeline runs on a local on-premises Spark cluster by using the Apache Spark Runner, which supports disaster recovery and hybrid operation without rewriting transforms. Runner support varies by operation, so compatibility must be checked in the Beam Capability Matrix before relying on a second runner. For migration, the Google-provided Java Database Connectivity (JDBC) to BigQuery template can be cloned and extended with additional Beam transforms to add transformation before loading into BigQuery.

Locality is controlled by three Dataflow parameters that separate control-plane metadata from data-plane compute. The regional endpoint controls where job metadata is stored, while the worker region and worker zone control where workers are deployed and data is processed. A job can therefore be restricted to a specific region or zone when processing must stay in one location. When encryption mitigates locality rules, customer-managed encryption keys (CMEK) in Dataflow encrypt pipeline state and authorize access to sources and sinks protected with Cloud Key Management Service (Cloud KMS) keys.

1.2Apache Spark Workloads on Managed Service for Apache Spark

Managed Service for Apache Spark is the fully managed service for running open-source Apache Hadoop, Apache Spark, and Apache Flink jobs without managing a permanent cluster. Legacy Spark, Hadoop, or Flink code migrates directly because the service provides managed clusters with native Spark and Hadoop APIs. Instead of operating one monolithic cluster, teams create many short-lived ephemeral clusters, run a job, and delete the cluster when processing completes. This model fits periodic extract, transform, and load (ETL) work, such as a retail batch job that reads point-of-sale files from Cloud Storage, runs a Spark ETL job orchestrated by Managed Service for Apache Airflow, and writes business metrics back to the data lake.

Integration with Google Cloud storage depends on two open Hadoop connectors. The Hadoop Cloud Storage Connector lets Spark read input files from and write output files to Cloud Storage, while the Hadoop BigQuery Connector is a Java library that lets Hadoop and Spark write directly to BigQuery through abstracted InputFormat and OutputFormat classes. Transformation logic stays in portable Spark code using the following options:

• Spark SQL for declarative queries over DataFrames and tables
• Scala, Java, Python, or R for custom transformation functions and libraries
• Cloud Data Fusion plugins from the Cloud Data Access and Integration (CDAP) library when using a visual pipeline instead of Spark code
• BigQuery SQL after loading when using an extract, load, and transform (ELT) pattern with a staging table

1.3Portable File Formats, Connectors, and Metadata

Portable file formats preserve schema and performance when data moves between engines and clouds. Apache Avro, Apache Parquet, and Apache Optimized Row Columnar (ORC) embed schema information with the data, which reduces transfer complexity compared with comma-separated values (CSV) where schema must be specified separately at load time. Parquet and ORC are columnar formats commonly used in Cloud Storage data lakes because they provide efficient storage and faster access for analytical workloads. Exporting existing warehouse tables to Avro, Parquet, or ORC in staging storage therefore creates a vendor-neutral intermediate that Spark, Beam, and BigQuery can all read.

Metadata portability depends on Dataproc Metastore, the managed Hive metastore service that stores databases, tables, columns, and partitions for data stored in Cloud Storage. The processing pipeline depends on both sides of this split: Cloud Storage holds the columnar data files while Dataproc Metastore holds the table definitions that make those files queryable. Because both layers use open Hive and Hadoop interfaces, Spark jobs running on-premises, on another cloud, or on Managed Service for Apache Spark can share the same table view without proprietary catalog code.

1.4Building Hybrid and Multi-Cloud Pipelines Without Refactoring

Hybrid and multi-cloud pipelines keep the same Beam or Spark logic while changing only the source, sink, and network path. A typical Amazon Web Services (AWS) to Google Cloud migration moves data from Amazon Simple Storage Service (Amazon S3) to Cloud Storage and then transforms and loads it into BigQuery for analysis. BigQuery Omni supports the opposite pattern by running BigQuery compute locally to storage on AWS or Azure to query data in Amazon S3 or Azure Blob Storage, which avoids moving data when breaking down silos. For durable handover between environments, Pub/Sub queues or Cloud Storage buckets receive data from transactional systems in the private environment and then serve as sources for Dataflow or Spark pipelines.

A portable migration follows an ordered workflow that preserves the open-source transformation step:

1. Extract to files: Export schema and data from the existing platform to Avro, Parquet, ORC, or CSV in on-premises staging storage.
2. Copy to Cloud Storage: Transfer files to a Cloud Storage bucket using Storage Transfer Service, the gcloud command-line interface (CLI), or Cloud Storage client libraries, choosing the bucket location closest to the source to reduce latency.
3. Transform and load to BigQuery: Run the existing Beam or Spark logic against Cloud Storage input and BigQuery output, or load to a staging table and transform with BigQuery SQL in an ELT pattern.

Long-term high-volume exchange between clouds depends on dedicated connectivity rather than the public internet. Cross-Cloud Interconnect (opens in a new tab) provides high-bandwidth dedicated connectivity between Google Cloud and other cloud providers in supported locations, with encryption options including virtual private network (VPN) tunnels, high-availability (HA) VPN over Cloud Interconnect, and media access control security (MACsec) for Cross-Cloud Interconnect. Using consistent tooling and processes across environments, plus Sensitive Data Protection for masking and de-identification of personally identifiable information (PII), keeps the same pipeline compliant on each cloud.

The following comparison shows why refactoring is avoided: the transformation stays in Beam, Spark, CDAP, or SQL while only the managed execution service changes.

Choose ETL in Beam or Spark when trimming or enriching before loading, and choose ELT in BigQuery when the warehouse itself can apply SQL transforms after loading.

Architecting multi-cloud data processing and federated analytics means running analytics and pipelines across Google Cloud, AWS, Azure, and private environments without centralizing all raw data first. BigQuery Omni (opens in a new tab) provides the federated query layer, handover networking patterns provide the secure movement boundary, and portable storage and pipeline services provide the processing layer. Together they let teams query where data must stay while still moving only the results or curated datasets that create value.

2.1Federated Analytics with BigQuery Omni

BigQuery Omni is a multicloud analytics capability that runs BigQuery compute locally to data stored outside Google Cloud. Query traffic flows from BigQuery Omni to Amazon Simple Storage Service (Amazon S3) or Azure Blob Storage, and results return without requiring a full dataset copy into Google Cloud. This breaks down data silos because data teams can join and analyze datasets that remain resident in AWS or Azure using familiar BigQuery SQL and external data source definitions. Choose BigQuery Omni when residency, egress cost, or data volume makes moving raw data impractical and in-place federated querying meets performance needs.

2.2Secure Connectivity for Hybrid and Multi-Cloud Analytics

Secure connectivity for hybrid and multi-cloud analytics controls where ingestion enters Google Cloud and where results are allowed to leave. Data flows from transactional systems in a private computing environment or another cloud into Google Cloud through the handover networking pattern, which uses intermediate queues or buckets as the trust boundary. When analytical results must flow back from Google Cloud to transactional systems, the architecture combines the handover pattern for ingestion with the gated egress pattern for controlled return traffic. For sustained, high-volume exchange between clouds, the service depends on Cross-Cloud Interconnect (opens in a new tab), which provides high-bandwidth dedicated connectivity between Google Cloud and other providers in supported locations.

Encryption at the connectivity layer depends on the selected hybrid option:

• VPN tunnels provide encrypted connectivity for lower-volume or internet-based hybrid links.
• High-availability (HA) VPN over Cloud Interconnect provides encrypted connectivity layered over dedicated interconnect.
• Media Access Control security (MACsec) for Cross-Cloud Interconnect provides encryption for dedicated multi-cloud interconnect links.

Initial one-time transfer from a private environment is a separate decision from long-term interconnect. Choose the initial transfer approach based on dataset size and available bandwidth, and reserve Cross-Cloud Interconnect for long-term, high-traffic exchange where dedicated bandwidth lowers unit cost and stabilizes throughput.

2.3Portable Ingestion and Processing Pipelines

Portable ingestion and processing pipelines use managed handover points and cloud-first pipeline services so the same pattern runs across environments. Data flows from on-premises or multi-cloud sources into Pub/Sub queues or Cloud Storage buckets, and those queues and buckets then serve as sources for downstream data-processing pipelines and workloads. Dataflow and Cloud Data Fusion are both fully managed services for building ETL and ELT pipelines, so selection depends on use-case requirements rather than server management. Because the pipeline reads from standard queues, buckets, and open formats such as Parquet and ORC, processing can be recreated in another region or cloud without redesigning ingestion.

A locality-aware batch pipeline moves through an ordered sequence:

1. Ingestion: Stores send point-of-sale files to a Cloud Storage bucket, where ingested data awaits processing.
2. Orchestration: Managed Service for Apache Airflow orchestrates the end-to-end batch pipeline and restarts jobs after disruption.
3. Processing: An Apache Spark job on Managed Service for Apache Spark reads from the ingestion bucket, transforms events, and produces business metrics.
4. Data lake: Processed data flows to Cloud Storage in columnar formats, while table, column, and partition metadata flows to Dataproc Metastore as the Hive metastore.

This order matters for recovery because processing and orchestration capacity can be recreated, while the ingested objects, processed lake files, and metastore metadata are the state that must be preserved.

2.4Data Residency, Sovereignty, and Protection Controls

Data residency, sovereignty, and protection controls determine where data at rest may exist, who may access it, and how policy is enforced. BigQuery enforces residency through dataset locations that can be regional or multi-regional, and Cloud Storage enforces residency through bucket locations that can be single-region, dual-region, or multi-region. Organization Policy Service applies location-restricted resources constraints across the resource hierarchy to allow or deny multi-regions such as Asia and Europe, regions such as us-east1 or europe-west1, and zones such as europe-west1-b. Access locality is further constrained by Google Cloud Armor for IP and geo-based rules and by VPC Service Controls for context-based perimeter security around data services.

The storage locality options trade control for resilience:

When single-region containment is mandatory, adopt a multi-cloud or hybrid copy strategy using Cloud Storage interoperability for Amazon S3, open object stores such as minIO and Ceph on-premises, Managed Service for Apache Airflow with gcloud storage commands, or Transfer service for on-premises data. When the requirement is about who can read data rather than where bits sit, use Cloud External Key Manager (Cloud EKM) for BigQuery and Compute Engine or customer-supplied encryption keys (CSEK) for Cloud Storage and Compute Engine, where keys remain outside Google infrastructure and data cannot be decrypted without them. Sensitive Data Protection complements these controls by discovering, classifying, and de-identifying assets through masking, encryption, and replacement of PII with randomly generated or predetermined keys.

2.5Optimizing Transfer Cost and Operational Consistency

Optimizing transfer cost and operational consistency means moving less data, paying less per byte moved, and running the same operations everywhere. BigQuery Omni reduces egress and duplication cost because only queries and results cross clouds instead of full table copies. Cross-Cloud Interconnect reduces per-byte and performance risk only after traffic becomes long-term and high-volume; for one-time migration, dataset size, bandwidth, and time constraints determine the cheaper transfer method. Storage synchronization between buckets is asynchronous, so when recovery point and recovery time objectives require proof of a second copy, dual-write to two single-region buckets or write to one bucket and copy to the second with a Cloud Storage rewrite.

Consistent tooling and processes across environments increase operational efficiency even though consistency alone does not satisfy residency. Reusing the same ingestion contracts, pipeline frameworks, encryption practices, and lifecycle policies lowers management overhead when workloads span Google Cloud, other clouds, and private infrastructure.

Enforcing data residency, sovereignty, and compliance mandates means keeping storage, compute, and backups inside approved geographic and legal boundaries. Data residency is the physical location of data plus the local regulations that govern how that data is stored, encrypted, and accessed. Data sovereignty extends that idea to legal control, requiring that PII and government data remain in a specific country and are processed under that country's laws. In Google Cloud, residency is enforced by combining regionalized resources, location constraints, access perimeters, and customer-controlled encryption.

3.1Defining Regional Granularity for Applications and Data

Defining regional granularity means deciding what level of location control an application and its data require before choosing services. This decision starts with four related questions that separate availability, access, storage, and product support. Answering them determines whether a multi-region design is allowed or a strict single-region design is required.

• Multi-regional use: Multi-regional services create highly available and resilient architectures, so the design must first confirm whether multi-regional storage and processing are permitted.
• Access location: Access restrictions control where an application can be reached from, enforced with Google Cloud Armor for IP and geo-based constraints and VPC Service Controls for context-based perimeter security.
• Data at rest location: Data-at-rest restrictions require managed services that can pin storage to a specific region, such as BigQuery locality controls that dictate where datasets are stored and backed up.
• Product regional support: Product availability varies by region, so the Cloud locations page and product-specific pages must be checked to validate usable regions and any mitigating features for products without regional restrictions.

3.2Enforcing Location With Organization Policy Service

Organization Policy Service provides centralized control over the Google Cloud resource hierarchy of organization, folders, and projects. The location-restricted resources constraint defines where location-based resources can be created by allowing or denying multi-regions such as Asia and Europe, regions such as us-east1 or europe-west1, or individual zones such as europe-west1-b. Only services on the resource locations supported services list honor this constraint, so unsupported services need separate validation. A complementary domain-restricted sharing constraint limits which identities can be added to IAM policies to an allowed_values list of Cloud Identity or Google Workspace customer IDs, which prevents external identities from gaining access to sovereignty-sensitive projects.

3.3Controlling Access Boundaries by Network and Identity

Access-boundary controls enforce sovereignty by limiting the network paths and service perimeters from which data can be reached. Google Cloud Armor sits in front of applications and filters incoming traffic flows from clients to backends based on IP address and geography. VPC Service Controls creates a context-based security perimeter around Google Cloud APIs and projects, so data flows from protected services to outside identities are denied even when credentials are valid. For Cloud SQL (opens in a new tab), VPC Service Controls can restrict use of the Cloud SQL Admin API and Cloud Storage API for import and export operations, which keeps backup and transfer flows inside the approved perimeter.

3.4Regionalizing Object and File Storage for Data at Rest

Regionalized object and file storage ensures that data at rest remains only in the required region, including object and file copies. Cloud Storage buckets can be created as single-region, dual-region, or multi-region locations to meet locality restrictions. Dual-region buckets store objects redundantly in a pair of regions chosen by the customer, while multi-region buckets store objects in at least two geographically separate areas using any combination of data centers in that multi-region, which can include data centers not explicitly listed as available regions. Data synchronization between those areas occurs asynchronously regardless of storage class, which maximizes availability during large-scale disruptions but leaves a small replication lag.

When strong proof of cross-region durability is needed for recovery time objective (RTO) and recovery point objective (RPO), two single-region buckets provide tighter control than one multi-region bucket. Data flows from the writer to the primary bucket and then to the secondary bucket either by dual-write from the application to both buckets or by writing once and using the Cloud Storage rewrite operation to copy the object to the second bucket. Filestore follows the same residency model by deploying managed file storage in specific regions and zones that match the locality requirement.

3.5Configuring Managed Databases and Analytics Locations

Configuring managed database and analytics locations requires matching each service's location model to the mandated region and keeping compute colocated with the data. The following patterns cover the core data services:

• BigQuery: BigQuery (opens in a new tab) dataset locations can be regional or multi-regional and dictate where datasets are stored. For resilience, backups must go to another geographic location outside the multi-region scope, and for the EU multi-region that means excluding Zurich and London from backup targets.
• Bigtable: Bigtable production instances use zonal locations within one region and require a minimum of two clusters in unique zones. Google automatically manages replication between the clusters, synchronizing data so each zone holds an independent copy and incoming traffic can fail over to another cluster in the same instance.
• Firestore: Firestore data can be stored in a regional location or a multi-region location that operates in a multi-zone and multi-region replicated configuration. A multi-region location withstands loss of entire regions without data loss, so it should be selected only when locality rules permit it.
• Cloud SQL: Cloud SQL instances store data at rest only in the selected region under the Service Specific Terms, chosen at creation or by editing the instance, including backup placement through custom backup locations. A cross-region read replica places a readable copy in a designated region to sustain availability during a regional outage while keeping each copy pinned to a known region.

Federated analytics add a colocation dependency between BigQuery and its sources. A BigQuery federated query depends on the source instance being in the same location as the BigQuery dataset, so traffic flows from BigQuery to a colocated source and back. If the BigQuery dataset is in the Belgium europe-west1 regional location, the corresponding Cloud SQL or Spanner instance must also be in Belgium. If the dataset is in the US multi-region, the Cloud SQL or Spanner instance must be in a single region in the US geographic area, and a Bigtable app profile must route data flows from BigQuery to a Bigtable cluster in that same area, such as us-west1 when the dataset is in US. If the dataset is in the single region asia-northeast1 in Tokyo, the Bigtable cluster must also be in asia-northeast1. Analysis tools follow the same rule, so a Managed Service for Apache Spark cluster using the BigQuery connector and a Vertex AI Workbench instance using Jupyter notebooks should each be colocated with the BigQuery dataset they query.

3.6Enforcing Sovereignty With Customer-Controlled Encryption

Customer-controlled encryption enforces sovereignty by separating where encrypted data lives from who can decrypt it. By managing keys in an external system outside Google's infrastructure, an organization can use a broader geographic architecture because data flows from storage to compute remain unreadable without the keys. Without the keys available, the data cannot be decrypted, which satisfies requirements focused on restricting access rather than strictly pinning every byte.

Operating external keys creates a direct dependency between the key manager and all reads and writes. Key infrastructure must account for latency and reliability with high-availability and recovery processes, because the service depends on the external manager for every decrypt and safe write. The keys are integral to writing data, so no data can be safely written without them, which makes RPO less critical and makes RTO the real concern because operations cannot resume until keys are reachable again.

3.7Sustaining Resilience Under Single-Region Restriction

Sustaining resilience under single-region restriction means achieving durability without letting Google Cloud replicate data outside the approved area. Single-region restriction means geographic redundancy cannot be achieved with Google Cloud alone, since all copies would otherwise leave the approved area. Resilience must then come from a multi-cloud or hybrid design that places a second copy with another provider or on-premises system in the same geographic area while meeting distance requirements. This approach keeps both data flows inside the mandated locality while still protecting against loss of one platform.

Adopting the multi-cloud or hybrid approach involves several practical mechanisms for moving and storing the second copy:

• Ensure that distance requirements between the Google Cloud region and the alternate location are met.
• Use Cloud Storage interoperability to configure access to Amazon S3 with Google Cloud tools when AWS is the second cloud.
• Use open-source object stores such as minIO and Ceph to provide an on-premises object store for other clouds and on-premises solutions.
• Use Managed Service for Apache Airflow with the gcloud storage command-line utility to transfer data from the on-premises object store to Cloud Storage.
• Use the Transfer service for on-premises data to copy data stored on-premises to Cloud Storage.

If locality rules permit encryption as a mitigating control, multi-region or dual-region buckets can be used with customer-controlled keys instead of building a second platform. In that model, encrypted objects flow from the writer to geographically separate Google data centers, but control flows from the customer-owned key manager, so sovereignty is preserved through key custody rather than strict physical pinning.

Key Facts

• Apache Beam pipelines on Cloud Dataflow separate pipeline definition with the Beam SDK from execution on runners, and the Dataflow regional endpoint controls job metadata location while worker region and worker zone control where workers run.
• Managed Service for Apache Spark runs portable Apache Spark, Hadoop, and Flink jobs on ephemeral clusters, reading Cloud Storage via the Hadoop Cloud Storage Connector and writing BigQuery via the Hadoop BigQuery Connector.
• BigQuery Omni runs BigQuery compute locally to Amazon S3 or Azure Blob Storage so federated queries return results without copying full datasets into Google Cloud.
• Cloud Storage single-region, dual-region, and multi-region buckets trade geographic containment for availability, with asynchronous replication and dual-write or rewrite required to prove a second copy.
• Cloud External Key Manager keeps keys outside Google infrastructure for BigQuery and Compute Engine while customer-supplied encryption keys are purged after each Cloud Storage operation, making recovery time objective the critical risk when keys are unreachable.