Resilient Multi-Region Data Processing Jobs

Architecting resilient workflows across regions means placing compute and data so processing survives a zonal outage and can recover from a regional outage. A resilient design uses regional resources (opens in a new tab) that spread work across zones, multi-regional resources that survive loss of one region, and a planned replica in a secondary region with data replication and traffic rerouting. The design must balance availability against cost, latency, consistency, and data sovereignty.

1.1Zonal, Regional, and Multi-Regional Resource Types

Resource scope in Google Cloud determines which failure a workflow can survive. Compute Engine (opens in a new tab) virtual machines and zonal Persistent Disk operate in a single zone, regional managed instance groups and App Engine applications spread redundantly across multiple zones in one region, and Google-managed multi-regional services distribute redundantly within and across regions.

Cloud Storage (opens in a new tab), BigQuery, Spanner, Firestore, Backup and DR, Cloud KMS, and Storage Transfer Service are examples of multi-regional resources. If a single region fails, only customers solely in that region are impacted, while customers using multi-regional products are not impacted. Google Cloud designs against correlated failures across regions so one regional failure does not cascade to another.

1.2Regional Topology That Survives Zonal Outages

The regional deployment archetype runs application instances in two or more zones in a single region with data replicated synchronously across all zones. Traffic flows from users to a regional external load balancer, then from the load balancer to frontend instances, then from the frontends through a regional internal load balancer to backend instances, and finally to a database that is replicated across zones. If a zone fails, the database fails over to a replica in another zone.

Compute for this pattern depends on regional managed instance groups (MIGs) (opens in a new tab). A regional MIG provisions Compute Engine virtual machines automatically across multiple zones according to demand and regional availability, and the MIG recreates failed virtual machines automatically to maintain minimum capacity. Because load balancers are regional resources, the load balancer itself is not affected by a single zone outage. The topology stays available as long as at least one functioning component with adequate capacity exists in every tier, but it is robust against zone outages and not against region outages.

1.3Failover Pattern That Survives Regional Outages

The failover pattern for regional outages maintains a passive replica of the infrastructure stack in a second region. The primary region handles active processing while the failover region holds copies of data and a lower-capacity replica of compute to optimize cost. When the primary region fails, the workflow shifts to the failover region in an ordered recovery:

1. Activate the failover stack: Bring up the application in the failover region, scaling from lower-cost standby capacity to production capacity.
2. Promote the data copy: Promote the database replica in the failover region to primary, using snapshots to multi-regional storage or managed multi-regional storage when data was not already multi-regional.
3. Reroute traffic: Use Cloud DNS routing policies to route traffic from the failed primary load balancer to the load balancer in the failover region.

For data, the supported strategies are using managed multi-regional storage, snapshotting zonal or regional data to multi-regional storage, or managing self-replicated copies to one or more other regions. For compute, the strategy is to manually or automatically bring up the application in another region referring to copies of primary data. Explicitly configured backups and replication to other regions depend on cross-region networking, as described in disaster recovery for locality-restricted workloads (opens in a new tab).

1.4Global Platform Dependencies and Isolation Boundaries

Global platform services provide the backbone that lets multi-region workflows operate and fail over. The global network backbone, points of presence (POPs), globally distributed load balancers, and DNS carry customer traffic within the Google network until close to its destination. Configuration of global resources, including Identity and Access Management (IAM) policies, global firewall rules, global load balancer configurations, and Pub/Sub topics, is stored in replicated databases.

All Google Cloud services depend on resilient core tools for networking in and out of data centers, identity authorization, logging, metadata storage, and workflow management. Lower-level dependencies include Borg for cluster scheduling, Colossus for cluster storage, and encryption and key management infrastructure for Cloud KMS and Cloud External Key Manager (EKM). Compute Engine virtual machine and Persistent Disk data planes depend on Borg and Colossus, while Spanner, Bigtable, and Cloud Storage depend on key management, logging and auditing, and internal replication services. Requests to customer-controlled endpoints, such as Pub/Sub message delivery or Cloud EKM key fetches, depend on the global network to reach those endpoints.

1.5Tradeoffs for Region Selection and Resilient Design

Tradeoffs for resilient design center on cost, performance, and locality constraints. A multi-zone architecture costs more than a single-zone deployment because it runs redundant resources, and a failover region adds further cost even when run at reduced capacity. A single-region architecture keeps inter-resource traffic within the region for low latency and avoids cross-region transfer costs, while cross-region replication adds networking cost and multi-regional services trade latency or consistency for availability.

Region selection must weigh functional and non-functional criteria:

• Availability of required Google Cloud services and Compute Engine machine types in each region
• End-user latency and geographic proximity to users and on-premises environments
• Resource cost, cross-regional transfer cost, and per-region quota limits
• Regulatory requirements for data residency and operational sovereignty
• Sustainability requirements, since regions have different carbon footprints

To meet data residency, a workflow can deploy to a Google Cloud region in the required jurisdiction and enforce placement with the Resource Location organization policy constraint, product-specific location controls, and Virtual Private Cloud (VPC) Service Controls perimeters. The recommended build order is first build the foundation, then provision computing resources, then provision data storage resources, then provision data analytics resources such as BigQuery, Pub/Sub, Managed Service for Apache Spark, and Dataflow. Before promoting to production, verify the failure assumption by simulating a zonal outage for a regional MIG.

Governance and compliance controls for multi-region workloads bind job execution to approved locations, protect data with region-scoped keys, and block and audit movement across boundaries. On Google Cloud, Assured Workloads (opens in a new tab) and organization policy enforce where resources can be created, Cloud Key Management Service (Cloud KMS) enforces who can decrypt data, and VPC Service Controls with Access Context Manager enforces which principals can reach data inside a perimeter. Together these controls let resilient architectures run jobs in multiple regions or zones without violating data residency and sovereignty requirements.

2.1Enforcing Data Residency for Regional Job Execution

Data residency describes where Customer Data is stored at rest, and Assured Workloads implements residency by combining a dedicated folder with a Resource Locations organization policy constraint. An organization sets constraints/gcp.resourceLocations at the organization, project, or folder level to restrict the locations of new in-scope resources, and during Assured Workloads setup the selected control package further restricts which regions can be chosen for resources created in that folder. This means a data processing job can only select execution zones and storage locations that both policies allow, which prevents accidental creation of datasets, models, or buckets in unauthorized countries while still permitting multi-region designs that use only approved regions.

The residency model distinguishes what Google commits to protect by location. Customer Data (opens in a new tab) at rest for services that can be configured for data location is covered by residency commitments, and Customer Data in use by the configured service is processed only within the country of the selected region or within the country or countries of the selected multi-region. If the selected region or multi-region is in the European Union, processing stays within the European Union but not necessarily in the same country.

Services built for residency make the region a parent boundary for all child work. In Anti Money Laundering AI (AML AI), for example, each Instance resource is created with a specified region, all child artifacts including datasets, engine configs, models, generated features, hyperparameters, weights, and prediction results are constrained to that region, and the instance requires all input and output data to be in the same region and project. This parent Location resource pattern ensures that storage at rest and machine learning processing stay in one region and that cross-region movement cannot happen by misconfiguration.

2.2Managing Regional Encryption with Cloud KMS

Regional encryption management centers on Cloud Key Management Service (Cloud KMS), the centralized service for creating, importing, managing, and using cryptographic keys for envelope encryption. With customer-managed encryption keys (CMEK) and Cloud KMS Autokey, services such as BigQuery and Cloud Storage fetch a customer-managed key encryption key (KEK) from Cloud KMS to wrap the data encryption key (DEK) that actually encrypts the data, so control of the KEK determines ability to decrypt. For sovereignty beyond Google-managed keys, Cloud External Key Manager (Cloud EKM) lets teams use externally managed keys with supported CMEK integration services or the Cloud KMS API directly, while Cloud Hardware Security Module (Cloud HSM) hosts keys in FIPS 140 validated hardware clusters fronted by Cloud KMS.

Google provides the following assurances for key material in Cloud KMS (opens in a new tab):

• Decrypted key material cannot be exported or viewed through the API or another user interface.
• Google personnel cannot access unencrypted customer key material, and key material is encrypted with a KMS master key in Keystore that is not accessible to Google personnel.
• On a Hardware Security Module (HSM), key material is never accessed in decrypted state by Cloud KMS API jobs, and HSMs are FIPS 140 validated.
• Google system operators are prevented by standard operating procedures from accessing, using, or extracting customer key material while performing duties.

These key controls build on default protection for data at rest and in transit. Google Cloud encrypts data at rest by default with FIPS 140 validated AES-256, supports CMEK for attached disks and client-side encryption before upload, and encrypts traffic between regions and zones at Layer 2 with Media Access Control Security (MACsec), VM-to-VM traffic within and peered Virtual Private Cloud (VPC) networks transparently, application traffic with Transport Layer Security (TLS) to HTTPS endpoints, and VPC to on-premises traffic with Cloud VPN encrypted tunnels. Because FIPS 140 validation is required for controls such as FedRAMP SC-13 Cryptographic Protection, per-region CMEK and HSM-backed keys give regulated multi-region jobs both residency and provable cryptographic control.

2.3Blocking and Auditing Cross-Region Data Transfers

Cross-region transfer governance combines a network perimeter that overrides identity permissions with continuous inspection and key-level audit. Storage services holding sensitive data sit inside a VPC Service Controls perimeter that blocks data access from outside the perimeter, even when Identity and Access Management (IAM) settings are permissive or misconfigured, and Access Context Manager grants entry only to designated IAM principals that match an access level template. Sensitive Data Protection (opens in a new tab) continually inspects, classifies, and de-identifies repository data by masking values with configured templates and options, so personally identifiable information (PII) such as credit card numbers, addresses, and phone numbers is discovered and tokenized before broader use.

Data flows from upload through enforcement in an ordered sequence:

1. Upload with identity check: A user with appropriate IAM permissions uploads data to BigQuery or Cloud Storage configured for CMEK and Cloud KMS Autokey.
2. Regional encryption: The storage service fetches the CMEK from Cloud KMS Autokey and encrypts the data under envelope encryption.
3. Inspection and de-identification: Sensitive Data Protection inspects and classifies the stored data and masks sensitive fields using templates.
4. Perimeter authorization: Designated IAM principals present context that matches the Access Context Manager access level template to reach data inside the VPC Service Controls perimeter.
5. Default denial: VPC Service Controls denies access to any principal not specified in the access level template, preventing exfiltration to outside projects, regions, or public access.

Audit completes the boundary by making key use observable and deniable to the provider. Key Access Justifications integrates with Cloud KMS and Cloud EKM to attach a detailed justification to every request to encrypt or decrypt with a customer key and to allow an automated policy to approve or deny that access. When combined with Access Approval and Access Transparency, which reveal administrator access with geographic and compliance attributes, the organization becomes the arbiter of decryption and can prove that cross-region transfers did not occur without approved justification.

Optimizing data locality and network performance means placing compute jobs in the same zone or region as the Cloud Storage data they read and write, choosing the right bucket location scope, and reducing cross-network traffic. Data flows from the storage bucket over the Google Cloud network to the compute resource, so greater physical distance adds latency, reduces throughput, and can add transfer and replication charges. A resilient design therefore pairs a durable source bucket with a colocated performance layer and runs each job where its working data lives.

3.1Place Compute Close to Storage

Compute placement determines how far every read and write must travel on the network. Storing data in the same zone or region as Compute Engine virtual machines or Google Kubernetes Engine (GKE) nodes provides better performance, with same-zone colocation giving the lowest latency between request and response. For AI and ML work, provision the performance layer in the same AI zone as the GPUs and TPUs so accelerators stay saturated and maximize goodput, which is useful throughput.

An AI zone (opens in a new tab) is a specialized zone for accelerator capacity that belongs to a parent region, such as europe-west4-ai1a in europe-west4, but it can be physically separate from that region's standard zones. Traffic flows from the AI zone compute to standard-zone regional services across that separation, which adds network latency. Run non-ML workloads in standard zones, not AI zones, because AI zones do not offer all Google Cloud services locally.

3.2Choose Regional, Dual-Regional, and Multi-Regional Storage

Bucket location controls the tradeoff between locality, availability, and cost for jobs that run in multiple zones or regions. Regional buckets replicate synchronously across multiple zones in one region and keep compute and storage colocated, while dual-region and multi-region buckets replicate asynchronously across regions for higher availability and disaster recovery. Multi-region storage can be cost-effective for moderate performance and ad hoc analytics, but performance can vary over time because storage and compute are not colocated.

Choose a regional deployment archetype when the workload must survive zone outages within one region, and choose a multi-regional archetype in active-active or active-passive mode when the workload must survive region outages. Dual or multi-regional deployments can also reduce latency and improve throughput through closer resource proximity to users or secondary compute, as described in disaster recovery for cloud infrastructure outages (opens in a new tab).

3.3Accelerate Access with Rapid Cache and Rapid Bucket

A tiered storage architecture separates durability from speed with a cold storage layer and a performance layer. The cold layer is a regional Cloud Storage (opens in a new tab) bucket in standard zones that holds training datasets and model checkpoints as the persistent source of truth. The performance layer is specialized zonal storage that acts as a high-speed cache or temporary scratch space in the same zone as the job.

Rapid Cache is a fully managed, SSD-backed zonal read cache that brings frequently read data from a source bucket into the zone. When a training job first reads a file, the file is pulled into the fast in-zone cache, and subsequent reads are served directly from the cache, bypassing the regional network. This pattern fits repetitive access in model training and low-latency model serving. Before the primary training epoch, perform a pre-read of the dataset to warm the cache so the first epoch does not stall on regional fetches.

Rapid Bucket is a high-performance capability that locates a bucket in a zone using the Rapid storage class for I/O-intensive read and write work. Create the zonal bucket, then mount it as a local file system with Cloud Storage FUSE (Filesystem in Userspace) (opens in a new tab), which lets applications use standard file-like semantics against Cloud Storage. Configure the application to store or access data through the mount point, for example by pointing the checkpointing directory to the mount point in training code. Rapid Bucket fits high-throughput reads and writes, high-speed checkpointing, and low-latency training and serving.

Protect zonal working data with an ordered backup lifecycle:

1. Write hot data in-zone: The job writes checkpoints and scratch output to the Rapid Bucket in the same zone as compute.
2. Copy for regional resilience: Data flows from the zonal bucket to a regional, dual-regional, or multi-regional bucket using Storage Transfer Service.
3. Expire temporary objects: Object Lifecycle Management deletes the zonal copies after a set time period to control storage cost.

3.4Control Network Egress and Cost

Network cost follows data movement across zone, region, and multi-region boundaries. Cross-zone and cross-region reads pay in latency and transfer charges, while wider replication adds storage and replication cost. Keeping the working set in-zone with Rapid Cache or Rapid Bucket reduces repeated regional fetches, and Rapid Cache eliminates data transfer fees when used with multi-region buckets.

Apply three economic rules alongside performance tuning. First, colocate stored data in the same zone as the compute that accesses it for the hottest datasets. Second, store short-lived datasets in regional buckets to avoid replication charges, and reserve dual-region or multi-region buckets for data that needs cross-region availability. Third, use the Rapid Cache recommender to analyze data usage and storage before enabling cache for a multi-region workload with strict latency requirements.

Key Facts

• Regional resources spread across zones survive a zonal outage, while only multi-regional resources or a separate failover region survive loss of an entire region.
• Regional failover recovery follows an ordered sequence: activate the failover stack, promote the database replica to primary, then reroute traffic with Cloud DNS to the failover load balancer.
• Assured Workloads with the Resource Locations organization policy restricts new resources to approved regions, so jobs can run in multiple approved regions without violating data residency.
• VPC Service Controls blocks data access from outside its perimeter even when IAM permissions would allow it, and Access Context Manager grants entry only to principals matching an access level.
• Colocating compute in the same zone or region as Cloud Storage data gives the lowest latency and lowest transfer cost, with Rapid Cache providing zonal read acceleration and Rapid Bucket providing zonal high-throughput read and write storage.