Cloud SQL and Redis Replication and Failover

Replication topologies for managed databases define how writes flow from a primary node to standby or replica nodes and how those replicas take over during failure. The topology balances write latency, data consistency, and recovery point objective (RPO) across zones and regions. A regional synchronous standby protects against zonal failure with minimal data loss, while a cross-regional asynchronous standby protects against regional outage with lower write latency and high availability (opens in a new tab).

1.1Synchronous and Asynchronous Replication Models

Synchronous and asynchronous replication models control whether the primary waits for the standby before confirming a write. Synchronous replication requires the primary to wait for standby acknowledgment, which increases write latency but provides strong consistency and near-zero RPO. Asynchronous replication lets the primary acknowledge first and send data later, which keeps latency low but allows recent writes to be lost if the primary fails.

The choice depends on distance and business priority. Synchronous replication fits the same region in a different zone, where the network round trip is short and data integrity is critical. Asynchronous replication fits a distant region, where waiting would make application response time unacceptable. A mixed three-node topology uses both: the primary replicates synchronously to a nearby standby for high availability and asynchronously to a distant standby for disaster recovery.

Data in the mixed topology flows from the primary to the nearby synchronous standby and separately from the primary to the distant asynchronous standby. A zone failure is survived through the local standby without data loss, while a regional outage is survived through the remote standby.

1.2Memorystore for Redis Failover and Data Protection

Memorystore for Redis Standard Tier uses a primary node that replicates to a replica node, while Basic Tier has no replica and cannot fail over. A normal failover occurs automatically when the primary becomes unhealthy and the replica is promoted to the new primary. A manual failover is initiated by the operator to test application behavior, and for Standard Tier instances the data is preserved and the IP address and connection string remain the same. The instance undergoes a short connection reset of a couple of minutes or less, so applications need retry logic to reconnect.

Manual failover supports two data protection modes that control how much replication lag is tolerated. The limited-data-loss mode, which is the default, checks the offset delta between primary and replica and aborts if the greatest delta is 30 MB or greater. The force-data-loss mode skips the offset check and uses aggressive failover strategies, so failover proceeds even when more than 30 MB of changes have not yet replicated. To preserve data with the default mode, operators can stop application writes and wait until lag drops below 30 MB before initiating failover.

1.3Monitoring Replication Lag and Complementary Persistence

Monitoring replication lag and complementary persistence shows whether a replica is keeping up and what remains if both nodes fail. The bytes pending replication metric measures how many bytes the replica still needs to copy to match the primary, and the primary offset increments for each byte that must be synchronized. Operators can view this metric on the instance details page in the Google Cloud console or in Metrics Explorer as redis.googlapis.com/replication/offset_diff. High or rising values indicate lag, while an empty value after a hardware error means the offset could not be obtained until the new replica is repaired.

Memorystore for Redis Cluster exposes cross-region replication health through offset and link metrics that show whether secondary shards are keeping up with the primary cluster:

• secondary_replication_links shows the number of shard links between primary and secondary clusters, which should equal the primary shard count and signals broken replication when it drops below that count.
• secondary_maximum_replication_offset_diff measures the maximum replication offset difference in bytes between primary and secondary shards across regions.
• secondary_average_replication_offset_diff measures the average replication offset difference in bytes, where high values indicate lag that can be addressed by pausing and resuming replication.

Replication is complemented by RDB persistence, which saves snapshots of cluster data to durable storage on a schedule defined by start time and an interval of 1h, 6h, 12h, or 24h. Nodes with replicas recover from the replica first rather than loading from a snapshot, but if primary and replica fail together they recover from the most recent successful snapshot. Only the latest snapshot is retained and some staleness is expected, so a highly available instance with replicas should be used in addition to persistence.

Monitor and Optimize Replication Performance and Health is the practice of tracking how far replicas lag behind the primary and whether replication work overloads the clusters. Replication lag directly controls data-loss risk at failover, while replication itself consumes CPU, network, and nodes on every participating cluster. Healthy operations keep lag near zero, size replicas to handle both application load and replication work, and test failover paths before a disaster requires them.

2.1Replication Lag and Data-Loss Risk

Replication lag is the delay between a write being accepted on the primary and the same change becoming available on a replica. In Cloud SQL (opens in a new tab) advanced disaster recovery (DR), the designated DR replica is a directly connected cross-region read replica, and its lag at promotion determines whether data is lost. A switchover waits until replication lag goes down to zero before the new primary starts accepting connections, so the switchover results in zero data loss because the old primary remains in read-only mode until the DR replica catches up. A replica failover promotes the DR replica immediately when the primary region is unavailable, so the replica failover can result in data loss if the DR replica still has unreplicated changes.

The same lag tradeoff appears in Oracle Data Guard (opens in a new tab) on Bare Metal Solution. The Silver model uses asynchronous replication and typically achieves an RPO of less than 60 seconds, which means up to 60 seconds of recent writes can be lost. Synchronous replication ensures zero data loss, but sending data between regions typically drives application response time beyond acceptable limits. The Gold model addresses this with a far sync instance that runs geographically near the primary database and includes a database control file plus standby redo logs, allowing all changes to be recorded synchronously with low latency outside the primary database's region extension.

2.2Bigtable Throughput and Latency Trade-Offs

Bigtable (opens in a new tab) replication improves read throughput and can reduce read latency by placing data closer to users, especially when multi-cluster routing sends each request to the closest cluster. Write throughput does not increase with replication because a write to one cluster must be replicated to all other clusters in the instance. Each cluster expends CPU resources to pull changes from the other clusters, so write throughput might actually go down because replication requires each cluster to do additional work beyond serving application traffic.

The difference between scaling up and scaling out illustrates the tradeoff. Adding 3 nodes to a single 3-node cluster, for a total of 6 nodes in one zone, doubles write throughput for that instance but leaves data available in only one zone. Adding a second 3-node cluster instead, for a total of 6 nodes across two zones, means the instance writes each piece of data twice — once when first received and again when replicated — so write throughput does not increase and might decrease, but data becomes available in two zones. For this reason, a cluster in a multi-cluster instance often needs more nodes than a cluster in a single-cluster instance with similar traffic. Replication latency also depends on distance: replicated clusters in different regions typically have higher latency than clusters in the same region, and multi-cluster routing is eventually consistent.

2.3Routing, Deadlines, and Failover Health Signals

App profiles control how Bigtable traffic flows to clusters and therefore how replication performance is observed. An app profile with multi-cluster routing automatically routes requests to the closest cluster from the perspective of the application, and writes are then replicated from that cluster to the other clusters in the instance. This automatic choice of the shortest distance results in the lowest possible latency. An app profile with single-cluster routing directs all traffic to one cluster, which can be optimal for separating workloads or for read-after-write semantics on a single cluster, but single-cluster routing will not reduce latency in the way multi-cluster routing does.

Failover behavior follows the routing choice. If an app profile uses single-cluster routing, failover is manual and depends on operator judgment about cluster health. Useful signals that failover might help include:

• The cluster starts to return a large number of transient system errors.
• A large number of requests start timing out.
• The average response latency increases to an unacceptable level.

These Bigtable signals can appear for many reasons, so failing over is not guaranteed to resolve the underlying issue, and operators should monitor the instance before and after failover to verify improvement. Bigtable reverts to eventual consistency after a failover, so reads from the new cluster can return stale data until outstanding writes finish replicating. If an app profile uses multi-cluster routing, Bigtable handles failovers automatically by routing traffic to the nearest available cluster, even during very short unavailability such as slow replies or transient errors. A shorter request deadline triggers an earlier reroute to the next closest cluster, while a longer deadline allows more time before failing over, but the deadline must remain long enough for the retried request to complete on the subsequent cluster. Bigtable resolves conflicting writes during failover with an internal last write wins algorithm.

2.4Sizing, Continuity Checks, and Zero-Loss Return to Primary

Sizing, continuity checks, and zero-loss return to primary ensure a replica can absorb production traffic without degrading service. In Cloud SQL advanced DR, all instances involved in replica failover and switchover retain their IP addresses, which simplifies reconnection but does not remove capacity risk. To reduce recovery time objective (RTO) after failover, the DR replica should be configured with the same size as the primary instance, and if the primary has high availability (HA) enabled, HA should also be enabled on the DR replica. A switchover also serves as a routine DR drill that tests whether lag, sizing, and endpoints are healthy before a real outage.

A switchover operation follows an ordered sequence:

1. Assign DR replica: Assign a directly connected cross-region read replica as the designated DR replica while both primary and replica are online.
2. Initiate switchover: Start the switchover from the healthy primary.
3. Drain lag to zero: Hold the old primary in read-only mode until replication lag reaches zero.
4. Promote new primary: Start accepting incoming connections on the new primary and convert the old primary into a read replica.
5. Update endpoint: Point the DNS write endpoint to the new primary instance, or reconfigure applications if no DNS write endpoint is used.

The distinction between database recovery and full application recovery matters for monitoring goals. In the Silver Data Guard model the database RTO may be under 1 minute, but application failover might take minutes to hours, and running applications and supporting databases in different regions increases network latency. Reverse replication for Spanner (opens in a new tab) migrations shows the same ordering concern on the fallback path: the flow reads changes on Spanner, optionally disables forward change data capture (CDC) before enabling reverse flow, captures and temporarily orders all changes because Spanner is distributed, then writes ordered data to the source while handling data-type changes and sharding.

Automated failover is the design of health detection, promotion logic, and connectivity handling so a standby database assumes primary traffic without manual intervention. In Google Cloud, automated failover divides into two models: self-managed databases on Compute Engine that recover through a managed instance group (MIG) and internal load balancer, and Cloud SQL (opens in a new tab) Enterprise Plus advanced disaster recovery (DR) that promotes a designated cross-region DR replica. The first model rebuilds from snapshots behind a stable IP, while the second promotes a hot replica across regions.

3.1Self-Managed Recovery Without a Synchronized Standby

Self-managed recovery without a synchronized standby rebuilds a database server from a known image and data snapshot instead of maintaining a hot replica. The production database writes database files and log files to an attached persistent disk, while a scheduled task creates regular snapshots of that disk. A custom image holds the database software version and a startup script that creates a new persistent disk from the latest snapshot and mounts it. An instance template built from that image guarantees the replacement instance is configured identically to the original, including IAM policies and firewall settings.

When the original instance fails, the managed instance group with a target size of 1 automatically brings up another database server of the correct version in the same zone and attaches a persistent disk containing the latest backup and transaction log files. The internal load balancer in front of the group preserves the same IP address for the database server, so clients do not need reconfiguration. Standard persistent disks are zonal, so snapshots are required to re-create disks after a zonal failure, and snapshots can restore across regions. Using regional persistent disks with synchronous replication removes the snapshot-restore step but changes budget and RTO and RPO tradeoffs.

3.2Health Checks and Failover Triggers

Health checks and failover triggers are the monitoring and group logic that decide when a replacement or promotion occurs. For the Compute Engine pattern, health checking configured with Cloud Monitoring metrics lets the managed instance group react to failures at the application level and the instance level. The group provides HA in depth because it detects an unhealthy instance, terminates it, and recreates a new instance from the instance template without operator action.

For Cloud SQL advanced DR, the trigger is an operator-invoked replica failover or switchover against a designated DR replica, used when the primary region has failed or for planned testing and return. Before invocation, the DR replica must be online and healthy and running the same database version as the primary, with point-in-time recovery (PITR) enabled on the primary. Excessive replication lag between primary and replica nodes increases inconsistency and data loss on failover, so monitoring replication lag with views such as pg_stat_replication and alerting on significant increases is part of trigger readiness.

3.3Cloud SQL Advanced DR Promotion Logic

Cloud SQL advanced DR promotion logic defines how a cross-region read replica becomes the new primary, and it is supported only on Cloud SQL Enterprise Plus edition instances. A DR replica must be a direct read replica in a separate region from the primary, not a cascading or external replica, and it must store transaction logs used for PITR in Cloud Storage. Additional version and configuration gates apply for MySQL and PostgreSQL to ensure the replica can safely take writes.

MySQL replicas must run MySQL 8.0.31 or later with binary logging enabled, match the primary major and minor version, and must not use filtering flags such as replicate_do_db, replicate_ignore_db, replicate_do_table, replicate_wild_do_table, replicate_ignore_table, or replicate_wild_ignore_table. PostgreSQL replicas must run PostgreSQL 12 or later, match the primary version, and must not have the cloudsql.logical_decoding flag, logical slots, or logical replication configured. Flags that require the replica value to be greater than or equal to the primary, including max_connections, max_worker_processes, max_wal_senders, max_prepared_transactions, max_parallel_workers, max_locks_per_transaction, max_parallel_maintenance_workers, and max_parallel_workers_per_gather, must be equal to the primary.

After replica failover completes, Cloud SQL takes a backup of the new primary in a separate asynchronous operation to fully re-enable PITR, which can take 5 to 15 minutes depending on disk size, during which PITR is not available. When the old primary comes back online, the process takes a backup and then recreates the old primary as a read replica of the new primary.

3.4Application Connectivity and Session Impact During Failover

Application connectivity during failover depends on whether clients connect through a stable endpoint or a direct instance address. In the self-managed MIG pattern, traffic flows from clients to the internal load balancer and then to the single active instance in the group, so the load balancer preserves connectivity by keeping the same IP address after replacement. In Cloud SQL advanced DR, traffic flows from applications to the DNS write endpoint if configured, and that endpoint automatically resolves from the old primary instance to the new primary instance after promotion.

If applications do not use a DNS write endpoint, they must be repointed to the IP address of the new primary instance, which breaks existing sessions and requires reconnection. Two configuration dependencies determine whether reconnection succeeds: the primary instance and DR replica must have the same SSL configuration before switchover or failover, or clients enforcing different encryption modes will fail to connect to the new primary. Similarly, Private Service Connect outbound connectivity configured on the DR replica continues to work after promotion, while missing outbound configuration on the DR replica requires enabling it on the new primary for dependent services to continue.

3.5Data Consistency, PITR, and RTO and RPO Tradeoffs

Data consistency during automated failover is controlled by replication type, snapshot frequency, and log recovery. HA within a single region typically uses synchronous replication between zones with zero or near-zero RTO and RPO and an active-active topology without a single point of failure. DR across two or more regions typically uses asynchronous replication, so RPO depends on how recently data was transferred and RTO depends on manual or semi-automated promotion and client redirection.

For snapshot-based recovery, RPO is dictated by the frequency of persistent disk snapshots, because the replacement disk contains only the data and transaction logs written up to the last snapshot. For continuous replication, Persistent Disk Asynchronous Replication provides low-RPO and low-RTO block storage replication for cross-region active-passive DR at the infrastructure level rather than the workload level. To avoid performance degradation after promotion, the DR replica should match the primary disk size or enable auto-growth, use the same machine type and size, mirror HA and data cache configuration, and carry appropriate database flags before and after switchover or failover.

Key Facts

• Synchronous replication waits for standby acknowledgment for strong consistency and near-zero RPO with higher write latency, while asynchronous replication acknowledges first for lower latency with larger potential data loss.
• Memorystore for Redis manual failover with default limited-data-loss mode aborts when bytes pending replication is 30 MB or greater, while force-data-loss mode skips that check.
• Cloud SQL switchover holds the old primary in read-only mode until lag reaches zero for zero data loss, while replica failover promotes immediately and can lose data equal to the lag.
• Bigtable replication improves read throughput and availability across zones but does not increase write throughput because each cluster must replicate and process every write.
• After Cloud SQL replica failover, point-in-time recovery (PITR) is unavailable for 5 to 15 minutes while a backup of the new primary completes.