Design, migrate, and run scalable, highly available databases on Google Cloud across relational and NoSQL services, and track your readiness for the Professional Cloud Database Engineer exam.
Prepare and test your skills
Prepare and test your skills
Validating recovery strategies through disaster recovery testing is the practice of regularly proving that database backups can be restored within recovery time objective (RTO) (opens in a new tab) and recovery point objective (RPO) targets. RTO answers how long after a disaster before service is running again, while RPO answers how much data can be lost. A test is only successful when restored data is correct, restoration finishes within RTO, and data loss stays within RPO during zonal outages, regional outages, or data corruption events.
Successful recovery testing measures three criteria together: data integrity, RTO, and RPO. Data integrity confirms that backups contain consistent and usable snapshots that can immediately bring applications back into service. RTO measures restoration latency from disaster to normal service, while RPO measures how much recent data was lost compared to the last usable backup. The goal of testing is to verify that business continuity requirements are still met, not just that a backup file exists.
Smaller RTO and RPO values generally mean higher cost and greater operational complexity. This tradeoff is tracked as service level objectives (SLOs), which are measurable parts of a service level agreement (SLA) (opens in a new tab). High availability (HA) and disaster recovery (DR) support different ends of this tradeoff, so drills must validate which one the design actually delivers:
| Approach | Failure scope | Replication | RTO and RPO |
|---|---|---|---|
| High availability (HA) | Zone failure within one region | Synchronous replication between zones, often active-active with no single point of failure | Zero or near-zero RTO and RPO |
| Disaster recovery (DR) | Region failure across two or more regions | Typically asynchronous replication with manual or semi-automated failover | Larger RTO and RPO that must be measured during drills |
A disaster recovery drill (opens in a new tab) validates the full recovery path for databases, not just one backup job. After the DR plan is complete, the plan must be tested regularly, with issues noted and the plan adjusted after each test. Primary systems keep running during the test, so each drill must cover operational dependencies that affect restoration latency and backup integrity.
A complete drill tests all of the following areas:
Safe restoration testing restores backups in an isolated non-production environment (opens in a new tab) that cannot conflict with production users or data. The test environment must be isolated from the primary system and its users to prevent conflicts or split-brain scenarios where two databases both accept writes. Testing must then run the entire application stack and critical infrastructure services on the freshly restored data to confirm that the deployed application works correctly.
A repeatable restoration test follows an ordered workflow:
Documented restoration steps and trained teams are part of what is tested. Documentation must list the exact restoration steps, and teams must practice executing those steps under failure conditions so human actions do not add unexpected latency during a real regional outage or corruption event.
Exam tip: Backup frequency must be equal to or more frequent than RPO — if RPO is 15 minutes, schedule backups to run at least every 15 minutes and alert when intervals exceed RPO.
Testable RPO design depends on two variables: backup frequency and backup retention. Backup frequency controls the maximum time between the last backup and a disaster, so a 24-hour backup schedule can lose almost 24 hours of updates. Backup retention controls how far back recovery can go, which protects against logical errors and corruption by allowing recovery to a point before the error existed.
Google Cloud Backup and DR tracks backup health and confirms that backups are stored securely, while Cloud Monitoring sends alerts when backup intervals exceed the defined RPO. Common backup building blocks include Cloud Storage (opens in a new tab) for scheduled exports and tiered retention, Cloud SQL automated backups with point-in-time recovery (PITR), and Spanner backups with PITR up to 7 days in the past. For resilience, backups and Persistent Disk snapshots should be replicated or stored across multiple regions, because zonal disks alone cannot survive a regional outage. Replication reduces potential data loss toward near zero but costs more, so critical databases and business-critical files use it while less critical data uses less frequent backups.
Testable RTO design depends on how ready infrastructure, software, file systems, and data are at the recovery site. A hot standby system keeps pre-deployed infrastructure, active database systems, and replicated data running in the recovery region, so failover is fastest but cost is highest. A lower-cost option configures and deploys a Compute Engine virtual machine (VM) in advance but keeps it stopped, so attached disks or static IPs may still incur charges while the stopped VM itself does not. A third option defers VM creation until recovery using Terraform, an instance template, and a custom image created in advance.
Drills must measure the real time cost of the chosen readiness level, including provisioning, disk restore, transaction log replay, and traffic cutover. Business-critical operations that are subject to RTO must depend only on data plane components that continuously process operations, not on management plane operations such as VM-creation APIs or IAM permission updates. Regional mitigations must also be tested regularly after they are built, because an untested cross-region failover often fails when a real regional outage occurs.
Architecting for RTO and RPO requirements means translating business continuity needs into database architectures that limit downtime and data loss. The recovery time objective (RTO) (opens in a new tab) is the maximum acceptable length of time an application can be offline, or how long after a disaster before service is running again. The recovery point objective (RPO) is the maximum acceptable length of time for which data can be lost, or how much data loss can be afforded. Smaller RTO and RPO values require greater complexity and cost to run, so architecture choices balance availability against operational complexity, latency, and cost.
Defining RTO and RPO targets starts with grouping applications by criticality and by scope of loss. Criticality tiers assign stricter targets to customer-facing or revenue-critical systems and looser targets to internal or departmental systems. Scope of loss simplifies requirements to three cases in Google Cloud: a zonal outage, a regional outage affecting more than one zone, or the extremely unlikely outage of multiple regions. Together, criticality and scope answer whether an application must run in multiple zones in one region or in multiple zones across multiple regions, and which Google Cloud products it can depend on.
| Application criticality | Example apps | Zone outage target | Region outage target |
|---|---|---|---|
| Tier 1, most important | Real-time payments, eCommerce storefronts | RTO zero, RPO zero | RTO zero, RPO zero |
| Tier 2 | Regional CRM or ERP systems | RTO 15 minutes, RPO 15 minutes | RTO 1 hour, RPO 1 hour |
| Tier 3, least important | Back office, leave booking, HR, accounting | RTO 1 hour, RPO 1 hour | RTO 12 hours, RPO 12 hours |
Business-critical operations subject to RTO and RPO must depend only on data plane (opens in a new tab) components that continuously process operations. The data plane carries live reads and writes, while the management plane manages configuration state and pushes configuration to the control plane and data plane. A workload that depends on a VM-creation API or an Identity and Access Management (IAM) permission update during an outage cannot meet its RTO because those management operations may be unavailable when recovery is needed.
High availability (HA) is the architecture for surviving zone failures within a single region with zero or near-zero RTO and RPO. Many managed services provide synchronous replication between zones within one region by default, which keeps a standby copy current without data loss and supports an active-active topology with no single point of failure (SPOF). Because replication is synchronous inside the region, a zone interruption can fail over to another zone with minimal service disruption and minimal data loss.
Preferred HA building blocks use wider locational scope and automatic recovery. Spanner (opens in a new tab) provides built-in redundancy and synchronization across zones and regions for active-active serving. Cloud SQL with its HA feature provides a near active-active topology with replicas across zones and RPO close to zero. Cloud Storage as a multi-regional service, regional persistent disks with synchronous replication to another zone, regional Compute Engine managed instance groups (MIGs), and Google Kubernetes Engine (GKE) regional clusters all resist zone outages without manual restore from backup. To strengthen this pattern, use three zones per region rather than two, overprovision capacity by about a third to absorb failover traffic, and distribute traffic with load balancing so requests flow from clients to healthy zones automatically.
Disaster recovery (DR) is the architecture for surviving regional outages with workloads deployed across two or more regions. Unless the workload uses multi-regional or global services, cross-region replication is typically asynchronous, which offers lower latency at the cost of risk of data loss during an outage. That tradeoff means DR usually has higher RTO and RPO than HA, plus manual or semi-automated failover steps that must be identified and tested in advance.
Cross-region DR depends on replicating data out of the primary region before the failure. Regional or zonal resources must replicate data to a secondary region, to multi-regional storage such as Cloud Storage, or to a hybrid option such as GKE and Google Distributed Cloud. Snapshots of persistent disk data should be stored in multi-region locations so disks can be recreated in another region. Managed multi-regional services reduce this work because Spanner and Cloud Storage already distribute data across regions, which is why they are recommended when data replication is critical and when an active-active topology across two regions is required.
Exam tip: HA applies to regional deployments with synchronous replication between zones for zero or near-zero RTO and RPO, while DR applies to multi-region deployments with typically asynchronous replication and higher RTO and RPO.
Controlling RPO with frequency and retention and RTO with readiness separates data-loss controls from downtime controls. Backup frequency sets the maximum gap between the last backup and a disaster, so transfer or snapshot jobs must run at least as often as the RPO requires. Backup retention sets how far back recovery can go, which protects against logical errors by allowing restore to a point before corruption occurred. Incremental snapshots and tiered Cloud Storage (opens in a new tab) classes affect storage cost, while data transfer and retrieval affect transfer cost.
RTO is controlled by how much infrastructure, software, and data are already running at the recovery site. A hot standby system with pre-deployed infrastructure and replicated data gives very low RTO at higher cost. A lower-cost option pre-configures a virtual machine (VM) and then stops it, so static addresses and disks may still incur charges but the stopped VM does not. The lowest-cost option defers creation to recovery time using automation such as Terraform (opens in a new tab), an instance template, and a custom image, which is possible because a Compute Engine VM can be created relatively quickly. After any mitigation is in place, the plan must be tested regularly with isolated tests that prevent conflicts with production.
Implementing point-in-time recovery (PITR) mechanisms means enabling continuous change logs so a database can be restored to a specific timestamp between backups. Cloud SQL (opens in a new tab) uses binary or transaction logs, Spanner (opens in a new tab) retains versioned data for PITR, and AlloyDB for PostgreSQL (opens in a new tab) uses continuous backup with data-change logs. In all three cases, PITR depends on uninterrupted log retention, and a restore creates a new instance or cluster rather than overwriting the source.
How PITR controls RPO and RTO centers on combining a base backup with replayed logs. Backup frequency determines the maximum gap between the last backup and a disaster, while backup retention determines how far back in time a recovery point can be placed. PITR narrows RPO to seconds or minutes because the logs recorded after the last full backup are replayed up to the chosen timestamp. After the restore, database functionality must be verified and the recovery system must match the production configuration, so adopting high availability alongside PITR helps achieve smaller RTO and RPO values.
Cloud SQL PITR restores a primary instance to a specific timestamp by replaying logs on top of a backup to create a new clone. The restore depends on automated backups and continuous logging being enabled before the event to recover from, so there must be an unbroken chain of binary logs since the last backup before that event. For MySQL, Cloud SQL uses the mysqlbinlog utility to replay up to the stop time, and events at and after the timestamp are not reflected in the new instance.
Enabling and using PITR requires log retention to be configured:
backup-start-time in UTC.point_in_time_recovery_enabled and transaction_log_retention_days in Terraform and to --enable-point-in-time-recovery and --retained-transaction-log-days in gcloud.gcloud sql instances describe that pointInTimeRecoveryEnabled: true appears in backupConfiguration.A timestamp restore is the recommended path. Data flows from the source instance backup plus its binary logs to a new clone instance through an ordered sequence:
2012-11-15T16:19:00.094Z.Spanner PITR lets an application recover data from a specific point in time in the past, up to a maximum of 7 days. Spanner PITR is designed for logical errors rather than only infrastructure loss, such as when an operator inadvertently writes bad data or an application rollout corrupts the database. Because Spanner is a managed multi-regional service with synchronous replication, PITR complements its high availability by allowing a precise rollback without rebuilding an entire regional topology. The recovery window is bounded by retention, so the target timestamp must fall within the retained 7-day period.
AlloyDB PITR with continuous backup fully restores an active cluster's data from any point in time within a recent range into a new cluster. The restore depends on continuous backup being enabled, which continuously captures backups and data-change logs for the source cluster. AlloyDB PITR is recommended for rapid recovery from large-scale data loss because the new cluster is rebuilt to the target time without manual log replay.
The available PITR window starts after the more recent of two moments: the limit of the recovery window, for example 14 days in the past for a 14-day window, and the creation time of the oldest backup taken since continuous backup was last enabled. If the cluster was created with continuous backup enabled and never disabled, the window effectively extends back to the oldest backup. If continuous backup is disabled and re-enabled, PITR is blocked until the first new backup exists, either an on-demand backup or the first daily backup after re-enabling. The original cluster must still exist, although its primary instance may be deleted if the target time is before the deletion, and PITR is paused on the original primary during cross-region failover until it becomes secondary and a new baseline backup is established.
Comparing PITR across Cloud SQL, Spanner, and AlloyDB shows the same pattern with different log sources, retention limits, and restore targets. All three combine a base backup plus retained changes flowing to a new resource, but they differ in what enables PITR and how long history is kept.
| Service | What enables PITR | Retention limit | Restore result |
|---|---|---|---|
| Cloud SQL | Automated backups plus binary or transaction logging | 1-7 days for Enterprise edition, 1-35 days for Enterprise Plus edition | New instance created as a clone; cannot restore into the existing instance |
| Spanner | PITR retention on the instance | Up to 7 days in the past | Data recovered to the selected timestamp |
| AlloyDB for PostgreSQL | Continuous backup with data-change logs | Recovery window such as 14 days, limited by oldest backup since continuous backup was enabled | New cluster created in a selected VPC network, with optional customer-managed encryption key |
Restores started from vaulted backups are tracked by Backup and DR Service, where the job appears with Resource type Cloud SQL and Job category Restore or Point-in-Time Restore (PITR). Logs for the job are visible in both the workload project that holds the target instance and the backup vault project that holds the backup.
Exam tip: Cloud SQL PITR and AlloyDB PITR always create a new instance or new cluster; neither service performs PITR into the existing instance or cluster.
Last updated