Design, migrate, and run scalable, highly available databases on Google Cloud across relational and NoSQL services, and track your readiness for the Professional Cloud Database Engineer exam.
Prepare and test your skills
Prepare and test your skills
Disaster Recovery Simulation and RTO/RPO Assessment is the practice of intentionally disrupting zonal and regional resources to prove that failover works and to measure actual recovery against targets. A simulation uses disaster recovery (opens in a new tab) workflows in a controlled test, restores backups in an isolated environment, runs the full application stack, and compares measured downtime and data loss to the established objectives. Recovery Time Objective (RTO) answers how long after a disaster before service runs again, while Recovery Point Objective (RPO) answers how much data loss in time is acceptable.
Smaller RTO and RPO values mean higher cost, complexity, and administrative overhead because they require more replication, more standby capacity, and more frequent backups. In Google Cloud, planning is simplified to scope of loss in zones and regions (opens in a new tab): a zonal outage, a regional outage, or the extremely unlikely outage of multiple regions. Business-critical operations subject to RTO and RPO must depend only on data plane components that continuously process operations, not on management plane operations such as a VM-creation API or an IAM permission update.
Recovery Time and Recovery Point Objectives are the two measurable targets that define disaster recovery success for databases and applications (opens in a new tab). RTO is the maximum acceptable length of time an application can stay offline, and RPO is the maximum acceptable length of time for which data might be lost. RPO varies by data use: frequently modified user data might have an RPO of minutes, while infrequently changed back-office data might tolerate hours.
Two variables control the achievable recovery point: backup frequency and backup retention. Backup frequency sets how much new data is at risk, because a disaster just before the next backup loses almost a full interval of updates. Backup retention sets how far back recovery can go, which protects against logical errors and corruption by allowing restore to a point before the error occurred. For recovery time, the controlling variable is readiness at the recovery site, from stopped virtual machines and templates up to a fully running hot standby system with infrastructure, software, file systems, and data ready.
Exam tip: The interval between backup or transfer jobs must not be longer than the RPO — if the RPO is 1 day, schedule the job at least once a day; if the RPO is 15 minutes, back up at least every 15 minutes.
High Availability Compared With Disaster Recovery explains two different protections with different scopes, replication, and recovery expectations. High Availability (HA) protects against zone failure inside one region, while Disaster Recovery (DR) protects against failure of an entire region across two or more regions. Simulations must test both scopes because even a regional product designed for 99.99% availability can still experience nearly an hour of downtime per year.
| Approach | Failure scope | Replication and topology | Recovery expectation |
|---|---|---|---|
| High Availability (HA) | Multiple zones within a single region | Synchronous replication between zones by default in many managed services, in an active-active topology with no single point of failure | Zero or near-zero RTO and RPO |
| Disaster Recovery (DR) | Two or more regions, without relying only on multi-regional services | Typically asynchronous replication to a secondary region, with manual or semi-automated failover | Larger RTO and RPO that must be measured against replication lag and failover steps |
Multi-regional services trade latency, consistency, and cost, most often through synchronous versus asynchronous replication. Asynchronous replication offers lower latency at the cost of risk of data loss during an outage, which directly increases RPO. Managed multi-regional services such as Spanner and Cloud Storage provide built-in redundancy and synchronization across regions and reduce manual recovery steps compared with self-managed replication between regional resources.
Simulation of zonal and regional disruptions validates that detection, failover, and notification workflows behave as designed under a real scope of loss. Google Cloud validates its own platform by physically disconnecting network connectivity to an internal-only, production-equivalent region to observe how products behave during total zonal or regional failure. Customer teams apply the same principle with fault injection testing and failure-scenario simulations that interrupt zones or regions for 1, 5, 10, or 30 minutes.
A well-designed simulation answers the same set of operational questions at every layer, including application, data, and cross-dependencies:
Test systems must be isolated from primary systems and users while primary systems continue to run. Isolation prevents conflicts or split brain scenarios where two databases both accept writes. After mitigation is in place, the failover and recovery path must be tested regularly, because an untested plan often fails when a real regional outage occurs.
Measurement of recovery compares three success criteria from each simulation: data integrity, actual RTO, and actual RPO. Data integrity confirms that backups contain consistent, usable snapshots that can immediately bring applications back into service, not just files that were copied. Actual RTO measures elapsed time from disaster to resumed service, and actual RPO measures the age of the restored data compared with the point of failure.
Validation combines automated checks, frequent backups, and health monitoring across the full stack:
Snapshot and retention policies directly shape measured results. Compute Engine persistent disk snapshots can be automated on hourly, daily, or weekly schedules, with only incremental block-level changes stored after the first snapshot, while Cloud Storage bucket retention policies control how long copies are kept. Because snapshots control consistency only at the block level, suspending writes during snapshots helps ensure file-level consistency for database and application file systems.
Failover and Redundancy Validation is the practice of proving that a standby database and application environment can take over production traffic within recovery goals. Validation combines failover drills, disaster recovery simulations, and monitoring to confirm that backups, replicas, templates, and network routing all work together. In Google Cloud, validation covers both self-managed databases on Compute Engine and managed services that provide built-in replication and high availability.
Cold, Warm, and Hot Redundancy Models describe how ready the standby environment is when a disaster occurs. A cold environment is not running and must be built from templates, backups, and logs, while a warm environment keeps a minimal database running to receive replication, and a hot environment runs production workloads in two places at once. Smaller RTO and RPO values require more continuously running resources, so cost rises as recovery gets faster and data loss gets smaller.
| Model | Standby state | Recovery characteristics |
|---|---|---|
| Cold | No running servers; custom images, Deployment Manager templates, and backups in Cloud Storage wait for use | Highest RTO and RPO, lowest running cost; recovery requires creating deployment, restoring backup, and applying logs |
| Warm standby | Minimal database instance runs at all times and receives replicated transactions through asynchronous or semisynchronous replication | Smaller RTO and RPO than cold without full high-availability cost; good trade-off between budget and availability |
| Hot | Both environments serve production traffic and stay synchronized | Near-zero RTO and RPO, highest cost and complexity; failover only requires resynchronizing databases and adjusting routing |
The choice of model determines what a failover test must prove. A cold test proves that images and backups can rebuild the system, a warm test proves that replication is current and the standby can scale up, and a hot test proves that traffic can shift without data loss.
Manual and Automated Failover Operations move production work from a failed primary to a standby through an ordered recovery sequence. A typical cold failover from on-premises to Google Cloud moves data from the backup source to the recovery network and then redirects clients to the recovered servers. Manual execution is slower and depends on operator knowledge and availability, while automation implements the steps more quickly and consistently once the process is proven.
A typical cold failover from on-premises to application environments in Google Cloud (opens in a new tab) follows this order:
Best practice is to start with a documented manual process, run it regularly, refine the operations playbook after each run, and then automate only after the manual process is reliable. Automated recovery also occurs at the infrastructure level when a managed instance group (MIG) detects failure through health checking, automatically recreates the virtual machine (VM) from the instance template, and attaches a persistent disk rebuilt from the latest snapshot.
Exam tip: Internal load balancing preserves the same database IP address after automatic replacement, while cross-environment failover requires updating Cloud DNS to point clients to the surviving web server.
Failback and Post-Restart Tasks return production traffic to the original environment after it can support workloads again. Failback for a cold recovery reverses the failover flow, with data flowing from the temporary Google Cloud database back to the restored primary. The sequence requires careful ordering because clients must be stopped before final logs are copied.
Failback for cold recovery from Google Cloud to the restored primary follows this order:
When both environments already support production in a hot design, failback is simpler because no primary change is needed. After the second environment is running again, resynchronize the databases and then allow traffic to distribute across both environments again by adjusting DNS settings.
Verifying Standby Availability and Connection Impact confirms that the replacement database has the correct version, configuration, security controls, and data before clients connect. Because a replacement created from an instance template inherits the same identity and access management (IAM) policies and firewall rules as the original, validation focuses on data currency and connectivity rather than rebuilding security. Persistent Disk snapshots define RPO by their frequency, and the latest transaction logs written to the disk are restored with the replacement instance.
Application connection impact depends on where the boundary between the client and database sits. An internal load balancer fronting a managed instance group keeps the same IP address for the database server, so clients need minimal reconfiguration when a replacement instance starts in the same zone. Cross-region or cross-environment failover crosses a DNS boundary, so clients depend on Cloud DNS updates and must tolerate propagation delay and potentially higher latency to the new zones and regions (opens in a new tab). Architectures that require zero data loss require data to be written in multiple locations before the write is committed, which reduces RPO toward zero but increases latency and can reduce application performance.
Monitoring, Alerting, and Regular Validation detect failure and confirm whether failover thresholds and recovery procedures work as designed. A monitoring procedure built on Cloud Monitoring tracks system health continuously and sends alerts when service fails, so the team can start manual steps or verify that automatic steps began. The operations playbook must be created before a disaster and updated as architectures change.
Common items to track for database failover (opens in a new tab) include:
Regular testing validates cluster failover thresholds and ensures the landscape fails over properly to a new region during a localized disaster. Disaster recovery procedures must be tested on a schedule to prove that standby instances, connection routing, and recovery steps still meet RTO and RPO goals.
Backup Integrity and Point-in-Time Recovery (PITR) Testing is the practice of proving that backups and transaction logs can actually restore a working system to a specific moment. A backup is only useful if it holds a consistent, usable snapshot that can bring applications back into service immediately after data loss or corruption. Rigorous testing restores that data in a separate environment, replays logs to the target time, and confirms the full application works with the restored data.
How Recovery Success Is Measured uses three criteria: data integrity, recovery time objective (RTO), and recovery point objective (RPO). Data integrity means the restored data is consistent, complete, and meets application requirements. The recovery time objective (RTO) is the maximum acceptable time the application can stay offline, or how long after a disaster before service resumes. The recovery point objective (RPO) is the maximum acceptable amount of data loss measured in time, or how much recent data the business can afford to lose.
Smaller RTO and RPO values require more frequent backups, more automation, and higher cost and complexity. Business continuity requirements set these targets, and periodic restoration tests verify that the organization can still meet them. Besides measuring time and data loss, a complete test must bring up the entire application stack and critical infrastructure services with the restored data to confirm correct behavior.
Verify Backup Consistency and Integrity confirms that each backup contains a restorable snapshot rather than a partial or corrupted copy. The process runs automated consistency checks after each backup to validate data integrity before the backup is trusted for recovery. Google Cloud Backup and DR service tracks backup health and confirms that copies are stored in secure, reliable locations. Backups are replicated across multiple regions for added resilience against zonal or regional loss.
Regular simulation of recovery from data loss (opens in a new tab) ensures backups can be restored efficiently under real conditions. Teams document the restoration steps and train to execute them during a failure, so recovery does not depend on memory or improvisation. Monitoring with Cloud Monitoring adds alerts when backup intervals exceed the defined RPO or when backup health degrades.
How Point-in-Time Recovery Works rebuilds a database to a specific moment by combining a base backup with the transaction logs written after that backup. Data flows from the stored backup to the recovery system first, then from the stored transaction logs to the same recovery system in order. Only after the latest backup is applied and the subsequent logs are replayed does the database reach the target point in time.
Database recovery depends on the recovery system mirroring the production system, including the same database version and mirrored disk configuration. For critical applications, near-continuous protection such as Cloud SQL PITR or incremental backups for large datasets reduces gaps between scheduled backups. Spanner PITR supports recovery to a point in the past up to a maximum of 7 days, which helps after operator errors or bad application rollouts.
Database backups and simple data backups differ in what must be restored:
| Backup type | What is restored | What makes recovery work |
|---|---|---|
| Data backups | A discrete copy of files or objects copied from one place to another | A known-good copy restored directly to production or to a recovery environment, typically with small to medium RTO and small RPO |
| Database backups | A base backup plus transaction logs replayed to a point in time | Matching version and configuration on the recovery system, plus ordered application of the latest backup followed by logs after that backup |
Database backups carry extra complexity because version mismatch or missing logs prevents a correct point-in-time state. A high-availability-first design that keeps a current standby or replica helps achieve smaller RTO and RPO when full log replay would take too long.
Align Backup Frequency With RPO determines how much data can be lost, so the interval between backups must align with the RPO target. If the RPO is 15 minutes, backups must run at least every 15 minutes to keep potential loss within the target. Optimizing intervals reduces the risk that a failure occurs after a long gap with no usable copy.
Google Cloud tools schedule and manage this cadence, including Cloud Storage for backup objects, Cloud SQL automated backups, and Spanner backups. Cloud Storage tiered classes such as Nearline, Coldline, and Archive lower storage cost for infrequently accessed backups, but they add retrieval costs and minimum storage durations. Cloud Monitoring watches adherence to the RPO and sends alerts when jobs run late, fail, or exceed the defined objective.
Exam tip: RPO controls backup frequency — schedule backups at least as often as the RPO allows data loss, such as every 15 minutes for a 15-minute RPO.
Test Restores in a Non-Production Environment prove that backups, logs, infrastructure, and procedures work together without risking live service. The isolated test environment receives the restored data, then runs the full application stack (opens in a new tab) so dependencies on compute, networking, security controls, and configuration are also validated. Testing only the database files is not enough, because the deployed application must work correctly with the restored data.
A rigorous restoration test for database systems (opens in a new tab) follows an ordered workflow:
This sequence confirms the dependency between backups, logs, and the application, and it exposes gaps in versions, permissions, firewall rules, or DNS changes before a real disaster. For scenarios beyond backups alone, combine tested restores with strategies such as cross-region replication or active-active failover to improve recovery time in extreme cases.
Last updated