Design, migrate, and run scalable, highly available databases on Google Cloud across relational and NoSQL services, and track your readiness for the Professional Cloud Database Engineer exam.
Prepare and test your skills
Prepare and test your skills
Online migration with continuous replication keeps a production database online while its data is copied to Google Cloud. The source database, such as Amazon RDS or Amazon Aurora, continues to serve reads and writes while changes flow to the target, such as Cloud SQL (opens in a new tab) or AlloyDB for PostgreSQL (opens in a new tab). The migration is complete only when traffic switches to the target, replication stops, and all reads and writes happen on the target instance. This pattern is also called trickle migration because work moves in small chunks.
Continuous replication fits mission-critical databases where downtime is expensive. It offers near-zero downtime and lower risk of data loss because a failure affects only one chunk, so rollback and retry take less time. The tradeoff is higher complexity, because setup needs more planning and time and connected applications often need extra refactoring.
Choosing between scheduled maintenance and continuous replication starts with how much downtime the workload can afford. Scheduled maintenance, also called one-time or big bang migration, takes the database offline to back up, transfer, restore, and switch applications. Continuous replication is required when the workload cannot afford downtime or cannot afford the cut-over window.
The cut-over window is the time needed to take a backup, transfer it to Cloud SQL, restore it, and switch applications. The decision follows two questions in order: first, can you afford any downtime, and then, can you afford the cut-over window. If the answer to either question is no, use continuous replication.
| Strategy | Downtime and risk | Cost, complexity, and recovery |
|---|---|---|
| Scheduled maintenance | Requires downtime for backup, transfer, restore, and switch; if migration fails, the process restarts and downtime grows | Lower cost and complexity with little application refactoring; best for small and infrequently used databases |
| Continuous replication | Near-zero downtime with lower risk of data loss; source stays online while target catches up | More complex setup with more planning, time, and application refactoring; best for mission-critical databases |
Strategies can differ even for databases on the same instance. A mix often works best, such as scheduled maintenance for small databases and continuous replication for mission-critical databases.
How online migration keeps production running is by keeping the source and target synchronized until cut-over. Data is duplicated in both the source and destination through the Y writing and reading approach, which is a form of parallel migration. A data-access microservice variation lowers refactoring effort because applications do not have to be changed as heavily to write and read in both places.
This approach depends on a ready Google Cloud foundation. The team must first build a resource hierarchy, configure Identity and Access Management, set up billing, set up network connectivity, harden security, and set up logging, monitoring, and alerting. When Database Migration Service (DMS) (opens in a new tab) is used, the migration depends on correct networking for Cloud SQL for PostgreSQL, so connectivity between the source network and the Google Cloud network must exist before replication starts.
Exam tip: Choose continuous replication when you cannot afford any downtime or cannot afford the cut-over window of backup, transfer, restore, and switch; choose scheduled maintenance only when that window is affordable.
Cut-over and fallback protection covers the point when production traffic moves to the Cloud SQL or AlloyDB for PostgreSQL instance. After cut-over, applications using the target must be monitored for a defined period to decide whether the migration holds or fallback is needed. The target might need a restart if database flags change, and rolling back can take more effort than fixing the target, so both paths must be weighed.
Fallback protection depends on keeping the old source usable after cut-over. In a fallback, writes on the target are replicated back to the original source, which reverses direction so the initial source becomes the new target. If rollback is needed, traffic can return to the old source with minimal data loss because the source stayed current. Before cleanup, the team creates a final backup of each source, collects source parameters and statistics to compare with the target, configures a maintenance window for the primary instance, keeps at least 20% available storage, and creates a metrics-based alert for disk use. An administrative operation must never start before the previous operation completes.
Execute offline migration strategies for extended outages means stopping the source system during a planned maintenance window while a full bulk copy moves to Google Cloud. Data flows from the source database to export files, from export files to a Cloud Storage (opens in a new tab) bucket, and from the bucket to the target database. Choose this path when data volume is very large, network bandwidth cannot move data in time, and the business can accept downtime for bulk transfer.
When to choose offline transfer over online transfer depends on dataset size, available bandwidth, transfer time, cost, and management overhead. For example, 1 GB can transfer in eight seconds over a 1 Gbps network, while 100 TB can take about 12 days before overhead. Estimates must also include approval time, bandwidth efficiency, and whether large transfers would overload production traffic during peak hours. When the network cannot meet the deadline, offline shipment avoids long outages from slow or congested links.
| Source and condition | Transfer path | Suggested product |
|---|---|---|
| Another cloud to Google Cloud | Network copy between providers | Storage Transfer Service |
| Private data center with enough bandwidth for deadline | Online upload from on-premises files | gcloud storage command or Storage Transfer Service for on-premises data |
| Private data center without enough bandwidth for deadline | Offline shipment of hardware | Transfer Appliance |
The native export and Cloud Storage staging workflow uses the database's own export utilities with Cloud Storage as a staging area. The source database produces portable files, the files upload to a Cloud Storage bucket, and the target imports from that bucket. This workflow fits large volumes because the bulk copy happens once while the source is offline, without continuous change capture. Use Apache Avro, ORC, or Parquet where schema is embedded with data to reduce complexity. If CSV is used, the schema must be specified at load time or auto-detected.
The workflow runs in order:
When transferring over the network, create the bucket in the location closest to the source data center, and if possible in the same location as the target dataset, to minimize latency.
Bulk transfer tools for offline paths move export files into Cloud Storage for later import. The gcloud storage command is the standard tool for small- to medium-sized online uploads from a private data center or another cloud. It supports multi-threaded parallel uploads, resumes after errors, encrypts traffic in transit, and fits as-needed transfers, very large files, streaming output, and syncing directories with moderate file counts.
Storage Transfer Service (opens in a new tab) for on-premises data moves data from network file system (NFS) storage to Cloud Storage at large scale, up to petabytes and billions of files. Agents in the data center crawl the source directory recursively and create objects with matching names in the destination bucket. The service supports full or incremental copies, scheduled recurring transfers, robust error reporting, bandwidth limits to protect other workloads, and a managed graphical interface. Transfer Appliance is the offline hardware option: the team copies files locally, ships the appliance, and data loads to Cloud Storage for import.
Exam tip: Choose Transfer Appliance when bandwidth cannot meet the deadline; choose gcloud storage or Storage Transfer Service when bandwidth is sufficient to move the bulk files in time.
Database-specific offline export, backup, and import follows the same pattern: back up or export at the source, hold the copy in Cloud Storage or as a backup, and restore or import in the new region or service. Cloud Bigtable supports backup and restore to another instance in a new region, and it also supports exporting tables to Cloud Storage for import into the destination region. Before using replication or restore for Bigtable, confirm the service is available in the destination region.
Firestore supports a managed export and import service that moves entities through a Cloud Storage bucket to a different location. Firestore also supports backup and restore, which is less expensive and more straightforward than export and import. Location planning must account for App Engine, because an existing App Engine app in us-central or europe-west can make the Firestore database multi-regional. Filestore follows the same offline model, where a backup of a file share can be restored to another region.
Downtime planning, validation, and safe decommissioning makes an extended-outage cutover controlled and observable. Planned downtime lets the traffic switch be monitored and avoids unpredictable client errors. Validation before cutover includes unit and integration tests on the migrated application, plus treating the new environment as a new software version with A/B testing or mirrored traffic.
Fallback depends on monitoring and retained copies:
Services in the source region shut down only after extended tests pass and the new region runs live without error. Keep source backups for a limited time until the new environment is stable, and consider promoting the old region to a disaster recovery site.
Design fallback procedures and post-migration validation keeps a safe path back to the source while proving the target is correct and performant. In migrations to Cloud SQL (opens in a new tab) and AlloyDB for PostgreSQL (opens in a new tab), the target receives production reads and writes after cutover while the source is preserved and kept current until the new environment is stable. Validation then compares data, configuration, and performance between source and target before the source is removed.
Cutover monitoring and the fallback decision is the defined observation period after applications switch to the target database. Applications using the target are monitored for errors, latency, and functional issues for a length of time defined in advance. During this window, the team weighs whether rolling back would take more effort than fixing issues directly in the target. A Cloud SQL or AlloyDB instance might also need a restart if database flags change, which can cause extra downtime that must be factored into the decision.
Maintain a reversion path with reverse replication means replicating writes from the new target back to the original source after cutover. The setup resembles forward migration but runs in reverse, so the initial source becomes the new target of replication and stays up to date. If rollback is needed, traffic can fall back to the old source with minimal data loss because those instances kept receiving changes. As a best practice, this reverse flow stays running until monitoring confirms fallback is no longer needed.
Before relying on fallback, the source end state is protected with final backups and configuration records. The team creates a final backup of each source database to preserve the end state for recovery and compliance. Database parameter settings and statistics are collected from the source and compared to the target, because different statistics make fair performance comparison hard.
Exam tip: Keep reverse replication from the target Cloud SQL instance back to the source running after cutover, so a rollback returns to a current source with minimal data loss instead of a stale backup.
Validate data and performance after migration is the set of integrity and performance checks that confirm the target matches the source. Validation of a restored database includes monitoring pg_restore output for errors, reviewing PostgreSQL logs for warnings during restore, and running basic queries and table counts. For broader migrations, the open source Data Validation Tool compares heterogeneous source and destination tables using multi-leveled validation functions built on the Ibis framework. Parameters gathered during inventory are reconciled so the target uses equivalent settings, and collected statistics are compared to explain performance differences.
Clean up the source and stabilize the target happens only after cutover completes and validation shows the target is healthy. Source databases are deleted only after final backups, parameter collections, and statistics comparisons finish, preserving compliance evidence and a last recovery point. On the target side, a maintenance window is configured for the primary instance to control when disruptive updates occur. Storage is kept with at least 20% space available for critical maintenance that Cloud SQL might perform (opens in a new tab), with a metrics-based alert on disk utilization to warn if space falls below 20%. Administrative operations run one at a time, so a new operation never starts before the previous one completes.
Last updated