Choosing the right tool depends on the source of the data and the total volume to be moved. The BigQuery Data Transfer Service is the primary tool for analytical migrations, specifically for moving data from third-party warehouses like Amazon Redshift or Teradata. For general file moves, the Storage Transfer Service is used to bring data from other clouds or local systems into Cloud Storage. These tools automate the process and help maintain data integrity during the transition.
A decision tree that guides the selection of Google Cloud database migration tools based on workload type, data volume, and network bandwidth.
When moving massive amounts of data, physical hardware or specialized agents may be necessary to overcome network limits. The Transfer Appliance is a high-capacity hardware device used to migrate up to a petabyte of data when internet speeds are too slow. For online moves from on-premises systems, Storage Transfer Service agents can be installed to manage large-scale transfers efficiently. These options ensure that business operations continue without interruption while data moves to the cloud.
For migrations involving different database engines, specialized Extract, Transform, and Load (ETL) tools are often required. These services help manage the transition by allowing for complex data cleaning and reorganization. Key tools include Dataflow, a serverless service for processing and enriching data streams; Cloud Data Fusion, a graphical tool for building managed data pipelines; and Dataproc, a service for running Apache Spark and Hadoop clusters to process large datasets. Using these tools ensures that the target schema is optimized for Google Cloud’s specific architecture.
Migrating to globally distributed systems like Cloud Spanner requires specific assessment and conversion steps. The Spanner Migration Tool (SMT) and HarbourBridge are used to evaluate existing schemas and perform data migrations from MySQL, PostgreSQL, or Oracle. These tools help map data types correctly to ensure the new database performs with high efficiency. For NoSQL workloads, HBase tools assist in moving data directly to Bigtable while preserving the original data structure.
Ensuring data accuracy after a migration is a critical final step for any project. The Data Validation Tool (DVT) is an open-source CLI tool that compares tables between the source and target environments. It uses multi-leveled validation functions to check for consistency in counts, sums, and averages across different database types. This process helps engineers confirm that no data was lost or corrupted during the heterogeneous migration process.
The Database Migration Service (DMS) is the primary serverless tool used to move database workloads from external environments into Google Cloud. It simplifies the process by automating the setup and management of migration jobs for MySQL, PostgreSQL, and SQL Server. Using DMS helps ensure data integrity and reduces the manual effort required compared to traditional migration methods. When choosing a migration path, you can select between creating a new Cloud SQL or AlloyDB instance or using an existing one. DMS supports continuous migration, which keeps the destination database synchronized with the source in near real-time using replication. This approach is ideal for maintaining operational continuity and minimizing downtime during the final cut-over to the cloud.
Manual export and import processes involve creating files like SQL dumps or BAK files and moving them to Cloud Storage. While these methods offer significant control, they often require more downtime and manual intervention than serverless options. Serverless exports are often preferred because they offload the work to a temporary instance, which prevents performance issues on the primary database during the move.
For complex scenarios, third-party tools provide Change Data Capture (CDC) capabilities to stream updates. Striim offers high-performance, low-latency data movement for heavy workloads. Debezium is an open-source platform specialized for capturing changes from transaction logs. Fivetran provides automated, no-code pipelines that handle schema changes automatically. Selecting the right tool depends on your specific needs for scalability and cost.
A successful migration requires a Technical Design Document (TDD) to outline Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). It is also important to address metadata limitations, such as the DEFINER clause in MySQL, which can cause import failures if users do not exist on the target. Testing the migration with a dry run is a critical best practice to ensure the application behaves correctly after the move.
Online migration uses Change Data Capture (CDC) to replicate data in real-time. This process starts with an initial data dump, followed by the continuous processing of any new changes made to the source. A key metric to monitor is the replication delay, which measures the time difference between a write on the source and its appearance on the destination. This method is ideal for meeting strict Recovery Time Objectives (RTO) to minimize service gaps.
Offline migration involves a one-time migration where a point-in-time snapshot is moved to Google Cloud. During this process, administrators must stop all writes to the source database to ensure data stays consistent. This often leads to significant application downtime while the data is being transferred and loaded into the new system. Offline migrations are often simpler to perform but require a larger window for the cutover phase.
Various tools help with these transitions depending on the database engine and source location. Database Migration Service (DMS) automates migrations for engines like MySQL, PostgreSQL, and Oracle. Datastream is a serverless service used for Change Data Capture to sync data across different cloud environments. Key offline tools include Storage Transfer Service for moving large datasets from external storage providers and RIOT, a command-line utility for live migration of Redis-compatible sources. Choosing the correct tool is essential for keeping the business running during the transition.
The final step in an online migration is the promotion operation. This action disconnects the destination instance from the source and makes it the primary database for all application traffic. Before starting a promotion, engineers must ensure the replication delay is at or near zero to prevent data loss. This transition point is known as the cutover, where the application officially begins using the Google Cloud database.
Fascinated by the world of cloud databases? Explore the methods for structuring, scaling, and securing database solutions on Google Cloud as you gear up for the Professional Cloud Database Engineer exam!
Prepare and test your skills
Prepare and test your skills
BigQuery Data Transfer Service and Storage Transfer Service differ by source and destination: BigQuery Data Transfer Service is the primary tool for analytical migrations, moving data from third-party warehouses like Amazon Redshift or Teradata into BigQuery, while Storage Transfer Service brings data from other clouds or local systems into Cloud Storage. Both automate transfers and help maintain data integrity during the transition.
Use Transfer Appliance when migrating up to a petabyte of data and internet speeds are too slow, because it is a high-capacity hardware device. Use Storage Transfer Service agents for online moves from on-premises systems, since they manage large-scale transfers efficiently while business operations continue without interruption.
Database Migration Service (DMS) is the primary serverless tool for moving database workloads from external environments into Google Cloud, automating migration jobs for MySQL, PostgreSQL, and SQL Server. It supports continuous migration that keeps the destination synchronized with the source in near real-time using replication, and you can choose a new or existing Cloud SQL or AlloyDB instance. DMS is ideal for minimizing downtime during the final cutover.
Online migration uses Change Data Capture (CDC) to replicate data in real time, starting with an initial dump and then continuously processing new source changes, with replication delay monitored to meet strict Recovery Time Objectives. Offline migration is a one-time move of a point-in-time snapshot to Google Cloud, requiring all writes to the source to stop and a larger cutover window with application downtime.
StudyCerts
StudyCerts