Prepare and test your skills
Prepare and test your skills
Data Transition and Integrity Management is the practice of moving large datasets and databases to Google Cloud securely while keeping data correct and services running. Storage Transfer Service (opens in a new tab) moves online object data to Cloud Storage, Transfer Appliance (opens in a new tab) moves very large offline datasets by shipment, BigQuery Data Transfer Service (opens in a new tab) loads analytics data on a schedule, and database backup tools move relational data to Cloud SQL (opens in a new tab) or AlloyDB (opens in a new tab). Validation tools then compare source and destination before cutover so missing rows or errors are caught while fallback is still possible.
Storage Transfer Service is the managed service for moving object data to Cloud Storage from other clouds, online sources, or on-premises file systems. Data flows from the source system, such as Amazon S3, Azure Blob Storage, Data Lake, or on-premises files, to a destination Cloud Storage bucket without manual copy handling. The service manages scheduling, retry, and throughput for the transfer, which reduces operational work compared to custom scripts. Choose Storage Transfer Service when data is reachable over the network and the goal is durable object storage in Google Cloud.
Transfer Appliance is a physical hardware appliance for migrating very large datasets without disrupting business operations. Data is first copied from the source environment to the appliance on premises, then the appliance is shipped to Google, and Google uploads the data from the appliance to the destination Cloud Storage. The appliance path avoids long saturation of wide-area bandwidth for massive datasets. Choose Transfer Appliance when the dataset exceeds 20 TB and is up to 1 petabyte.
Exam tip: Choose Storage Transfer Service for online transfers from S3, Blob, or on-premises file systems, and choose Transfer Appliance when offline shipment is needed for data over 20 TB up to 1 petabyte.
BigQuery Data Transfer Service is the managed service that automates recurring data movement into BigQuery. Data flows from software-as-a-service (SaaS) applications, Teradata, or Amazon Redshift to BigQuery datasets according to the configured schedule. Because the transfer is managed and recurring, analytics tables stay current without building a custom pipeline for each source. Choose BigQuery Data Transfer Service when the goal is to refactor to BigQuery for ongoing analytics rather than a single file copy.
The three bulk-transfer options differ by source, destination, and trigger, so the choice depends on where data starts and where it must land:
A decision tree that branches on BigQuery analytics need and dataset size to pick BigQuery Data Transfer Service, Storage Transfer Service, or Transfer Appliance.
| Transfer option | Source to destination | Choose when |
|---|---|---|
| Storage Transfer Service | S3, Blob, Data Lake, or on-premises files to Cloud Storage | Data is reachable online and the target is object storage |
| Transfer Appliance | On-premises data via shipped hardware to Google Cloud | Data exceeds 20 TB up to 1 petabyte and network transfer would disrupt operations |
| BigQuery Data Transfer Service | SaaS apps, Teradata, or Redshift to BigQuery | Analytics data needs scheduled, managed loading into BigQuery |
Object consolidation points to Storage Transfer Service or Transfer Appliance, while recurring warehouse loading points to BigQuery Data Transfer Service.
Built-in database engine backups are the recommended method for one-time, scheduled-maintenance migrations to Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL. The flow moves from the source database through an export file in Cloud Storage to the target instance, where the file is restored. One-time migration jobs are not supported in Database Migration Service, so this dump-and-load path is used when significant downtime is acceptable and source databases are relatively static. Managed migration services around this flow provide secure and reliable transfer, a common monitored interface, proven migration models, and lower cost than custom migration servers.
PostgreSQL migrations use the pg_dump and pg_dumpall utilities, which have specific limits that affect planning:
pg_dump does not include users and roles, so use pg_dumpall to migrate those accounts.When using pg_dump, compress data to reduce cost and transfer duration and verify the result after restore. The --jobs option runs multiple dump jobs at once, and the -z option sets compression from 0 to 9 with a default of 6, where higher values make smaller files but use more client resources. After restore, monitor pg_restore output for errors, review PostgreSQL logs for warnings, and run basic queries and table counts to verify integrity. Flat-file CSV export and import is an alternative when the migration needs transformations, checks, or consolidation of multiple instances or databases into one.
Data Validation Tool is an open source Python command-line tool based on the Ibis framework that verifies correctness after a Data Warehouse, Database, or Data Lake migration. The tool compares tables in the heterogeneous source and the Google Cloud destination using multi-leveled validation functions, so row counts, schemas, and values can be checked across different engines. Validation runs after data lands in the target and before production cutover, which catches missing rows or transformation errors while fallback is still possible. Other specialized tools extend the same pattern to specific targets, including HarbourBridge for Spanner evaluation and migration, HBase tools for Bigtable, and a Hive-to-BigQuery framework that uses Cloud SQL to track migration progress.
Inventory discovery and TCO assessment is the process of automatically finding existing servers, databases, and data platforms, measuring how they are used, and estimating what they will cost on Google Cloud. Google Cloud Migration Center (opens in a new tab) is the central service for this work because it collects asset data, evaluates utilization, recommends target sizing, and generates a Total Cost of Ownership (TCO) report. A complete assessment prevents incorrect assumptions about scope, performance needs, and cost before a migration plan and timeline are defined.
Google Cloud Migration Center performs a complete assessment of the application and database landscape, including the technical fit for moving databases to Google Cloud. Data flows from the source environment into Migration Center through automated collectors and imports from other cloud providers, so on-premises servers, AWS assets, and databases appear in one inventory. Migration Center then analyzes that inventory and produces size and configuration recommendations for each source database plus a TCO report for servers and databases. Use automated discovery instead of manual spreadsheets because manual approaches are prone to error and can lead to incorrect scope and sizing.
A complete inventory defines the migration scope by recording every asset, configuration, and dependency that must move or be replaced. For Amazon RDS or Amazon Aurora instances, the inventory includes generic properties such as database engine version and edition, CPUs, and memory size, as well as topology, backup policies, parameter settings, and special customizations. For batch data platforms, the inventory covers three parts in order:
For each part of the data infrastructure, record storage platform type and migration method, computation platform deployment plan and custom configuration changes, network latency between source and target and between clients and each environment, software versions, and cluster size including number of nodes, CPUs, and memory per node.
Source and target systems often differ even when the engine name matches. Amazon RDS or Aurora and Cloud SQL (opens in a new tab) for PostgreSQL or AlloyDB (opens in a new tab) for PostgreSQL can differ in infrastructure, storage, authentication and security, replication, backup, high availability, resource capacity model, engine feature integrations and extensions, and default database parameter values. Capturing these differences during inventory prevents failures caused by assuming a feature, parameter, or operation works the same way on Google Cloud.
Resource evaluation turns inventory data and performance measurements into a correctly sized Google Cloud target. Benchmarking collects performance information about the workloads to be migrated and helps estimate the performance needs of the target environment and the right target architecture. For databases, pay special attention to disk size and throughput, IOPS, and number of vCPUs, because disk performance on Cloud SQL is based on disk size and the number of vCPUs. Incorrect target sizing can cause long migration times, database performance problems, database errors, and application performance problems.
Data platform design adds two placement rules that affect both performance and cost. Data flows from the storage layer to the computation layer for processing, and then results flow back to a target system that can be the source storage layer. To reduce communication time and latency, outbound data transfer cost, and I/O operations between layers, store data in the same zone where it is processed. Keep the storage layer separate from the computation layer on different platforms in the same zone and region, for example by moving storage from HDFS to Cloud Storage (opens in a new tab) while running computation on Managed Service for Apache Spark, so the computation layer does not have to run all the time.
TCO calculation estimates the future operating cost of the proposed Google Cloud environment from the discovered inventory and utilization data. In the assessment phase for batch pipelines, calculating TCO is step five after building inventories, cataloging pipelines, training teams, and building a proof of concept, and before choosing which workloads to migrate first. Migration Center creates the TCO report for servers and databases from its sizing recommendations, which links cost directly to the recommended machine shapes and database configurations. Specialized database tools add financial detail about cost savings after migration, migration costs, and the new operating budget.
Exam tip: Migration Center provides both target sizing recommendations and the TCO report for servers and databases; manual inventory alone does not provide either.
Complementary database assessment tools extend Migration Center when deeper database analysis is needed, especially for heterogeneous migrations. The three main options differ in depth, engine coverage, and outputs:
| Tool | Best use | Key outputs |
|---|---|---|
| Google Cloud Migration Center | Initial overview of servers, applications, and databases | Technical fit, size and configuration per source database, TCO report |
| migVisor | Heterogeneous migrations and proprietary feature analysis | Discovery analysis, migration complexity, financial report, phased migration plan |
| Database Migration Assessment Tool (DMA) | Open source, detailed object-level assessment | Features in use, database logic, and objects such as schemas, tables, views, functions, triggers, and stored procedures |
migVisor supports a variety of database engines, identifies artifacts and incompatible proprietary features that can cause migration problems, points to workarounds, and recommends a target Google Cloud solution with initial sizing and architecture. Its assessment output includes comprehensive discovery of current deployments, migration complexity based on proprietary features used, a financial report, and a phased plan to move databases and applications with minimal disruption. Collection temporarily increases database server utilization by typically less than 3%, so run it during non-peak hours. Database Migration Assessment Tool (DMA), built by Google engineers, uses downloadable collection scripts for each database engine to collect workload, feature, object, and code details and estimate migration effort.
Migration strategy selection and dependency mapping is the process of inventorying applications, databases, and pipelines, evaluating dependencies and business constraints, and grouping workloads into systematic migration waves. Google Cloud Migration Center (opens in a new tab) provides sizing recommendations and a TCO report, while specialized tools add compatibility and effort evidence for each wave. This evaluation determines what must move together, what the target must provide for similar performance, and whether each database can tolerate downtime.
A dependency-aware inventory defines the migration scope by recording what exists, how parts connect, and what each part requires in the target environment. The inventory covers three related objects: the data infrastructure with its storage layers and computation layers, the data pipelines scheduled for migration, and the datasets that those pipelines read and must migrate. Building the inventory with automated collection is preferred because manual approaches are prone to error and can lead to incorrect assumptions about dependencies.
The data platform separates into a data storage layer where data is saved and a data computation layer that loads data from storage, processes it, and saves results to a target system that can be the source storage layer. Data flows from the storage layer to the computation layer and then back to the target system, so a pipeline depends on both its source datasets and the specific storage and compute platforms underneath it. Keeping storage and computation on separate platforms in the same zone and region preserves this flow while improving flexibility, reducing the need to keep computation running, and lowering communication latency, outbound transfer cost, and I/O operations between layers.
Each infrastructure component adds constraints that must be mapped before waves are organized:
Incorrect target sizing causes long migration times, database performance problems, database errors, and application performance problems. Benchmarking current workloads helps estimate the performance needs of the Google Cloud target and defines the right architecture for each wave. Differences between Amazon RDS or Amazon Aurora and Cloud SQL, including infrastructure, storage, authentication and security, replication, backup, high availability, capacity model, engine features, extensions, and default parameter values, must also be recorded so the target instance matches the source for similar performance needs.
Assessment tools that support strategy decisions turn inventory data into sizing, fit, complexity, and cost evidence used to choose strategies and order waves. Google Cloud Migration Center performs an overview assessment of the application and database landscape, including technical fit for migration to Google Cloud, with size and configuration recommendations for each source database and a TCO report for servers and databases. Migration Center can also import data from other cloud providers to assess an AWS environment.
Specialized database tools add detail about compatibility and effort alongside Migration Center. migVisor supports a variety of database engines and is particularly suitable for heterogeneous migrations, identifying incompatible proprietary artifacts and features that can cause migration problems and pointing to workarounds while recommending a target Google Cloud solution with initial sizing and architecture. The migVisor assessment output provides:
The migVisor report includes engine version and edition, CPUs and memory size, topology, backup policies, parameter settings, and special customizations, and it typically adds less than 3% database server load and can run during non-peak hours. The open source Database Migration Assessment Tool (DMA), built by Google engineers, provides a complete assessment through collection scripts for each engine, covering features in use, database logic, and database objects including schemas, tables, views, functions, triggers, and stored procedures. Together these tools establish the dependency map, the effort estimate, and the target sizing that strategy selection depends on.
The migration strategy decision selects scheduled maintenance or continuous replication for each database based on affordable downtime, cut-over time, and refactoring tolerance. Scheduled maintenance, also called one-time migration or big bang migration, is ideal when downtime is affordable and offers lower cost and complexity with little refactoring of workloads and services. Continuous replication, also called online migration or trickle migration, is suited for mission-critical databases and offers near-zero downtime with lower risk of data loss by splitting effort into several chunks so rollback and retry take less time.
The two strategies trade simplicity against availability and planning effort:
| Strategy | Downtime and risk | Cost, complexity, and refactoring |
|---|---|---|
| Scheduled maintenance | Requires an affordable cut-over window to back up the database, transfer it to Cloud SQL, restore it, and switch applications; if migration fails before completion, the process restarts and downtime grows | Lower cost and complexity with little refactoring |
| Continuous replication | Provides near-zero downtime for mission-critical databases by replicating changes until cut-over, when replication stops and all reads and writes move to the target instance | More complex setup with more planning and time, plus added effort to refactor applications that connect to database instances |
Exam tip: Choose scheduled maintenance only when both some downtime and the full cut-over window are affordable; otherwise choose continuous replication.
Continuous replication offers two variations to manage application changes during parallel operation. The Y writing and reading approach duplicates data in both the source and destination instances during migration, while a data-access microservice reduces the refactoring effort required by the Y approach. Strategies can vary even for different databases on the same instance, so a common wave pattern migrates small and infrequently used databases with scheduled maintenance and mission-critical databases with continuous replication.
Organizing systematic migration waves groups dependent databases, applications, pipelines, and datasets so each wave can move, validate, and cut over in order. The assessment phase follows an ordered sequence that builds from discovery to wave selection:
This order matters because sizing, TCO, and proof-of-concept results depend on complete inventory and cataloging, and wave order depends on those results. A migration is considered complete only after the switch from the initial source instance to the target instance takes place, replication is stopped, and all reads and writes occur on the target. Defining preparation tasks, execution tasks, testing and validation, production cut-over, fallback scenarios, source cleanup, and post-migration tuning for each wave keeps dependent systems moving together rather than migrating in isolation.
Use Storage Transfer Service for online transfers when data is reachable over the network and the destination is a Cloud Storage bucket, and use Transfer Appliance for offline shipment when the dataset exceeds 20 TB and is up to 1 petabyte. Storage Transfer Service manages scheduling, retry, and throughput from sources such as Amazon S3, Azure Blob Storage, Data Lake, or on-premises files, while Transfer Appliance avoids long saturation of wide-area bandwidth by copying data to physical hardware on premises for shipment to Google for upload.
Google Cloud Migration Center collects asset data through automated collectors and imports from other cloud providers, then produces size and configuration recommendations for each source database and a Total Cost of Ownership (TCO) report for servers and databases. It also evaluates utilization and technical fit for moving databases to Google Cloud, which prevents incorrect assumptions about scope, performance needs, and cost before a migration plan is defined.
Choose scheduled maintenance, also called one-time or big bang migration, when downtime and the full cut-over window are affordable because it offers lower cost and complexity with little refactoring. Choose continuous replication, also called online or trickle migration, for mission-critical databases because it provides near-zero downtime with lower risk of data loss but requires more planning, time, and complexity.
Data Validation Tool is an open source Python command-line tool based on the Ibis framework that compares tables in the heterogeneous source and the Google Cloud destination using multi-leveled validation functions. It checks row counts, schemas, and values after data lands in the target and before production cutover, so missing rows or transformation errors are caught while fallback is still possible.
An enterprise is planning a live migration from an on-premises Microsoft SQL Server environment to Cloud SQL for PostgreSQL using Google Cloud Database Migration Service (DMS) with continuous Change Data Capture (CDC).
During pre-production migration testing, the team encounters two critical issues on the destination database:
Which strategy should the cloud architect recommend to ensure data integrity and prevent migration job failures?
Last updated