StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Designing data migrations
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
Before moving any data to Google Cloud, teams must understand who uses the data and what they need it for. Stakeholder and user-centric analysis means talking to business leaders, analysts, and other data users to figure out what the current system does well and where it falls short. Functional requirements describe what the system must do, such as generating specific reports or processing certain data types. Non-functional requirements describe how the system should perform, including speed, scalability, security, and cost. Mapping user personas helps teams understand different roles, daily tasks, pain points with existing systems, and how users extract value from data. These findings become measurable success criteria for the migration, such as reducing report generation time by a certain percentage or enabling new types of analytical queries that were previously impossible. This analysis directly informs whether to start with an offload approach (copying data and downstream processes first) or pursue a full migration (moving upstream pipelines immediately), and it guides decisions about evolving the data schema in BigQuery and selecting modern pipeline tools like Dataflow.
Teams must document what data they have, where it lives, and how it flows through the organization. This involves mapping the existing data warehouse schema, identifying all source systems, and tracing data flows between components. Understanding the current technological landscape helps determine migration complexity and ensures compatibility with GCP services.
There are two main migration paths. The offload approach moves the schema and data to BigQuery while keeping upstream pipelines feeding the old warehouse, then establishes incremental copies to synchronize data over time. The full migration approach moves both the data and the upstream pipelines to Google Cloud, which allows teams to retire legacy systems after validation. Many organizations start with offloading and then fully migrate use cases in later iterations to reduce risk.
A critical assessment involves examining data pipelines that move and transform data. These pipelines can run in batch mode on scheduled intervals or in streaming mode handling real-time events. Teams must identify dependencies between pipelines, including fan-in (multiple pipelines merging into one) and fan-out (one pipeline triggering multiple others). Understanding these dependencies helps spot potential bottlenecks and cascading failure points. Some dependencies in legacy systems exist only because of resource limitations rather than true data relationships, which can cause widespread failures when issues occur.
Teams should also assess their current ETL (Extract, Transform, Load) processes, which often run outside the data warehouse to preserve query resources for analysis. Many organizations use ELT (Extract, Load, Transform), where data loads into the warehouse first and then transforms using SQL. Additionally, CDC (Change Data Capture) patterns track data changes over time. Each approach has different implications for migration, since cloud-native services like BigQuery offer serverless, massively parallel processing that handles both ELT and concurrent querying efficiently.
The target state plan defines how the organization reaches its desired future state after migration. Organizations choose between offloading and fully migrating their data workloads, with incremental synchronization keeping environments aligned during the transition. The full migration approach allows teams to retire legacy tables once validated, while the offload approach maintains the existing infrastructure as a backup.
Migrating data pipelines requires choosing transformation patterns. ETL transforms data using external compute engines before loading, while ELT loads data into the warehouse first and then transforms it using SQL, leveraging BigQuery's parallel processing power. Implementing CDC helps feed clean, historical versions of data directly into target systems. Selecting the right pattern optimizes performance and reduces operational overhead.
Complex migration roadmaps must address pipeline dependencies by analyzing the orchestration DAG (directed acyclic graph representing task workflows). Legacy resource limitations often cause cascading failures where one failing pipeline stops all dependent tasks. To mitigate this risk, teams should optimize dependencies using a phased approach: migrate the legacy orchestration structure as-is to minimize initial transition risks, analyze and parallelize tasks to maximize cloud compute scalability, and extract common tasks into their own DAGs to simplify maintenance.
Once workloads reside in BigQuery, organizations can evolve table schemas without disrupting business users. Deploying facade views (a design pattern that masks underlying structures) protects downstream analytics tools from structural changes. To deploy cloud resources securely and repeatedly, teams should adopt Infrastructure as Code using tools like Terraform, which prevents configuration drift and automates compliance guardrails across environments.
Moving data to Google Cloud requires ensuring data remains consistent, accurate, and available throughout the migration process. Organizations must implement robust validation frameworks and choose appropriate replication technologies that align with business requirements.
Change Data Capture (CDC) is a software pattern that tracks and captures data changes from source systems in real-time. Datastream enables CDC implementation, allowing organizations to continuously replicate data from on-premises or cloud-based databases to BigQuery, Cloud Storage, and Cloud SQL. CDC captures incremental changes rather than requiring full dataset transfers, which provides minimal downtime, data consistency, and real-time synchronization throughout the migration lifecycle.
A comprehensive validation framework must include monitoring and reconciliation processes that verify data integrity between source and target systems. This involves automated checks comparing record counts, validating schema consistency, and verifying data values at both ends of the replication pipeline. Organizations should implement reconciliation queries that identify discrepancies such as missing records, duplicate entries, or data corruption that may occur during transfer.
When planning migration, organizations choose between primary approaches. The offload approach migrates the use case schema and data first, then establishes incremental copies from the old to the new data warehouse to keep data synchronized while maintaining existing infrastructure. The full migration approach migrates both the use case and its upstream data pipelines to Google Cloud, allowing organizations to deprecate legacy systems after validation is complete.
Organizations should consider various data pipeline patterns when implementing continuous replication. EL (Extract and Load) extracts and loads data without transformation, suitable when automated services handle ingestion. ELT (Extract, Load, and Transform) loads data into the warehouse first, then applies transformations using SQL, leveraging cloud-native scalability. ETL (Extract, Transform, Load) transforms data outside the warehouse, useful when preprocessing is required before loading. The choice depends on data volume, transformation requirements, and the need for real-time versus batch processing.
Network architecture directly impacts the security, cost, and performance of moving data to Google Cloud. A well-designed network establishes secure, high-throughput pathways while minimizing latency and egress costs, accounting for the data's source, volume, and business requirements for uptime.
To build secure pathways, teams configure several core Google Cloud networking services. A Virtual Private Cloud (VPC) provides the isolated network foundation. While VPCs are global, their subnets are regional, which is critical when migrating workloads between regions because subnet IP addresses cannot be reused across regions. For connectivity to on-premises or other cloud environments, teams choose between Cloud VPN for encrypted tunnels over the public internet or Cloud Interconnect for dedicated, high-bandwidth, low-latency connections. Cloud Interconnect VLAN attachments are region-specific and must be reconfigured if the migration involves changing the target Google Cloud region.
The network design must facilitate migration tools and ensure data integrity. Using Cloud Interconnect over VPN is preferred for large-volume migrations to reduce public internet egress fees and improve throughput. Network routes and DNS configurations must be updated to reflect new regional endpoints post-migration to avoid unnecessary cross-region traffic. All connectivity methods must be configured with correct firewall rules within VPCs, IAM policies to control access, and potentially VPC Service Controls to create a security perimeter and prevent data exfiltration. The network must accommodate the chosen migration service, whether it's Database Migration Service, Storage Transfer Service, or Transfer Appliance.
Before executing the migration, the network architecture requires thorough preparation and validation. This involves re-evaluating any existing Cloud Interconnect or VPN connections and potentially redirecting them to the new target region to maintain connectivity during and after migration. Teams should validate network latency and bandwidth between source and target environments through performance benchmarks to set realistic expectations for migration timelines and identify potential bottlenecks. The network design should support contingency plans, including maintaining or quickly re-establishing connectivity to the source environment if a rollback is necessary.
Choosing the right migration service depends heavily on data volume, network bandwidth, and transfer windows. Google Cloud's data migration services are optimized for specific source types and transfer methods.
BigQuery Data Transfer Service (DTS) automates scheduled data ingestion from SaaS applications, external cloud storage, or legacy warehouses like Teradata and Redshift. Storage Transfer Service (STS) automates online data transfers from other cloud providers or on-premises filesystems to Cloud Storage. Transfer Appliance provides secure, physical hardware for offline data transfers when network bandwidth is limited and datasets exceed 20 TB. Selecting the correct pathway ensures transfer timelines align with business requirements without disrupting active production environments.
For online transfers from other cloud providers or on-premises file systems, organizations use the Storage Transfer Service or the Google Cloud CLI. Using gcloud storage is recommended for transfers under 1 TB, while STS handles large-scale, multi-threaded parallel transfers scaling up to petabytes. These online options leverage the existing network connection but can be controlled by a bandwidth cap to prevent impact on live workloads. The target Cloud Storage bucket should be colocated with the destination BigQuery dataset region to ensure low latency and minimize network costs.
Migrating structured data warehouses requires special configurations for schema compatibility and high performance. Migrating from Teradata utilizes BigQuery DTS combined with an on-premises migration agent, which can perform partition-by-partition extraction using the Teradata Parallel Transporter. For Amazon Redshift migrations, BigQuery DTS initiates an unload operation to an intermediate Amazon S3 bucket before loading data into BigQuery. These managed workflows support automatic schema mapping to native BigQuery types, though users can provide custom schema files to configure target clustering and partitioning.
Once data is transferred, validating the migration's success is critical. Engineers can employ the Data Validation Tool (DVT), an open-source Python CLI tool, to automate comparison of source and target tables using multi-level validation functions. To optimize query performance in BigQuery, teams should evaluate denormalization strategies such as utilizing nested and repeated fields rather than traditional star or snowflake schemas. Validating data integrity and optimizing table structures ensures migrated analytical workloads perform efficiently on the cloud architecture.
Stakeholder and user-centric analysis forms the foundation of any successful data migration to Google Cloud Platform. This analysis involves examining current needs, workflows, processes, and tech…
Stakeholder and user-centric analysis forms the foundation of any successful data migration to Google Cloud Platform. This analysis involves examining current needs, workflows, processes, and tech…
Continuous data validation and replication ensure that data remains consistent, accurate, and available throughout the migration process. When moving data to Google Cloud, organizations must imple…
Continuous data validation and replication ensure that data remains consistent, accurate, and available throughout the migration process. When moving data to Google Cloud, organizations must imple…