Prepare and test your skills
Prepare and test your skills
Large-scale data transfer and management is the process of moving files as they are into Cloud Storage (opens in a new tab) and then loading them into BigQuery (opens in a new tab) for analysis. Data flows from the source system to staging storage, then from staging storage to a Cloud Storage bucket, then from the bucket to BigQuery tables. Cloud Storage acts as the required staging point for almost all migrations, while BigQuery is the destination where schema and data become queryable tables.
Online versus offline transfer is the choice between moving data over a network and moving it on physical hardware shipped to Google. Online transfer uses the public internet, Direct Peering, Cloud Interconnect, or a Google-provisioned connection between clouds. Offline transfer uses Transfer Appliance hardware that is filled on site and shipped to a Google upload facility. The decision depends on dataset size, available bandwidth, project deadline, and the cost of buying more bandwidth versus accepting longer latency. For example, 100 TB takes over 10 days at 1 Gbps and over 100 days at 100 Mbps, so limited bandwidth can make offline transfer faster and cheaper despite shipping delay.
The staging flow from source to Cloud Storage to BigQuery is the ordered pipeline that preserves schema and file integrity for warehouse migrations. The same order applies to most sources:
A three-step process flow showing data moving from a source platform to staging storage, then uploaded to a Cloud Storage bucket, then loaded into BigQuery tables, with tool and format notes under each step.
Use Avro, ORC, or Parquet to reduce transfer complexity because schema information is embedded with the data. If CSV or another delimited format is used, the schema must be supplied separately at load time or auto-detected from file contents. When creating the Cloud Storage bucket for a network transfer, choose the bucket location closest to the data center to minimize latency and, if possible, the same location as the BigQuery dataset.
Google transfer options divide by source location and whether available bandwidth can meet the deadline. Storage Transfer Service handles cloud-to-cloud and large on-premises transfers, the gcloud storage command handles smaller on-premises transfers, and Transfer Appliance handles offline transfers:
| Source and condition | Online or offline | Choose |
|---|---|---|
| Another cloud to Google Cloud | Online | Storage Transfer Service |
| Cloud Storage bucket to another Cloud Storage bucket | Online | Storage Transfer Service |
| Private data center with enough bandwidth | Online | gcloud storage command for small to medium, or Storage Transfer Service for on-premises data for large scale |
| Private data center without enough bandwidth | Offline | Transfer Appliance |
Ingress to Cloud Storage is free, but transfers from another public cloud incur egress and possible read-operation charges from that provider. Private data centers may incur added costs for more bandwidth, plus Cloud Storage storage and operation costs, appliance costs, and personnel costs. After files arrive in Cloud Storage, processing such as Dataflow can run on the new files as they arrive.
The gcloud storage command is the standard command-line tool for small- to medium-sized online transfers to Cloud Storage. Data flows from the source file system over the network directly to the destination Cloud Storage bucket, with multi-threaded parallel uploads, resume after errors, and encryption in transit. The command supports objects up to the maximum Cloud Storage object size, but large-object transfers are more likely to fail than short-running transfers.
The gcloud storage command fits interactive and program-driven work:
If bandwidth is limited, reduce data size with compression or increase the connection to Google Cloud before relying on repeated retries. If the gcloud CLI cannot be used, a partner third-party tool is an alternative.
Storage Transfer Service is a fully managed service that automates large-scale online transfers into Cloud Storage from other public clouds, HTTP URLs, and on-premises Network File System (NFS) storage. Data flows from the NFS source directory through on-premises agents over the network to the destination bucket, preserving names so /dir/foo/file.txt becomes an object with the same name. The on-premises version scales to petabytes of data and billions of files and supports full copies and incremental copies with a managed graphical interface.
Storage Transfer Service for on-premises data fits managed, repeated, or high-volume work:
Setup depends on agents installed on computers in the data center, and all agents must see the same NFS mounts with the same relative paths. Transfers parallelize automatically across agents, so more agents use more bandwidth, and agents are fault-tolerant, self-updating, and self-healing so a transfer continues if one agent goes down. The service retries transient errors automatically, shows file counts, speed, and error samples during the run, and writes a tab-delimited file (TSV) with a full record of touched files and errors.
Transfer Appliance is Google-owned high-capacity hardware that moves data offline when network bandwidth cannot meet the deadline or is too costly to acquire. Data flows from the data center network to the appliance during local data capture, then the appliance is shipped to a Google upload facility where data is uploaded to Cloud Storage. You request the appliance in the Google Cloud console, Google ships one or more units, and you have a set number of days to capture data and ship it back.
Transfer Appliance fits constrained or urgent network conditions:
Transfer Appliance typically has higher latency into Google Cloud than online transfer and is available only in certain countries. Choose it when the online time math fails the schedule, such as 100 TB over a slow link that would take months, rather than for every large dataset.
BigQuery Data Transfer Service (BigQuery DTS) (opens in a new tab) is a fully managed service that loads schema and data directly into BigQuery from supported warehouses and from Cloud Storage on a one-time or scheduled basis. For Amazon Redshift and Teradata warehouses, BigQuery DTS copies tables directly while still using Cloud Storage for staging, which avoids a separate manual upload step. For any warehouse, files can be extracted to Cloud Storage first and then loaded with BigQuery DTS, the BigQuery Storage Write API, or a batch load.
BigQuery DTS therefore sits after Cloud Storage in the pipeline when recurring warehouse freshness matters. If analysts can work with data arriving at intervals, create scheduled transfers that repeat the initial pattern. If transformation is needed during load, route the Cloud Storage data through Cloud Data Fusion, Dataflow, Managed Service for Apache Spark, or a partner tool before or while writing to BigQuery. For initial loads of a few terabytes, dumping files and using gcloud storage is often more efficient than row-by-row JDBC extraction, while larger or cloud-hosted sources favor BigQuery DTS or Storage Transfer Service.
Exam tip: Use Storage Transfer Service to move objects into Cloud Storage, use BigQuery Data Transfer Service to load into BigQuery tables, and use Transfer Appliance only when bandwidth cannot meet the deadline.
Deployment orchestration and management tooling is the combination of Infrastructure as Code (IaC) and continuous integration and continuous delivery (CI/CD) services that provisions migrated infrastructure and delivers applications consistently. In Google Cloud, Terraform (opens in a new tab) provisions and configures the foundation and landing zone, while Cloud Build compiles source into artifacts, Artifact Registry stores those artifacts, and Cloud Deploy promotes them through target environments. Together these tools move teams from manual, environment-specific steps to automated deployments that scale across the cloud lifecycle.
Provisioning migrated foundations with Infrastructure as Code means defining resource hierarchy, networking, identity, and security in versioned configuration files instead of manual console steps. Teams that already use Terraform with configuration management tools in the source environment can reuse that practice on Google Cloud to build the landing zone that supports workloads and connects the source and Google Cloud environments. The choice of starting point depends on how much opinionated structure the team wants, from raw building blocks to a full enterprise foundation.
| Option | Start when | Builds on |
|---|---|---|
| Resources in the Google Cloud Terraform Provider | Starting from scratch with base building blocks | Google Cloud APIs directly |
| Terraform blueprints and modules for Google Cloud | Starting from a baseline of opinionated Terraform modules | Resources in the Google Cloud Terraform Provider |
| Enterprise Foundations Blueprint | Starting from an opinionated foundation and landing zone | Terraform blueprints and modules for Google Cloud |
| Cloud Foundation Fabric | Starting from an opinionated foundation and landing zone with standalone modules | Resources in the Google Cloud Terraform Provider with its own Terraform modules |
Choosing a higher-level blueprint reduces design work but accepts more opinionated defaults, while starting from provider resources gives maximum control with more assembly effort. Once the IaC approach is selected, the plan and build phase provisions the foundation in order:
Automating builds and application delivery moves deployable artifacts from source code to running environments without manual handoffs. Cloud Build executes builds on Google Cloud by importing source code from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket, running the build to specification, and producing artifacts such as Docker containers or Java archives. Those artifacts flow from Cloud Build into Artifact Registry, which provides a single location for managing packages and container images and integrates with CI/CD tools and Google Cloud runtimes. Cloud Deploy then automates delivery of the stored applications to a series of target environments in a defined promotion sequence, with runtimes such as Google Kubernetes Engine (GKE) (opens in a new tab) and Kubernetes, Cloud Run, App Engine flexible environment, and Cloud Run functions as deployment destinations.
This chain creates clear dependencies: the build depends on the source repository, the deployment depends on the artifact in the registry, and the runtime depends on the promoted release. Infrastructure Manager fits alongside this chain for infrastructure changes, since teams can refactor provisioning processes to use Cloud Build, Cloud Deploy, and Infrastructure Manager together. Assessing current processes for artifact generation, storage, scanning and signing, runtime configuration injection, logging, authentication, and resource provisioning shows which steps target only the source environment and must be retargeted to Google Cloud.
Refactoring deployment and operational processes for the target environment means changing where resources are provisioned and where workloads are built and deployed. The refactored processes provision and configure resources in the Google Cloud environment instead of the source environment, and they build, configure, and deploy workloads in Google Cloud instead of deploying them in the source environment. Teams can rebuild similar processes natively in Google Cloud, retarget processes that run in a third-party environment outside the source, or combine both approaches depending on current design and desired end state.
Because combined workload and process changes increase complexity and risk, substantial refactoring effort is handled as a separate, dedicated project rather than inside the workload migration. A general artifact-first refactoring sequence preserves rollback options while shifting delivery:
Keeping images available in both the source and Google Cloud repositories during migration enables rollback if unanticipated issues occur, and only after validation do builds store artifacts in Google Cloud alone. When Artifact Registry is the system of record, the service depends on protective controls for operational integrity, including access control, data exfiltration prevention, vulnerability scanning, and Binary Authorization to ensure only trusted code runs.
Exam tip: Store artifacts in both the source registry and Artifact Registry during migration for rollback, then switch builds to Artifact Registry only after workloads are validated on Google Cloud.
Infrastructure and workload migration strategies are the combined choices of how much downtime a workload can tolerate, which Google Cloud tool will move it, and in what order groups of workloads will cut over. The goal is to move on-premises servers, virtual machines (VMs), and applications to Google Cloud with the right balance of speed, cost, and risk. Assessment builds knowledge of the current environment, planning organizes work into batches and waves, and execution repeats a cycle of deploy, test, and optimize.
Choosing a migration strategy means matching each workload to how much downtime and data loss it can tolerate. Mission-critical workloads that cannot tolerate downtime or data loss require zero or near-zero downtime strategies that keep the source running while data is copied and synchronized in the background. Secondary or backend workloads that can tolerate an outage can use simpler strategies that require a downtime window for the final copy and cutover. Zero and near-zero downtime approaches reduce business disruption but are usually more costly and complex to design and implement.
Selecting migration tools means choosing managed migration services that fit the migration strategy, source and target environments, and characteristics of the data. Managed migration services encapsulate the complex logic of copying, synchronizing, and monitoring a move, which makes cutover more seamless than custom scripts alone. The choice depends on several peer factors:
Once the strategy is set, the team reviews these factors and picks the tool optimized for that combination of source, destination, and change rate.
Lift-and-shift and replatform tools for compute workloads divide by whether the workload stays as a VM or is converted into containers. Migrate to Virtual Machines (opens in a new tab) is the lift-and-shift tool that moves VMs from on-premises VMware environments and VMs from other cloud environments to Compute Engine (opens in a new tab). VMware HCX is the lift-and-shift tool that moves VMware VMs to Google Cloud VMware Engine, where it abstracts vSphere-based on-premises and cloud resources and presents them as one continuous hybrid infrastructure. Migrate to Containers is the replatform tool that converts VM-based workloads from on-premises physical servers, VMs, VMware on-premises, AWS, Azure, or Compute Engine into containers running in Google Kubernetes Engine (GKE) (opens in a new tab), Autopilot, GKE Enterprise, or Cloud Run.
| Tool | Strategy | Source to destination |
|---|---|---|
| Migrate to Virtual Machines | Lift and shift | On-premises VMware VMs and other-cloud VMs to Compute Engine |
| VMware HCX | Lift and shift | On-premises VMware VMs and alternate-cloud VMware VMs to Google Cloud VMware Engine |
| Migrate to Containers | Replatform | Physical servers and VMs to GKE, Autopilot, GKE Enterprise, and Cloud Run |
Choose Migrate to Virtual Machines when the destination is Compute Engine VMs with minimal change, choose VMware HCX when the destination must remain a VMware vSphere environment in Google Cloud, and choose Migrate to Containers when the goal is to run the existing workload as containers on GKE or Cloud Run.
The assessment phase is the discovery work that determines requirements, dependencies, and order for moving the source environment to Google Cloud. The team builds a comprehensive inventory of workloads, catalogs them by properties and dependencies, educates teams on Google Cloud, and runs experiments and proofs of concept to test performance and managed services. The team then calculates total cost of ownership (TCO) by comparing hardware, software, power, cooling, and maintenance against elastic Google Cloud costs, including often-overlooked network utilization charges and savings from rightsizing with Migrate for Compute Engine, Compute Engine sizing recommendations, and autoscaling groups. Assessment is continuous, so discovery keeps recalibrating application-to-infrastructure mapping, resource consumption, new landing zone needs, and blocking issues that could prevent migration.
The planning phase organizes the assessed workloads into groups and defines the order and safeguards for moving them. A cloud migration project is divided into waves, where a wave is a group of applications with common characteristics or interdependencies, and workloads within a wave are divided into move groups that must migrate together in sprints. A sprint plan groups workloads and data into batches, chooses the order of batches and the order of workloads inside each batch, and produces a technical design document, responsibility matrix, and T-minus timeline. Batching and ordering weigh business value, unique deployment patterns, responsible teams, number and scope of dependencies, refactoring effort, compliance and licensing, and availability and reliability requirements, with early batches chosen to build team knowledge on Google Cloud.
The execution phase is the repeatable cycle that migrates each sprint and validates that Google Cloud works as expected before the source is retired. Each iteration follows an ordered cycle:
Risk is reduced by validating each step, setting a maximum allowed time that triggers rollback, planning gradual rollouts with small changes first, alerting source and target operations teams, removing proof-of-concept resources from production, and defining retire criteria such as migrated backups, consistent and usable data, and working high availability and disaster recovery in the target.
Exam tip: Choose zero or near-zero downtime for mission-critical workloads, and choose a downtime-required strategy for tolerant workloads because zero-downtime designs cost more and add complexity.
Transfer Appliance should be used offline when available network bandwidth cannot meet the project deadline or is too costly to acquire, while online transfer moves data over the public internet, Direct Peering, Cloud Interconnect, or a Google-provisioned connection between clouds. For example, 100 TB takes over 10 days at 1 Gbps and over 100 days at 100 Mbps, so limited bandwidth can make offline transfer faster and cheaper despite shipping delay. Data flows from the data center network to the appliance during local data capture, then the appliance is shipped to a Google upload facility where data is uploaded to Cloud Storage.
Storage Transfer Service moves objects online into Cloud Storage, while BigQuery Data Transfer Service (BigQuery DTS) loads schema and data into BigQuery tables. Storage Transfer Service handles cloud-to-cloud transfers, Cloud Storage bucket to bucket copies, and large on-premises transfers from Network File System storage, whereas BigQuery Data Transfer Service copies tables directly from Amazon Redshift and Teradata warehouses while still using Cloud Storage for staging and also loads from Cloud Storage on a one-time or scheduled basis. Transfer Appliance is used offline only when bandwidth cannot meet the deadline.
Terraform provisions and configures the migrated foundation and landing zone in versioned configuration files, while Cloud Build compiles source into artifacts, Artifact Registry stores those artifacts, and Cloud Deploy promotes them through target environments. Cloud Build imports source from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket and produces artifacts such as Docker containers or Java archives, Artifact Registry provides a single location for managing packages and container images, and Cloud Deploy automates delivery to runtimes such as Google Kubernetes Engine (GKE) and Kubernetes, Cloud Run, App Engine flexible environment, and Cloud Run functions.
Migrate to Virtual Machines is the lift-and-shift tool that moves virtual machines from on-premises VMware environments and other clouds to Compute Engine, VMware HCX is the lift-and-shift tool that moves VMware virtual machines to Google Cloud VMware Engine, and Migrate to Containers is the replatform tool that converts physical servers and virtual machines to containers on Google Kubernetes Engine (GKE), Autopilot, GKE Enterprise, or Cloud Run. Choose Migrate to Virtual Machines when the destination is Compute Engine virtual machines with minimal change, choose VMware HCX when the destination must remain a VMware vSphere environment in Google Cloud, and choose Migrate to Containers when the goal is to run the existing workload as containers.
An enterprise is planning to migrate three distinct data workloads to Google Cloud to support an analytics initiative:
Which combination of Google Cloud services should you recommend for these workloads?
Last updated