Prepare and test your skills
Prepare and test your skills
Evaluate Data Migration Tools and Transfer Methods is the process of choosing how to move data into Google Cloud without changing it, such as moving files as objects into Cloud Storage (opens in a new tab). Storage Transfer Service (opens in a new tab) handles online transfers over the network, the gcloud storage command handles small online uploads, and Transfer Appliance handles offline transfers with shipped hardware. The choice depends on data volume, available bandwidth, time to complete, cost, and security needs.
Online versus offline transfer is the choice between sending data over a network and sending data on physical hardware. Online transfer flows from the source system over Cloud Interconnect or the public internet to a Cloud Storage bucket. Offline transfer flows from the source data center to a Transfer Appliance, and then after shipping, from the appliance to Cloud Storage.
A decision tree branching on source location, data size, and bandwidth to choose gcloud storage, Storage Transfer Service, or Transfer Appliance for Cloud Storage.
Transfer time grows with dataset size and shrinks with available bandwidth, plus management overhead for planning, approvals, and retries. For private data center transfers, bandwidth can be increased with a public internet connection and public API, Direct Peering with a public API, or Cloud Interconnect with a private API. For cloud-to-cloud transfers, Google provisions the connection between cloud vendor data centers and no customer network setup is required. Weigh added bandwidth cost against deadline value, because large transfers that share a connection with production traffic can degrade that traffic.
| Transfer scenario | Size or bandwidth condition | Recommendation |
|---|---|---|
| Another cloud provider to Cloud Storage | Any size | Storage Transfer Service |
| Cloud Storage bucket to Cloud Storage bucket | More than 1 TB | Storage Transfer Service |
| On-premises to Cloud Storage | Less than 1 TB | gcloud storage command |
| On-premises to Cloud Storage | More than 1 TB with enough bandwidth | Storage Transfer Service for on-premises data |
| On-premises to Cloud Storage | More than 1 TB without enough bandwidth | Transfer Appliance |
The gcloud storage command is the command-line tool for small to medium online transfers from a private data center or another cloud provider to Cloud Storage. Data flows from local files, program output, or a watched directory directly to Cloud Storage objects over a typical enterprise network. Although the command supports uploads up to the maximum Cloud Storage object size, large-object transfers are more likely to fail than short-running transfers.
The command fits interactive and lightweight work rather than petabyte-scale migration:
Storage Transfer Service (STS) (opens in a new tab) is the fully managed service that automates online transfers into Cloud Storage from other public clouds and between Cloud Storage buckets. Data flows from Amazon S3, Microsoft Azure Blob Storage, an HTTP URL list, or a source Cloud Storage bucket to a destination Cloud Storage bucket without custom scripts or retry logic. For HTTP URL lists, the customer supplies each file size in bytes with a Base64-encoded MD5 hash, and for S3-compatible storage (opens in a new tab) the transfer uses an access key ID and secret access key.
Bucket-to-bucket transfers work within the same Google Cloud project or between different projects to consolidate data, create backups, or change bucket location. Transfers of less than 1 TB between buckets can use gcloud, while transfers of more than 1 TB should use Storage Transfer Service for built-in security, reliability, and performance. Storage Transfer Service does not lock reads or writes during transfer, so downtime is reduced with a seed transfer of bulk data followed by a sync transfer that copies only changes, since transfers are incremental by default.
Transfer speed depends on whether the job is request-bound or bandwidth-bound. Billions of small files are bound by request rate, while large objects are bound by network bandwidth, with regional bandwidth fairly shared across projects. When sufficient bandwidth exists, Storage Transfer Service can complete around 1000 tasks per transfer job per second, and a slow job can be accelerated by splitting it into multiple jobs using include and exclude prefixes. When source and destination share the same location, storage class, and encryption key, the service creates a new metadata entry that points to the source data instead of copying bytes, so the copy completes very quickly.
Exam tip: Use the gcloud storage command for transfers under 1 TB and Storage Transfer Service for transfers over 1 TB from on-premises or between Cloud Storage buckets.
Storage Transfer Service for on-premises data moves Network File System (NFS) data to Cloud Storage at large scale, up to petabytes of data and billions of files, with full or incremental copies. The system depends on transfer agents, which are Docker containers installed on computers in the data center that belong to an agent pool. All agents in a pool share the same configuration and move data in parallel. After setup, a user starts a transfer in the Google Cloud console by providing a source directory, destination bucket, and time or schedule.
A transfer follows an ordered flow from data center to cloud:
The service fits managed, recurring, and controlled transfers:
More agents produce more speed because work parallelizes across all agents to use available bandwidth. Agents are fault tolerant, self-updating, and self-healing, so a transfer continues with remaining agents if one agent goes down. For long transfers that run for days or weeks, use Cloud Monitoring alerts on agent health and throughput, and load the full error record into BigQuery when console error samples need deeper review for causes such as files changing during copy or permissions errors.
Transfer Appliance (opens in a new tab) is Google-owned hardware that is shipped to the customer for large-scale offline data movement when network bandwidth is unavailable, limited, or too expensive to acquire in time. Data flows first from local systems to the appliance during data capture, then the customer ships the appliance back and Google copies the data into Cloud Storage. Latency into Cloud Storage is typically higher than online transfer, and the appliance is available only in certain countries.
Transfer Appliance fits constrained or urgent moves:
Cost and speed drive the decision. With 1 Gbps connectivity, 100 TB takes over 10 days online, which may be acceptable, while the same 100 TB over 100 Mbps takes over 100 days and favors offline transfer. With 100 Mbps, 300 TB takes about 9 months to upload, while Transfer Appliance can capture 300 TB in under 25 days plus about 10 business days before data is accessible in Cloud Storage, without consuming outbound network bandwidth. To acquire an appliance, request it in the Google Cloud console with the data size, receive one or more appliances, complete capture within the allowed days, and ship them back.
Implement Performance Tuning for High-Throughput Transfers is the practice of finding the limiting factor for a transfer and configuring jobs and storage to move data as fast as that limit allows. Storage Transfer Service (opens in a new tab) is the managed service for moving more than 1 TiB between Cloud Storage buckets, from other clouds, and from on-premises sources, using a highly parallelized architecture with automatic retries and load balancing. Tuning focuses on whether a job is bound by request rate or by network bandwidth, and on choosing storage that meets latency needs.
A transfer bottleneck is the resource that determines how long a job takes when everything else has spare capacity. Jobs with billions of small files are typically QPS-bound, meaning the rate of per-object requests limits speed, while jobs with large objects are typically bandwidth-bound, meaning available network throughput limits speed. Bandwidth limits are set at the region level and are fairly allocated across all projects, so a single job cannot exceed its fair share even if the source is fast.
| Workload pattern | Limiting factor | Tuning behavior supported by the documentation |
|---|---|---|
| Billions of small files | Requests per second (QPS) | Split one job into multiple small jobs to increase total tasks per second |
| Large objects | Network bandwidth between source and destination | Throughput scales only if sufficient bandwidth is available |
| Same location, storage class, and encryption key, or deletes | Metadata operations only | Completes very quickly without copying bytes; parallel jobs increase speed |
When location, storage class, and encryption key are the same, Storage Transfer Service does not create a new copy of the bytes and instead creates a new metadata entry that points to the source blob. Deletes are also metadata-only operations. Storage Transfer Service can complete around 1000 tasks per transfer job per second when sufficient bandwidth is available.
Splitting a transfer means dividing one large transfer job into multiple smaller jobs that run at the same time. Each job contributes its own task throughput, so the combined jobs can exceed the approximate 1000 tasks per second limit of a single job when bandwidth allows. Filtering with include and exclude prefixes is the supported way to divide objects so different jobs transfer different files in parallel.
This approach helps most when a job is QPS-bound because of many small files. If a job consists of a large number of small files, scheduling multiple concurrent jobs can scale performance as long as sufficient bandwidth is available. Storage Transfer Service transfers (opens in a new tab) are incremental by default, so follow-up runs move only data that changed since the prior run.
Exam tip: Use gcloud for transfers of less than 1 TB between Cloud Storage buckets, and use Storage Transfer Service for transfers of more than 1 TB.
Low-latency storage configuration places data and replication behavior close to the workload that needs fast access. Google Cloud Hyperdisk provides better performance, flexibility, and efficiency than Persistent Disk, and with Hyperdisk Balanced an administrator can provision IOPS and throughput separately and dynamically to tune a volume to a wide variety of workloads. Regional Persistent Disk (opens in a new tab) replicates writes synchronously between two zones in the same region, so a backup virtual machine (VM) can force-attach the disk in the secondary zone after a zonal outage.
Shared and object storage follow the same principle of matching replication to availability and latency needs. Data in Cloud Storage regional buckets is replicated synchronously across zones in the region, while data in dual-region or multi-region buckets is stored redundantly in at least two geographic locations with metadata written synchronously and data replicated asynchronously. For dual-region buckets, turbo replication ensures objects are replicated across the region pair with a recovery point objective (RPO) of 15 minutes. A Filestore regional instance replicates data synchronously across three zones within a region and can be mounted on multiple VMs for shared configuration files, tools, and logs.
Transfer time estimation calculates how long a move will take from dataset size, network capacity, and operational overhead. Ideally 1 GB transfers in eight seconds over a 1 Gbps network, which scales to about 12 days for 100 TB before overhead, and large transfers can overload a company network if run during peak work hours. Estimation (opens in a new tab) should include:
Frequency and monitoring keep recurring transfers aligned with recovery and freshness goals. The interval between successive transfer jobs must not be longer than the RPO objective, so an RPO of 1 day requires a transfer at least once a day, and performance should be measured first with a representative subset of real data. Storage Transfer Service provides Pub/Sub notifications for transfer events, and subscribing to them helps handle unexpected failures or changes in completion time. Storage Transfer Service does not lock reads or writes during a transfer and does not provide a service level agreement (SLA) for transfer performance or latency, so some performance fluctuation may occur.
Optimize Network Latency and Regional Placement is the practice of placing data close to compute resources and users and routing traffic over the fastest network path. In Google Cloud, latency is controlled by three choices: the Google Cloud region that hosts Compute Engine resources and Cloud Storage buckets, the Network Service Tiers tier that carries traffic between the internet and Google Cloud, and the deployment pattern that decides whether frontends and backends run in one region or many.
Total user-visible latency is the sum of four segments, and region selection can only improve some of them. An architect can optimize app latency and the distance to the Compute Engine region (opens in a new tab), but cannot control the user's last-mile connection or the distance to the closest Google edge point of presence (POP). Region choice matters most when the Google network segment is a large share of total latency, such as for wired broadband users, and matters less when the last mile dominates, such as for 3G cellular users.
The four latency segments are:
Because apps have different tolerances, asynchronous apps and mobile apps with a threshold of 100 milliseconds or more can often run in a single region, while real-time games where a few milliseconds change the experience should run in multiple regions close to users.
Network Service Tiers (opens in a new tab) controls the path that traffic takes between systems on the internet and Google Cloud workloads, including virtual machine (VM) instances, load balancers, and Cloud Storage buckets. Premium Tier is the default, performance-optimized tier that carries traffic on Google's low-latency global private fiber network with over 200 points of presence (POPs). Standard Tier is the cost-optimized tier that carries traffic over transit ISP networks over the public internet, with quality comparable to other cloud providers.
With Premium Tier, inbound traffic from an internet user enters Google's network at the POP closest to the user, is routed inside Google's network to the VM or closest Cloud Storage bucket, and outbound traffic exits at the POP closest to the destination. With Standard Tier, inbound traffic enters Google's network at a POP close to the region of the destination IP address, and outbound traffic normally exits from the same region as the sending VM or bucket, regardless of destination. This difference means Premium Tier reduces hops between users and Google's network, while Standard Tier adds internet hops and potential latency between the client and the region.
| Tier | Routing behavior | Best use |
|---|---|---|
| Premium Tier | Inbound enters at POP closest to user; outbound exits at POP closest to destination over Google global network; supports global external IP addresses and multi-region backends | Global users, latency-sensitive storage access, global load balancing, and multi-regional Cloud Storage buckets |
| Standard Tier | Inbound enters at POP close to resource region over ISPs; outbound exits near the sending region over ISPs; regional external IP addresses only | Single-region VMs or regional buckets where cost matters more than latency |
Exam tip: A multi-regional Cloud Storage bucket used as a load balancer backend requires Premium Tier; Standard Tier requires the bucket to be in the same region as the load balancer forwarding rule, or requests produce an error.
Regional placement decides how far storage and compute sit from users and from each other, which directly controls storage access latency. Placing Cloud Storage buckets and Compute Engine VMs in the same region keeps reads and writes on Google's internal network and avoids cross-region round-trips. Even with a single-region backend, global load balancing and Cloud CDN can reduce user latency by terminating connections and serving cacheable content at the edge POP closest to the user.
Common deployment patterns trade latency against complexity:
Choose a single region when operational simplicity outweighs small latency gains. Choose a distributed frontend with a central backend when frontend caching can absorb most requests. Choose fully distributed backends only when every request must be answered locally and the team can manage global replication.
Storage proximity alignment means placing Cloud Storage data where compute and users can reach it with the fewest slow segments. By default, access to Cloud Storage buckets is treated as Premium Tier, which routes users through the closest edge POP and then over Google's backbone to the bucket. To use Standard Tier with Cloud Storage, the bucket must be configured as a backend of an external Application Load Balancer, and the bucket region must match the forwarding rule region.
For low-latency storage operations, co-locate regional buckets with the Compute Engine VMs that read and write them, use Premium Tier so traffic stays on Google's fiber network, and add Cloud CDN and global load balancing (opens in a new tab) to serve repeated reads from the edge. For globally distributed reads, use multi-regional or dual-regional buckets with Premium Tier so users in the US, Europe, and Asia can reach nearby replicas. Standard Tier fits only when VMs and buckets can all live in a single region and the workload is not latency or performance sensitive.
Use the gcloud storage command for transfers under 1 TB, Storage Transfer Service (STS) for online transfers over 1 TB when sufficient bandwidth exists, and Transfer Appliance for large offline moves when bandwidth is unavailable, limited, or too expensive to acquire in time. For on-premises to Cloud Storage, the recommendation is the gcloud storage command for less than 1 TB, Storage Transfer Service (STS) for more than 1 TB with enough bandwidth, and Transfer Appliance for more than 1 TB without enough bandwidth.
Online transfer sends data over a network from the source system over Cloud Interconnect or the public internet to a Cloud Storage bucket, while offline transfer copies data to a Transfer Appliance in the data center that is shipped back so Google can copy the data into Cloud Storage. The choice depends on data volume, available bandwidth, time to complete, cost, and security needs.
Split the Storage Transfer Service (STS) job into multiple smaller concurrent jobs filtered with include and exclude prefixes so their combined throughput can exceed the approximate 1000 tasks per transfer job per second limit of a single job when sufficient bandwidth is available. This helps because jobs with billions of small files are typically bound by requests per second while jobs with large objects are typically bound by network bandwidth.
Premium Tier is the performance-optimized tier that carries traffic on Google's low-latency global private fiber network with inbound traffic entering at the point of presence closest to the user and outbound traffic exiting at the point of presence closest to the destination. Standard Tier is the cost-optimized tier that carries traffic over transit ISP networks with inbound traffic entering near the resource region and outbound traffic normally exiting near the sending region, and it requires a regional bucket to match the load balancer forwarding rule region.
An organization is deploying a media distribution application on Google Cloud serving users across North America, with processing workloads running in both the us-central1 and us-east4 regions. The architecture requires minimal read latency for compute workloads in both regions, continuous availability during a single region outage, and optimal network delivery performance for external users by ingesting traffic onto Google's high-performance backbone at the nearest point of presence.
Which storage and networking architecture should you implement?
Last updated