Prepare and test your skills
Prepare and test your skills
Large-scale data migration to Google Cloud is a choice between online transfer with Storage Transfer Service (opens in a new tab) and offline transfer with Transfer Appliance (opens in a new tab). Storage Transfer Service is a fully managed online service that moves data over the network into Cloud Storage, while Transfer Appliance is physical hardware that is loaded on-premises and shipped to Google to avoid heavy network use. The decision is driven by total data volume, available network bandwidth, and project timeline, plus cost, security, and whether the move can run without disrupting business operations.
Planning for large-scale migration defines what data can move, where the data must land, and what resources are available to move it. Data flows from sources such as on-premises storage, edge systems, or other clouds to the destination Cloud Storage bucket. The transfer team must confirm regulatory limits, required regions, destination bucket structure, and whether transfers must continue on an ongoing basis.
The transfer team evaluates options across five peer factors:
Collecting requirements follows an ordered flow where each step depends on the previous one:
Storage Transfer Service is Google Cloud's fully managed online service for moving large object and file datasets into Cloud Storage without managing infrastructure or writing transfer code. Data flows from the source system over the network to the destination Cloud Storage bucket. The service is optimized for transfers involving more than 1 TiB of data and can move petabytes quickly and reliably, but it depends on available network bandwidth between source and destination.
The service supports movement across diverse source and sink combinations:
The service automates transfers to eliminate manual processes and custom scripts. Its highly parallelized architecture accelerates speed with automatic retries and load balancing for reliability. Security is built in through end-to-end encryption in transit with TLS 1.3 for HTTPS communication plus data integrity validation to detect corruption. Common uses are migration to Cloud Storage, backup and cross-region bucket copies, feeding data processing pipelines for BigQuery or Managed Service for Apache Spark, and archival of cold data from costly on-premises storage.
Transfer Appliance is physical hardware for offline migration of very large volumes to Google Cloud without disrupting business operations. Data flows first from on-premises systems to the appliance locally, then the appliance is shipped to Google Cloud where data is loaded into Cloud Storage, so the bulk movement does not consume company network bandwidth. Transfer Appliance is recommended when data exceeds 20 TB and scales up to 1 petabyte.
The offline model changes the cost and timeline tradeoff compared with online transfer. Ingress to Cloud Storage is free, but online moves can incur egress and read-operation charges from another cloud provider and added bandwidth costs from a private data center, while an appliance adds product costs plus personnel and logistics for handling hardware. Transfer time for online moves is bounded by physics: 1 GB takes about eight seconds over 1 Gbps in ideal conditions, scaling to about 12 days for 100 TB before adding management time and efficiency losses. When bandwidth is limited, peak-hour congestion is a risk, or the deadline cannot absorb a multi-day or multi-week network copy, the offline appliance avoids overloading the production network.
The choice between online and offline transfer turns on volume thresholds, network capacity, and how quickly the data must be usable in Cloud Storage. Small bucket-to-bucket moves under 1 TB typically use gcloud, moves over 1 TB use Storage Transfer Service, and on-premises moves over 20 TB up to 1 petabyte are candidates for Transfer Appliance. Performance bottlenecks also differ by data shape: billions of small files tend to be QPS-bound, while large objects tend to be bandwidth-bound, with Storage Transfer Service able to complete around 1000 tasks per transfer job per second when sufficient regional bandwidth is available.
A decision tree branching on dataset size and available bandwidth to choose gcloud for under 1 TB, Storage Transfer Service for online moves over 1 TiB, or Transfer Appliance for offline moves of 20 TB to 1 PB.
| Option | Transfer method | Best-fit volume | Network dependence |
|---|---|---|---|
| Storage Transfer Service | Fully managed online copy over the network to Cloud Storage | Optimized above 1 TiB, scales to petabytes | Depends on allocated bandwidth; self-hosted agents can control routing and bandwidth use |
| Transfer Appliance | Offline hardware loaded locally, then shipped to Google Cloud | Recommended above 20 TB up to 1 petabyte | Avoids bulk network use and business disruption during the copy |
Downtime can be reduced for online bucket migrations because transfers are incremental by default and do not lock reads or writes. A seed transfer first copies the bulk data while the source stays open, then a sync transfer copies only data that changed after the source is locked, so the locked window is short. Storage Transfer Service provides no service level agreement (SLA) for performance or latency, so teams should measure speed with a representative data subset, schedule around peak hours, and monitor Pub/Sub notifications for failures or changes in completion time.
Exam tip: Use gcloud below 1 TB, Storage Transfer Service above 1 TiB and for petabyte-scale online moves, and Transfer Appliance for on-premises moves above 20 TB up to 1 petabyte when bandwidth or timeline rules out network transfer.
Real-time ingestion and streaming architecture is a design for continuous data movement where events are ingested as they occur and processed without waiting for a batch window. In Google Cloud, Pub/Sub provides the durable ingestion layer that accepts messages from publishers, and Dataflow (opens in a new tab) provides the parallel processing layer that transforms the stream and delivers results to sinks. Data flows from publishers to a Pub/Sub topic, then from the topic to Dataflow workers, then from Dataflow to destinations such as BigQuery or Cloud Storage.
Pub/Sub is a scalable messaging service that decouples event producers from event processors for low-latency delivery. Publishers send messages to a topic, Pub/Sub stores the messages durably, and subscribers such as Dataflow streaming jobs receive them independently. This decoupling lets ingestion continue during transient spikes in traffic and lets processing scale separately from publishing.
Pub/Sub protects streaming data with encryption both at rest and in transit by using Google-owned and Google-managed encryption keys by default. When an organization needs control over keys, Pub/Sub supports customer-managed encryption keys (CMEKs) for message encryption at the application layer. When an organization has data residency requirements, Pub/Sub uses message storage policies to ensure that message data is stored in specific locations.
Pub/Sub handles throughput pressure with publisher-side controls and project-level quotas. To avoid errors during transient spikes, publishers can configure flow control in the publisher settings to limit the rate of publish requests. To handle failed publishes, operators adjust retry-request variables to retry delivery as necessary. Because publish and subscribe throughput quotas apply at the project level, deploying each pipeline into its own project gives each pipeline independent throughput on a shared topic.
Dataflow is a managed stream and batch processing service that runs the stages of a pipeline in parallel across multiple workers. A Dataflow streaming job reads continuously from sources such as Pub/Sub, applies transforms for filtering, enrichment, and aggregation, and writes continuously to sinks such as BigQuery and Cloud Storage. The service depends on Compute Engine virtual machine (VM) instances to run jobs, so the job depends on sufficient Compute Engine quota to start and to autoscale.
A streaming pipeline follows an ordered flow from source through processing to sink:
Calling external services from inside transforms adds per-call overhead that can reduce performance and exhaust quotas. Dataflow parallelizes work across many workers, so too much traffic can overwhelm an external service, and autoscaling can add more workers that increase pressure further. To reduce overhead, the pipeline should batch multiple data elements into single requests where possible, and operators should limit batch sizes and concurrent calls to sustainable levels for the external system.
Performance of a streaming design depends on the pipeline code, the Dataflow service, and factors outside both. Three external factors constrain how far the design can scale: scalability of external systems, choice of data formats, and data location and network topology.
Using Google Cloud managed sources and sinks simplifies capacity planning because services such as Pub/Sub and BigQuery provide built-in scalability and quotas that fit most requirements. The tradeoff is that quotas and limits still apply at both layers, so architects must monitor Dataflow quotas, Compute Engine quotas, and Pub/Sub quotas together. When a single streaming pipeline needs maximum allowable throughput on a topic, placing each publishing or subscribing pipeline in its own project isolates its quota consumption.
| Layer | Primary role | Scaling control |
|---|---|---|
| Pub/Sub | Durable event ingestion and delivery to and from streaming pipelines | Publisher flow control, retry settings, and per-project throughput quotas |
| Dataflow | Parallel transformation and delivery from sources to sinks | Parallel workers, autoscaling, and Compute Engine quota |
The table shows a division of responsibility where Pub/Sub absorbs ingestion spikes and Dataflow scales compute. A spike first hits Pub/Sub flow control and retry behavior, then hits Dataflow worker parallelism and downstream sink capacity.
Secure streaming design combines encryption, residency controls, and audit visibility across both Pub/Sub and Dataflow. Pub/Sub encrypts messages by default and supports CMEKs and message storage policies for key control and location restriction. Cloud Logging records Admin Activity audit logs by default for API calls that modify configuration or metadata, and operators can enable Data Access audit logs to track reads of configuration and user-provided data. Cloud Logging can also store log data in a specified region to help meet residency requirements.
Operating streaming pipelines requires different deployment handling than batch pipelines. A passing build packages tested pipeline code into deployment artifacts, such as a self-executing JAR file or template files stored in a Cloud Storage bucket, and continuous delivery copies those artifacts to preproduction for end-to-end tests before updating production. For batch pipelines, continuous deployment can directly launch a new Dataflow job or let systems such as Managed Service for Apache Airflow or Cloud Scheduler launch jobs when required. For streaming pipelines, replacing or updating a running job is more complex, so teams either update the existing streaming pipeline in place or run parallel Dataflow jobs to coordinate the cutover and minimize business disruption.
Exam tip: Use Pub/Sub message storage policies to control where message data is stored, and use CMEKs when you need to control the encryption keys; the two controls solve different residency versus key-control requirements.
Optimizing network connectivity and security for data transit means choosing a private connection between on-premises, edge, and Google Cloud that balances bandwidth, latency, cost, and encryption. Cloud Interconnect (opens in a new tab) extends an on-premises network to Virtual Private Cloud (VPC) networks in Google Cloud without traversing the public internet, while Cloud VPN provides encrypted connectivity over the public internet. Traffic from an on-premises network flows through VLAN attachments and Cloud Router into the VPC network, and encryption is added with high-availability (HA) VPN over Cloud Interconnect or Media Access Control Security (MACsec) when required.
Google Cloud offers four main connectivity patterns for hybrid and multicloud transit:
Dedicated Interconnect is a direct physical connection between an on-premises network and Google's network in a colocation facility. Traffic flows directly between the on-premises routers and the VPC network, not through the public internet and not through a service provider. The connection requires the customer to install and maintain routing equipment in a colocation facility that supports the regions to connect.
Dedicated Interconnect scales for demanding data needs with one or more 10-Gbps, 100-Gbps, or 400-Gbps Ethernet circuits, with a maximum of 8 circuits per connection. VLAN attachment capacity is flexible from 50 Mbps to 50 Gbps, and high-bandwidth use can make Dedicated Interconnect cost-effective. Choose Dedicated Interconnect when the organization can be present in a supported colocation facility and needs sustained high throughput with an end-to-end SLA under Google-defined redundant topologies.
Partner Interconnect is a connection to VPC networks through a supported service provider that connects directly to Google. Traffic flows from the on-premises network through the service provider's network to Google's edge network, not through the public internet. The customer does not install routing equipment in a colocation facility because the provider supplies more points of connectivity.
Partner Interconnect offers flexible capacity from 50 Mbps to 50 Gbps per VLAN attachment, so the customer pays only for needed capacity. Capacity increases depend on the service provider's available capacity. For Layer 2 connections, the customer configures Border Gateway Protocol (BGP) on on-premises routers and Cloud Routers, while for Layer 3 connections Cloud Router configuration is fully automated and on-premises configuration depends on the provider. Choose Partner Interconnect when the organization cannot physically meet Google in a colocation facility or needs lower bandwidth without managing colocation hardware.
| Dimension | Dedicated Interconnect | Partner Interconnect |
|---|---|---|
| Connection path | Direct from on-premises to Google, no provider or internet in path | From on-premises through service provider to Google, no internet in path |
| Capacity | 10-Gbps, 100-Gbps, or 400-Gbps circuits, up to 8 circuits; VLAN attachments 50 Mbps to 50 Gbps | VLAN attachments 50 Mbps to 50 Gbps, limited by provider capacity |
| Setup and routing | Routing equipment in colocation facility; customer configures BGP on both sides | No colocation equipment; Layer 3 automates Cloud Router configuration |
| SLA and cost | End-to-end SLA for Google-defined topologies; dedicated circuit pricing | SLA covers Google to provider; provider may offer end-to-end SLA; billed by attachment capacity plus egress plus provider charges |
Exam tip: Cloud Interconnect traffic is not encrypted by default — deploy HA VPN over Cloud Interconnect to encrypt traffic for either Dedicated Interconnect or Partner Interconnect.
Cloud VPN is a managed virtual private network (VPN) service that creates encrypted Internet Protocol Security (IPsec) tunnels over the public internet using private IP addresses. HA VPN over Cloud Interconnect combines HA VPN with Cloud Interconnect to encrypt traffic between the on-premises network and Google's network at the network connectivity layer. Media Access Control Security (MACsec) for Cloud Interconnect provides line-rate encryption for Interconnect and Cross-Cloud Interconnect where supported.
Cloud Interconnect by itself provides private transport but does not encrypt, so sensitive traffic needs application-layer encryption, Transport Layer Security (TLS), or HA VPN over Cloud Interconnect. Cloud VPN is suitable to start quickly or for lower bandwidth needs, and VPN tunnels can achieve a 99.99% SLA with the right HA topology. Choose HA VPN over Cloud Interconnect or MACsec when compliance or business requirements demand encryption in transit at the connectivity layer in addition to private transport.
Cross-Cloud Interconnect is a managed dedicated physical connection between a Google Cloud VPC network and a network hosted by another cloud service provider. Google provisions the physical connection between the clouds, so no intermediate customer routing equipment or colocation contract is needed. A separate pair of connections, one primary and one redundant, is purchased for each provider to qualify for the Cloud Interconnect SLA.
Cross-Cloud Interconnect supports Amazon Web Services, Microsoft Azure, Oracle Cloud Infrastructure, and Alibaba Cloud in supported locations. Capacity is available in 10-Gbps or 100-Gbps pairs, with up to 100 Gbps per link, and data flows from the Google Cloud region to Google's edge point of presence (PoP), through the cross-cloud facility, then from the other provider's edge to its hosting region. Choose Cross-Cloud Interconnect when distributed applications need high-throughput, low-latency private IP connectivity between clouds without managing cross-connects.
Cloud Router is a distributed and fully managed BGP speaker that exchanges routes between on-premises, other clouds, and Google Cloud. Cloud Router works with Cloud Interconnect, Cloud VPN, and router appliances to create dynamic routes in VPC networks based on BGP-received and custom-learned routes. On-premises routers peer with Cloud Routers, and learned on-premises prefixes are applied as custom dynamic routes in the VPC network.
This routing dependency means the VPC network depends on Cloud Router for automatic path updates when links or topologies change. Cloud Interconnect provides access to all Google Cloud products and services from on-premises except Google Workspace, plus supported application programming interfaces (APIs) through Private Google Access from on-premises hosts. For managed services such as Cloud Storage and BigQuery that sit outside the VPC by default, use Private Service Connect endpoints, Private Google Access, or transfer services so that traffic can use the private connection with internal IP controls.
Storage Transfer Service is optimized for fully managed online moves above 1 TiB into Cloud Storage and can scale to petabytes when sufficient network bandwidth is available. Transfer Appliance is recommended for offline moves from on-premises systems above 20 TB up to 1 petabyte by loading data locally and shipping the hardware to Google Cloud to avoid bulk network use and business disruption.
Dedicated Interconnect is a direct physical connection between on-premises routers and the Virtual Private Cloud network that requires customer routing equipment in a colocation facility and supports up to 8 circuits of 10-Gbps, 100-Gbps, or 400-Gbps with VLAN attachments from 50 Mbps to 50 Gbps. Partner Interconnect connects through a supported service provider without colocation equipment and offers VLAN attachments from 50 Mbps to 50 Gbps limited by provider capacity.
Data flows from publishers to a Pub/Sub topic, then from the topic to Dataflow workers, then from Dataflow to sinks such as BigQuery or Cloud Storage. Pub/Sub provides the durable ingestion layer that decouples producers from processors, while Dataflow provides the parallel processing layer that transforms the stream and delivers results to sinks.
Cloud Interconnect does not encrypt traffic by default and provides private transport without traversing the public internet. When encryption at the connectivity layer is required, deploy high-availability (HA) VPN over Cloud Interconnect or Media Access Control Security (MACsec) for Cloud Interconnect where supported.
A media archiving enterprise needs to migrate 600 TB of uncompressed video assets from on-premises NFS and SMB storage arrays to Google Cloud Storage.
The organization has established the following operational constraints:
Which data transfer strategy should you recommend to meet these requirements?
Last updated