Prepare and test your skills

Prepare and test your skills

Cross-service data loading and integration is the practice of staging raw files in Cloud Storage and then loading them into managed services such as BigQuery (opens in a new tab) for analysis. Data flows from source systems to a Cloud Storage landing zone, then from Cloud Storage to BigQuery tables. The movement can happen through batch loads, scheduled transfers, or processing jobs that write directly to BigQuery.
Loading files from Cloud Storage into BigQuery moves objects stored in a bucket into a BigQuery table. BigQuery supports common file examples found in Cloud Storage, including Parquet, JSON, and CSV, and a load job can either replace a table or add to it. A JSON load can use an explicit schema that defines column names and types, while Parquet and JSON loads can use truncate mode to replace the existing table contents. BigQuery can also query Cloud Storage without loading by using a temporary table or an external table, including an external table that uses Hive partitioning. The BigQuery DataFrames API can also turn a CSV file in Cloud Storage into a DataFrame for Python analysis.
The BigQuery Data Transfer Service (DTS) is the managed service for scheduling recurring loads into BigQuery. DTS can schedule recurring loads from Cloud Storage to BigQuery and can also replace extract-and-load pipelines when the source is supported directly or through third-party integrations. Supported sources include software-as-a-service apps, Teradata, Amazon Redshift, Amazon S3, Google Play, YouTube Channel reports, and Google Ads. A DTS configuration supports backfill runs to load historical data, sends run notifications to Pub/Sub when a transfer succeeds or fails, and produces log messages for each transfer run. Alternatives include Cloud Storage triggers with Cloud Functions and Managed Airflow with Composer, while orchestration can also use a pull model that retrieves files with a protocol such as SFTP or a push model where data flows through Pub/Sub or Apache Kafka and lands in the bronze tier.
Staging raw data in Cloud Storage means ingesting source files into an initial landing zone before loading them downstream. Exporting a BigQuery public dataset to a Cloud Storage bucket mimics an external system delivering raw files to that landing zone. In an extract, load, transform (ELT) pipeline, the extract-and-load part writes data into the warehouse first and the transform part then uses SQL so the data can be consumed downstream. Getting data to Cloud Storage depends on the source and size:
| Option | What it moves | Choose when |
|---|---|---|
| Storage Transfer Service | Data from other clouds, online resources, or local systems such as S3, Blob, Data Lake, and on-premises file systems into Cloud Storage | The source is outside Google Cloud and the result should land in a bucket |
| Transfer Appliance | Large volumes of data on a hardware appliance to Google Cloud | The volume exceeds 20 TB and is up to 1 petabyte and operations should not be disrupted |
| Image Import | On-premises virtual disks, including VMDK and VHD formats, into images used to create virtual machines | The goal is to migrate golden disks or golden images with software and configuration, not data files |
Processing and synchronizing data with managed frameworks is handled by Managed Service for Apache Spark (opens in a new tab), the fully managed service for running Hadoop, Spark, and Flink jobs that integrate Cloud Storage and BigQuery. The service integrates with the BigQuery connector, a Java library that lets Hadoop and Spark write data directly to BigQuery by using abstracted versions of the InputFormat and OutputFormat classes, and with the Cloud Storage connector for reading and writing bucket data. The service favors many ephemeral clusters over one monolithic cluster, so each job can use its own configuration and scale and the team pays only for resources while jobs use them.
To keep a data lake synchronized with operational databases, database ingestion uses Change Data Capture (CDC). A Managed Service for Apache Spark job subscribes to a Pub/Sub topic that receives change events, processes the stream, and applies the changes to the raw datastore in Cloud Storage.
Permissions for secure data access control which identity can read bucket files and write warehouse tables. A service account used for loading and processing commonly needs Storage Object Admin to access Cloud Storage objects and Dataproc Worker to run processing jobs. Creating that setup depends on other roles: Project Creator to create a project, Service Usage Admin to enable the Compute Engine, BigQuery, and Cloud Storage APIs, and Create Service Accounts with Project IAM Admin to create the service account and grant it roles. Access to BigQuery datasets is managed separately by granting and revoking dataset access for users, groups, or service accounts.
Exam tip: Storage Transfer Service moves outside data into Cloud Storage, while BigQuery Data Transfer Service moves data into BigQuery on a scheduled, managed basis.
Command-line and local data ingestion is the process of uploading files and folders from a local computer into Cloud Storage (opens in a new tab) buckets using command-line tools. Cloud Storage is a central landing zone for big data because data flows from buckets into other Google Cloud services in the same location. Once data is in the cloud it should stay there, where transfers to services such as Compute Engine are very fast. Before using the commands below, install the Google Cloud CLI and initialize it with gcloud init.
Cloud Storage buckets support three common big data patterns:
The gcloud storage cp command copies local files to a Cloud Storage bucket and also copies objects between buckets. Data flows from the source path to the destination bucket address, which uses the form gs://example-bucket. To copy a single large local file, run gcloud storage cp local-file gs://example-bucket, and to copy between buckets, run gcloud storage cp gs://example-source-bucket/file gs://example-destination-bucket. The cp command automatically performs parallel multi-threaded and multi-processing copies as needed, which optimizes transfer speed for many files without manual tuning.
To upload a directory tree, add the --recursive flag so subdirectories are included in the transfer. For example, gcloud storage cp top-level-dir gs://example-bucket --recursive copies files from the local directory top-level-dir to gs://example-bucket. The same cp command can copy entirely in the cloud, such as gcloud storage cp gs://example-bucket/top-level-dir/subdir/** gs://example-bucket/top-level-dir/subdir/subdir2, where the service detects multiple files and creates them in the new directory subdir2. Use --recursive whenever the source is a folder rather than a single file, because only the flag tells the tool to walk the directory tree.
Wildcards in gcloud storage select a matching set of object names for one operation. The gcloud storage cp command accepts patterns such as image* to match only files that start with image, for example gcloud storage cp top-level-dir/subdir/image* gs://example-bucket --recursive. Data flows from each local file matching the pattern to the destination bucket, while non-matching files are skipped. The same pattern logic applies to removal, such as gcloud storage rm gs://example-bucket/top-level-dir/subdir/image*, which deletes only the bucket objects that match the pattern.
The gcloud storage rsync command synchronizes a local directory with a bucket so the destination matches the source. Data flows from the source directory to the destination bucket, and the command copies new and changed files in either direction or between two buckets. For example, gcloud storage rsync local-dir gs://example-bucket --recursive makes gs://example-bucket match the contents of the local directory local-dir. Add --recursive to include subdirectories in the synchronization.
By default the gcloud storage rsync command leaves extra files at the destination untouched. When the flag --delete-unmatched-destination-objects is added, the service deletes files at the destination that are not present at the source. In the example above, files in gs://example-bucket that do not exist in local-dir are removed when the flag is used. Choose the flag when the destination must become an exact mirror of the source, and omit it when the destination should keep additional files.
Resumable transfers in gcloud storage protect large uploads and downloads from network failures. The upload resumes based on how many bytes the Cloud Storage server actually received, so the tool avoids resending bytes that already arrived and the upload can eventually complete. The same logic applies to downloads based on the size of the local file. This behavior matters most for large files because the chance of an interruption increases with the size of the data being transferred over the internet.
Choosing between gcloud storage and [gsutil](https://docs.cloud.google.com/storage/docs/gsutil) means choosing between the recommended CLI and a legacy tool. Both tools use gs://BUCKET_NAME/OBJECT_NAME addresses and support wildcards, uploads, downloads, copies, moves, deletes, listings, and bucket management over HTTPS with transport-layer security (TLS). After March 2027, gsutil will no longer ship with the Google Cloud CLI and will be distributed only as a standalone tool through PyPI. The two tools differ in features, performance tuning, and future support:
| Tool | Support status | Features and performance |
|---|---|---|
gcloud storage | Recommended current CLI | Supports newer features such as soft delete and managed folders and needs less manual optimization to reach fast upload and download rates |
gsutil | Legacy, minimally maintained | Does not support soft delete and managed folders, needs more manual optimization, and requires migration to gcloud storage for new work |
The gsutil tool supports parallel transfers with the -m flag and bulk input with the -I flag, which reads source paths from standard input. For example, cat example_file_list.txt | gsutil -m cp -I gs://example-destination-bucket/ copies each local path or gs:// URL listed in the file to the destination bucket. Run gsutil version -l to check the install type: using cloud sdk: False means standalone gsutil, while using cloud sdk: True means gsutil came with the Google Cloud CLI and gcloud storage commands are already available.
Exam tip: Use gcloud storage cp and gcloud storage rsync for new work because gsutil will leave the Google Cloud CLI package after March 2027 and lacks soft delete and managed folders support.
Large-scale and automated data migration is performed by Storage Transfer Service (opens in a new tab), a fully managed Google Cloud service for moving large amounts of object and file data without managing infrastructure or writing custom code. The service is optimized for transfers involving more than 1 TiB of data and can move petabytes of data quickly and reliably. Data flows from a configured source to a configured sink, with the service handling parallelism, retries, and validation so teams can focus on applications instead of transfer mechanics.
Supported sources and destinations define which source and sink combinations Storage Transfer Service can orchestrate. The service moves data across cloud providers, on-premises environments, and public web sources, with Cloud Storage (opens in a new tab) as a common destination for migration, backup, and analytics. Before creating a job, confirm that the source and sink combination is supported.
Common uses for these movements are migrating data from another cloud, data center, or HTTP/HTTPS URL to Cloud Storage, replicating data to Google Cloud or copying a Cloud Storage bucket to another region for backup, moving data from other clouds or the edge to Google Cloud for processing in BigQuery or Managed Service for Apache Spark, and moving cold data from costly on-premises storage to Cloud Storage for archival.
Automation in Storage Transfer Service replaces manual processes and custom scripts with managed transfer jobs that move data at scale. An operator defines the source, the destination Cloud Storage bucket, and the transfer options once through the Google Cloud console, REST APIs, the gcloud command-line tool, or Java and Python client libraries, and the service then executes the movement. This approach supports both large one-time migrations and ongoing synchronization needs such as backup copies and data processing pipelines, where new files arriving in Cloud Storage can then be processed by tools such as Dataflow.
The network path controls whether Google manages the transfer infrastructure or the customer controls routing and bandwidth with self-hosted agents. Traffic flows from the source system to the destination system over the selected path, and the choice depends on where the data lives and how much control over the network is required.
| Option | Who manages infrastructure | Control over routing and bandwidth |
|---|---|---|
| Google-managed transfers | Google Cloud | Simpler setup with less direct control |
| Self-hosted agents | Customer | Granular control over network routing and bandwidth consumption |
Choose Google-managed transfers when simplicity is the priority, such as cloud-to-cloud moves between object stores. Choose self-hosted agents for hybrid scenarios involving on-premises storage, where the agents run in the customer environment and give direct control over how transfer traffic uses the corporate network.
Security, performance, and reliability in Storage Transfer Service protect data in transit and ensure complete delivery without operator intervention. End-to-end encryption protects data in transit, with support for TLS 1.3 for all HTTPS communication, and data integrity validation checks that data was not corrupted during transfer. Performance comes from a highly parallelized architecture that accelerates transfer speeds, while automatic retries and load balancing keep large transfers reliable even when performance fluctuates.
Exam tip: Storage Transfer Service is optimized for transfers over 1 TiB and provides no SLA for transfer performance or latency, so expect possible performance fluctuations on very large jobs.
gcloud storage cp --recursive copies directory trees to Cloud Storage, while gcloud storage rsync --delete-unmatched-destination-objects makes the destination an exact mirror of the source.gcloud storage is the recommended CLI and supports soft delete and managed folders, while gsutil is legacy and will leave the Google Cloud CLI package after March 2027.Storage Transfer Service moves data from outside Google Cloud into Cloud Storage, while BigQuery Data Transfer Service (DTS) schedules recurring loads into BigQuery on a managed basis. Storage Transfer Service supports sources including Amazon S3, Azure Blob Storage, on-premises storage, HDFS, and publicly accessible URLs and is optimized for transfers over 1 TiB, while DTS supports recurring loads from Cloud Storage and directly supported sources such as software-as-a-service apps, Teradata, Amazon Redshift, Amazon S3, Google Play, YouTube Channel reports, and Google Ads with backfill runs and Pub/Sub run notifications.
Use gcloud storage cp with the recursive flag to copy files or directory trees to Cloud Storage, and use gcloud storage rsync with the recursive flag to synchronize a local directory so the destination matches the source for new and changed files. By default rsync leaves extra files at the destination untouched, while adding the delete-unmatched-destination-objects flag deletes files at the destination that are not present at the source to make it an exact mirror.
gcloud storage is the recommended current command-line interface for Cloud Storage, while gsutil is a legacy minimally maintained tool. gcloud storage supports newer features such as soft delete and managed folders and needs less manual optimization for fast transfer rates, while gsutil does not support soft delete and managed folders, needs more manual optimization, and will no longer ship with the Google Cloud CLI after March 2027.
Use Transfer Appliance when the volume exceeds 20 TB and is up to 1 petabyte and operations should not be disrupted, and use Storage Transfer Service when the source is outside Google Cloud and the result should land in a Cloud Storage bucket. Storage Transfer Service moves data from sources such as S3, Blob, Data Lake, and on-premises file systems into Cloud Storage, while Transfer Appliance moves large volumes on a hardware appliance to Google Cloud.
An enterprise maintains 20 TB of unstructured file data on an on-premises Network File System (NFS) share with good internet connectivity. The organization needs to migrate this data to a Cloud Storage bucket and establish an automated, daily synchronization schedule to copy new and updated files with minimal operational overhead.
Which approach should the cloud engineer implement?
Deploy Storage Transfer Service agents in Docker containers on local machines with access to the NFS share, assign them to an agent pool, and configure a recurring Storage Transfer Service job.
Write a custom shell script using gcloud storage rsync and configure an on-premises cron job to execute the transfer every night.
Configure BigQuery Data Transfer Service to connect directly to the on-premises POSIX file system and schedule daily data ingestion.
Order a Google Cloud Transfer Appliance, load the 20 TB dataset onto the hardware, ship it to Google, and configure daily recurring synchronization through the appliance interface.
Last updated