Prepare and test your skills
Prepare and test your skills
High-Performance Storage and Object Operations is the use of command-line tools to move and manage many Cloud Storage buckets and objects quickly and consistently. The work centers on two tools: gsutil (opens in a new tab), the legacy Python application for Cloud Storage, and gcloud storage (opens in a new tab), the recommended command set inside the Google Cloud CLI. All transfers use HTTPS with transport-layer security (TLS), and at least one side of a gcloud storage transfer must be a Cloud Storage URI such as gs://my-bucket.
Storage command-line tools are the programs that create buckets and move objects without using the console. gsutil is a Python application that performs bucket and object management from the command line. The tool can be used for a wide range of tasks:
gcloud storage is the recommended replacement because it requires less manual tuning to reach the fastest upload and download rates. In contrast, gsutil is minimally maintained and does not support newer features such as soft delete and managed folders. After March 2027, gsutil will no longer ship with the Google Cloud CLI and will be distributed only as a standalone tool through PyPI.
The two tools differ in maintenance, features, and performance work:
| Tool | Maintenance | Feature support | Performance tuning |
|---|---|---|---|
| gsutil | Legacy, minimally maintained | No support for soft delete or managed folders | Requires more manual optimization |
| gcloud storage | Recommended in Google Cloud CLI | Supports hierarchical namespace buckets and current features | Parallel processing built in with less manual tuning |
Use gsutil version -l to check the installation type before migrating. A result containing using cloud sdk: False means standalone gsutil, so install the Google Cloud CLI first, while using cloud sdk: True means the CLI is already present.
Cloud Storage resource addresses use the gs:// prefix in the form gs://BUCKET_NAME/OBJECT_NAME. The Cloud Storage address names the bucket that holds the data and the object path inside that bucket, and commands can also use wildcards to match many objects with one pattern. This lets one command start from a local directory or bucket and fan out to many matching sources and destinations.
For operations with very large file counts, gsutil reads the source list from standard input with the -I flag instead of from command-line arguments. The input is one local path or gs:// URL per line, so a file such as example_file_list.txt can list objects from different buckets and local paths together. Data flows from the list file through standard input into gsutil, and gsutil then copies each entry to the destination bucket, as in cat example_file_list.txt | gsutil -m cp -I gs://example-destination-bucket/.
Parallel and resilient transfers are the copy behaviors that control speed and completion when many files are involved. Parallel processing achieves high throughput by transferring many files or parts at the same time. With gsutil, parallelism is requested explicitly with the -m flag, as in gsutil -m cp, while gcloud storage applies parallel processing automatically. The gcloud storage service does not support parallel invocations from multiple terminals on the same machine, and it simplifies three or more consecutive asterisks (***) into a single asterisk (*).
Copy behavior also differs in ways that matter for automation. gcloud storage cp recognizes simulated folders created in the console as 0-byte placeholder objects and can copy them, whereas gsutil cp cannot. When one source in a multi-resource copy is invalid, gcloud storage cp attempts to copy all remaining resources while gsutil cp may stop the entire operation. When downloading, gcloud storage cp creates missing local directories in the destination path, while gsutil cp fails if the destination directory does not exist.
Exam tip: gcloud storage cannot copy local file to local file — at least one argument must be a Cloud Storage URI such as gs://my-bucket.
Access Control Lists (ACLs) are per-bucket and per-object rules that grant read or write access to specific users or groups. The gsutil commands acl get, acl set, and acl ch read, replace, and change those grants, and each has a gcloud storage equivalent that operates on a bucket or object resource. The gcloud form uses describe to read the current ACL and update with --acl-file, --add-acl-grant, or --remove-acl-grant to replace or adjust grants. Because gcloud storage and gsutil format listings, datetimes, and error messages differently, scripts that parse gsutil acl or ls output must be updated and retested, since gcloud storage always reports datetimes in UTC and returns errors at the end of output.
Core CLI configuration and environment management is how the Google Cloud CLI (opens in a new tab) remembers which project, account, region, and behavior to use for every command. A configuration is a named collection of key-value properties that supplies those defaults, so switching configurations changes the full working context without retyping flags. Combined with predictable output flags, configurations let one workstation or script move safely between projects and accounts.
Initializing the gcloud CLI environment is the process that links local tools to an account and a default project. Cloud Shell runs in the browser on a Google-provisioned virtual machine that already includes the latest Google Cloud CLI and a persistent home directory, while a locally installed CLI must be kept current manually. In both cases gcloud init performs authorization and initial setup, and later commands inherit the project and account it stores.
A typical setup follows an ordered flow:
A left-to-right process flow showing four ordered gcloud CLI setup steps: install or update, initialize with gcloud init, set project, then verify with config list and info.
gcloud components update to get the latest version.gcloud init to sign in, pick a configuration, and choose default project behavior.gcloud config set project PROJECT_ID so later resource commands target that project.gcloud config list to show current properties and gcloud info to show environment details, version, and active configuration.Only after initialization does project selection take effect, and only after the project property is set do resource commands run against the intended project without an explicit flag.
A gcloud CLI property is a single setting such as project, account, compute zone, region, or prompt behavior that influences how commands run. A gcloud CLI configuration groups those properties into a profile, starting with a single configuration named default created by gcloud init. Creating additional named configurations allows one machine to hold separate contexts for development, test, and production, each with its own project and account.
Common management commands include:
gcloud config set defines a property such as compute zone for the current configuration.gcloud config get fetches the value of a single property.gcloud config list displays all properties for the current configuration.gcloud config configurations create creates a new named configuration.gcloud config configurations list displays all available configurations.gcloud config configurations activate switches the active configuration.The active configuration supplies the project and account to every gcloud invocation, so activating a different configuration redirects subsequent commands to a different project and identity. Use gcloud config set project when only the project changes within the same context, and use configurations activate when project plus account and other defaults must change together.
Authorization for the gcloud CLI determines which identity commands act as when they call Google Cloud APIs. User account authorization ties commands to a human login, while service account authorization ties commands to a JSON key file that can be copied across machines. The Compute Engine virtual machine case is special because all users with root access share the machine, so a shared service account avoids mixing personal credentials.
| Authorization method | Use when | Activation |
|---|---|---|
| User account | Script or automation runs on a single machine by one operator | gcloud init or gcloud auth login sets the current account as active |
| Service account | Automation is deployed across machines in production or runs on a Compute Engine instance where all users have root access | gcloud auth activate-service-account --key-file [KEY_FILE] using a downloaded JSON key |
Credentialed accounts can be inspected with gcloud auth list, and access can be removed with gcloud auth revoke. For virtual machine access, gcloud compute ssh handles authentication to the instance and gcloud compute config-ssh writes SSH configuration, so scripts do not need to manage keys manually.
Script-friendly output controls (opens in a new tab) make gcloud CLI results predictable enough for Jenkins jobs and other automation. The --filter flag selects which resources are returned, such as instances in zone:us-central1-a or projects where labels.env=test AND labels.version=alpha. The --format flag controls how the selected data is rendered for the next tool in the pipeline, including json, yaml, csv, text, and table forms.
Interactive prompts would stall automation, so prompts must be suppressed when defaults are acceptable. Setting the disable_prompts property to True in the configuration disables prompts for all commands in that context, while adding the global --quiet or -q flag disables prompts for one invocation and accepts default values for confirmations. Data flows from the API response through the filter to the formatter and then to the script, so filtering first reduces volume and formatting to JSON preserves field names for parsing.
Exam tip: Parallel execution of multiple gcloud CLI commands is not supported, so run automated gcloud calls sequentially even when scripts handle other work in parallel.
Data warehouse orchestration in BigQuery (opens in a new tab) is the practice of organizing BigQuery operations into larger workloads where external tasks trigger BigQuery jobs and BigQuery results trigger downstream work. A workload can chain data loading, transformation, and reporting steps across data resources and processes, with tasks executed in the correct order using the correct data. Scheduling methods and tools help design, build, implement, and monitor these chains so dependencies and parameters stay consistent.
Event-driven and time-driven scheduling are the two methods that decide when a BigQuery workload runs. An event is a state change, such as a change to data in a database or a file added to a storage system. In event-driven scheduling, an action such as a website interaction or an object landing in a Cloud Storage bucket triggers data processing immediately on arrival. In time-driven scheduling, work runs on a clock, such as loading new data once per day or running frequently enough to produce hourly reports. Both methods can combine in one workload, for example loading objects into a data lake in real time while generating activity reports on that lake only daily.
Running BigQuery jobs programmatically is how operators and code submit the same work to the BigQuery execution service through different interfaces. Data flows from the chosen interface to the BigQuery service, which then generates an execution plan and writes results to persistent storage.
bq query command in the bq command-line tool.jobs.query or jobs.insert method in the BigQuery REST API.The bq query command provides the command-line path for orchestration, letting scripts and scheduled workflows submit the same GoogleSQL work that runs in the console. Because results are written to persistent storage, later steps in a workload can read those results or serve them from cache on repeat runs.
Scheduling tools for BigQuery workloads manage complex chains such as combining multiple Google Cloud or third-party services with BigQuery jobs or running multiple BigQuery jobs in parallel. The tool enforces dependency and parameter management so each task runs in order with the right inputs. Google Cloud recommends Dataform, Workflows, Managed Service for Apache Airflow, and Vertex AI Pipelines for most use cases.
| Tool | Focus | Code type | User profile |
|---|---|---|---|
| Dataform | Data transformation | JavaScript, SQL, Python notebooks | Data analyst or administrator |
| Workflows | Microservices | YAML or JSON | Data architect |
| Managed Service for Apache Airflow | ETL or ELT | Python | Data engineer |
| Vertex AI Pipelines | Machine learning | Python | Data analyst |
The four tools differ in complexity and fit. Dataform is the simplest option at complexity level one, is serverless, and is not suitable for chains of external services. Workflows and Vertex AI Pipelines are intermediate complexity and serverless, but Workflows is not suitable for data transformation and processing while Vertex AI Pipelines is not suitable for infrastructure tasks. Managed Service for Apache Airflow is the most complex at level three, is fully managed rather than serverless, and is not suitable for low-latency or event-driven pipelines.
Scheduled queries are the simplest form of workload scheduling and run recurring queries directly in BigQuery. A scheduled query workflow supports only straightforward query chains with no external dependencies and follows time-driven scheduling. Queries scheduled in this way must be written in GoogleSQL and can include data definition language (DDL) and data manipulation language (DML) statements. DDL statements define and change table structures that establish the warehouse schema, while DML statements insert, update, and delete the rows stored under that schema.
Exam tip: Choose scheduled queries only for straightforward, time-driven query chains with no external dependencies; choose Dataform, Workflows, Managed Airflow, or Vertex AI Pipelines when the workload combines services or runs parallel jobs.
gcloud storage is the recommended Cloud Storage command set and applies parallel processing automatically, while gsutil requires the explicit -m flag and will no longer ship with the Google Cloud CLI after March 2027.gs://BUCKET_NAME/OBJECT_NAME, and at least one argument of a gcloud storage copy must be a Cloud Storage URI.gcloud config configurations activate switches that full context.gcloud storage is the recommended command set inside the Google Cloud CLI and applies parallel processing automatically with less manual tuning, while gsutil is a legacy minimally maintained Python application that requires the explicit -m flag for parallelism. gsutil does not support newer features such as soft delete and managed folders, and after March 2027 it will no longer ship with the Google Cloud CLI and will be distributed only as a standalone tool through PyPI.
Scheduled queries should be used only for straightforward time-driven query chains with no external dependencies that are written in GoogleSQL and can include data definition language (DDL) and data manipulation language (DML) statements. Dataform, Workflows, Managed Service for Apache Airflow, and Vertex AI Pipelines should be used when the workload combines multiple services or runs parallel jobs with dependency and parameter management.
The active configuration is a named collection of key-value properties such as project, account, region, and zone that supplies defaults to every gcloud invocation. Switching the active configuration with gcloud config configurations activate redirects subsequent commands to a different project and identity, while gcloud config set project changes only the project within the same context.
A cloud architect is developing an administrative shell script to automate operational maintenance across multiple environments that use distinct service accounts and settings.
The script must satisfy the following technical requirements:
Which combination of gcloud CLI commands and flags meets these requirements?
Last updated