An analytics team needs to collect daily marketing metrics and customer transaction records stored in an external cloud provider's object storage (such as Amazon S3 or Azure Blob Storage). The team wants to load this data automatically into a BigQuery dataset on a recurring 24-hour schedule without writing custom extraction scripts or managing virtual servers. Which Google Cloud service should the team configure to meet these requirements?

BigQuery Data Transfer Service: ### What BigQuery Data Transfer Service is **BigQuery Data Transfer Service (DTS)** is a fully managed, serverless service designed to automate data movement directly into **BigQuery** datasets on a scheduled, recurring basis. It enables organizations and analytics teams to establish seamless ingestion pipelines from various external cloud providers, SaaS platforms, and internal Google services without writing custom code or managing underlying infrastructure. ### How it meets the requirements - **Automated scheduled ingestion (✓):** The service natively supports defining recurring transfer schedules (such as daily 24-hour intervals or custom cadences) that trigger ingestion automatically. - **External cloud provider support (✓):** BigQuery Data Transfer Service provides built-in connectors for third-party cloud object storage like **Amazon S3** and **Azure Blob Storage / ADLS**, as well as major SaaS platforms like **Salesforce** and **Google Ads**. - **Direct BigQuery loading (✓):** Transferred data is loaded directly into specified destination BigQuery tables and datasets, avoiding intermediate manual import pipelines. - **No code or server management (✓):** Data transfers can be configured entirely through the **Google Cloud console**, the `bq` command-line tool, or the Data Transfer Service API without developing custom extract-transform-load (ETL) scripts. ### Key advantages and capabilities - **Pre-built connectors:** Out-of-the-box support for external cloud storage (Amazon S3, Azure Blob Storage), relational databases, and popular enterprise SaaS applications. - **Automated backfills:** Built-in ability to manually or automatically trigger historical backfill runs to recover from data gaps or outages. - **Flexible write dispositions:** Supports both `WRITE_APPEND` (for incremental batch additions) and `WRITE_TRUNCATE` (for complete table refreshes). - **Monitoring and alerting:** Integrates with email notifications and **Pub/Sub** topics to track transfer run statuses and failures. ### Why this is optimal **BigQuery Data Transfer Service** is the purpose-built solution in Google Cloud for loading external cloud and SaaS data directly into BigQuery. It eliminates maintenance overhead, reduces implementation time from weeks to minutes, and ensures reliable, automated scheduled delivery.

Bigquery Data Transfer Service Automated Ingestion | Unlock the power of your data in the cloud! Get hands-on with Google Cloud's core data services like BigQuery and Looker to validate your practical skills in data ingestion, analysis, and management, and earn your Associate Data Practitioner certification!

FundamentalAssociate Data Practitioner

A data engineering team is setting up a recurring data ingestion pipeline using Google Cloud Storage Transfer Service to sync files from an external object storage bucket to a destination Cloud Storage bucket. The team has the following requirements: The transfer must execute automatically every 24 hours. Existing files in the destination bucket must only be replaced if the corresponding source file has been modified or updated. The destination Cloud Storage bucket must remain an exact replica of the source, meaning files deleted from the source bucket must also be removed from the destination bucket. Which configuration settings should the team choose when creating the transfer job?

Load data into Google Cloud storage systems using the appropriate tool (e.g., gcloud and BQ CLI, Storage Transfer Service, BigQuery Data Transfer Service, client libraries)

FundamentalAssociate Data Practitioner

A data engineer is writing an automated bash script to ingest on-premises data into Google Cloud. The script needs to upload a local directory named /data/sales_records, which contains multiple nested subdirectories and CSV files, into a Cloud Storage bucket named gs://retail-analytics-lake. Which gcloud command should the engineer use to perform this upload?

Load data into Google Cloud storage systems using the appropriate tool (e.g., gcloud and BQ CLI, Storage Transfer Service, BigQuery Data Transfer Service, client libraries)

FundamentalAssociate Data Practitioner

A business analyst is analyzing an e-commerce orders dataset in BigQuery. The retail operations manager asks for a regional performance breakdown to answer two specific business questions: What is the total monetary revenue generated by each sales region? What is the average order value across transactions in each sales region? The dataset orders contains columns order_id, region, and order_amount. Which GoogleSQL query should the analyst write to segment the dataset and answer both business inquiries?

FundamentalAssociate Data Practitioner

A retail company needs to build a data pipeline that currently processes historical sales logs in batch from Cloud Storage, transforms the data, and loads it into BigQuery. In the near future, the company plans to ingest and transform live transactional events in real time from Pub/Sub using the exact same transformation logic and programming framework without rewriting application code. Which Google Cloud service should the company use to implement this unified batch and stream data processing pipeline?

FundamentalAssociate Data Practitioner

An organization manages multiple versions of a fraud detection machine learning model in Model Registry. The engineering team wants to streamline the model release process so that downstream serving applications can automatically consume newly promoted models (such as staging or production) without needing application code or configuration updates whenever a new version ID is generated. Which approach should the team implement in Model Registry?

IntermediateProfessional Cloud Architect

An enterprise is modernizing its event ingestion architecture on Google Cloud to support rapid feature rollout for downstream analytics teams. The infrastructure team has limited operational bandwidth and wants to avoid accumulating operational technical debt, such as manual capacity planning, broker maintenance, partition rebalancing, and custom cross-region data replication. However, some developers argue for deploying and managing open-source messaging software on Compute Engine to maintain multi-cloud API portability. Which architectural trade-off recommendation best balances organizational agility and technical debt reduction with the enterprise's operational needs?

Associate Data Practitioner

Bigquery Data Transfer Service Automated Ingestion

Related questions