Prepare and test your skills
Prepare and test your skills
Differentiate Managed and Visual Extraction Services means comparing fully managed tools that load data into BigQuery without managing servers. Cloud Data Fusion (opens in a new tab) builds visual pipelines that clean and reshape data during loading, while BigQuery Data Transfer Service (opens in a new tab) runs automated, scheduled loads from known sources without code. Both write to destination datasets in BigQuery (opens in a new tab), but they differ in how work is defined and when transformation happens.
Cloud Data Fusion for visual pipeline building is a fully managed service for creating ETL and ELT pipelines on a graphical canvas. A developer drags connectors and transformations onto the canvas instead of writing extraction code, and the service runs the resulting pipeline on Google Cloud. The pipelines reuse an open-source library of preconfigured connectors and transformations, so common sources and cleaning steps do not have to be rebuilt. Choose Cloud Data Fusion when extraction requires visual construction plus transformation before loading into BigQuery.
BigQuery Data Transfer Service for scheduled ingestion is a managed service that automates data movement into BigQuery on a regular schedule. After a transfer is configured, data flows from the source system to the destination BigQuery dataset automatically, and backfills can recover from outages or gaps. Data flows only into BigQuery, because the service cannot transfer data out of BigQuery. The service is accessed through the Google Cloud console, the bq command-line tool, or the BigQuery Data Transfer Service API.
The transfer configuration lives in the same location as the destination dataset, and the service processes and stages data in that same location. It supports scheduled loads from many source groups:
In addition to external loads, the service also powers dataset copies and scheduled queries inside BigQuery. Choose BigQuery Data Transfer Service when the need is recurring, automated transfers from SaaS applications, app stores, marketing systems, storage, or supported databases directly into BigQuery.
Exam tip: BigQuery Data Transfer Service loads data only into BigQuery and cannot transfer data out of BigQuery.
Deciding between transformation during load or after scheduling depends on whether data must be reshaped and where compute is available. Cloud Data Fusion transforms data as part of the visual pipeline that writes to BigQuery, while BigQuery Data Transfer Service moves data on a schedule as defined by the connector. Transforming in the current environment depends on local compute capacity and may need extra transfer time or network bandwidth, while transforming on Google Cloud uses a managed ETL or ELT pipeline that writes directly to BigQuery.
| Service | How work is defined | Choose when |
|---|---|---|
| Cloud Data Fusion | Visual pipeline with preconfigured connectors and transformations | Data must be transformed or enriched while it is extracted and loaded |
| BigQuery Data Transfer Service | No-code transfer configuration with schedule and destination dataset | Data must move on a schedule from a supported SaaS, storage, database, or Google service into BigQuery |
Evaluate Dataflow for Real-time Stream and Batch Processing means understanding when a unified batch and streaming service fits an extraction task. Dataflow (opens in a new tab) is a serverless and fully managed data-processing service that runs both bulk and continuously arriving data with one model. A pipeline is defined with the Apache Beam model and run by the Dataflow service, which provides virtually limitless capacity without servers to manage. Data flows from sources such as Cloud Storage (opens in a new tab) into Dataflow for transformation, and then Dataflow writes results to destinations such as BigQuery (opens in a new tab).
The unified batch and streaming model means the same Beam-based logic can process large files and live event streams. Dataflow is beneficial when new files land in a Cloud Storage bucket and need processing as they arrive, and when historical backfills and live streams must share the same transformation logic. Dataflow is also beneficial for migration work where extracted warehouse files are staged in Cloud Storage and then transformed while loading into BigQuery. Choose this unified model when latency, volume, and transformation complexity vary over time but the pipeline logic should stay consistent.
Managed scale and complex transformation features let Dataflow handle multi-step enrichment on Google Cloud. Because transformation runs on Google Cloud, teams avoid extra transfer time and bandwidth that enrichment in the source environment would require. Capacity scales automatically instead of depending on fixed compute in the source environment.
Implementing pipelines with Apache Beam connectors means writing pipeline logic once in Beam and letting Dataflow run it for batch or streaming data. The pipeline depends on Beam I/O connectors for its sources and destinations, including BigQueryIO for BigQuery, BigtableIO for Bigtable, AvroIO for Avro files, and ParquetIO for Parquet files. A common extraction and load pattern moves data through an ordered sequence:
The same pattern supports change-data-capture flows where Datastream writes change events to Cloud Storage and a Dataflow template then loads them to BigQuery, Spanner, or Cloud SQL. Google-provided templates cover both modes, with streaming templates such as Bigtable change streams to BigQuery and batch templates such as Cloud Storage Avro to Bigtable, BigQuery to Bigtable, and Cloud Storage Parquet to Bigtable.
Choosing code-defined transformation over visual or Spark options depends on complexity and team skills. Dataflow fits loads that need further transformation while data moves from Cloud Storage into BigQuery, especially when logic is complex or must serve both one-time bulk loads and ongoing arrivals.
| Option | How you build it | Choose when |
|---|---|---|
| Dataflow | Define transformation graphs with the Apache Beam model | The load needs complex transformation and enrichment for batch, streaming, or both |
| Cloud Data Fusion | Build pipelines graphically with preconfigured connectors and transformations | The team wants fully managed, code-free ETL or ELT assembly |
| Managed Service for Apache Spark | Run Apache Spark and Apache Hadoop workloads on a managed service | The work is already expressed as Spark or Hadoop jobs |
Exam tip: Choose Dataflow when the requirement is custom, code-defined transformation with unified batch and streaming, not scheduled file copies without transformation.
Analyze Use Cases for Each Extraction Tool means matching each Google Cloud extraction service to the right migration task. BigQuery Data Transfer Service (opens in a new tab) automates scheduled loads without code, Cloud Data Fusion (opens in a new tab) builds visual ETL and ELT pipelines without code, and Dataflow runs custom large-scale transformations with code. Database Migration Service (opens in a new tab) fits the separate case of migrating an operational database and its related apps, rather than loading warehouse export files.
Dataflow for custom transformation at scale is a serverless and fully managed service that runs complex transformation and enrichment graphs using the Apache Beam model. Data flows from a staging bucket in Cloud Storage into Dataflow, Dataflow applies the transformation graph with virtually limitless processing capacity, and Dataflow writes the result directly to BigQuery, often using the Beam BigQuery I/O connector. New files that arrive in Cloud Storage can be processed as they arrive, so the pipeline depends on the staged files and the destination BigQuery dataset. Choose Dataflow when data must be trimmed, joined, enriched, or reshaped before loading, especially for large or continuous volumes where local compute would limit throughput. The tradeoff is that Dataflow requires building a Beam pipeline in code, so it fits teams comfortable with complex logic.
BigQuery Data Transfer Service for recurring no-code loads automates data movement into BigQuery on a scheduled basis. After a transfer is configured in the Google Cloud console, bq command-line tool, or transfer API, the service loads data on a regular schedule and can run backfills to recover from outages or gaps. The transfer configuration lives in the same location as the destination BigQuery dataset, and the service processes and stages data in that same location. The service loads only into BigQuery and also supports dataset copies and scheduled queries.
It supports many source types that can be selected without building a pipeline:
Choose this service for recurring ingestion from a supported source, including direct warehouse-to-BigQuery moves from Amazon Redshift and Teradata where Cloud Storage is still used for staging. The limitation is flexibility, because the service moves data as defined by the connector on a schedule and does not replace custom transformation when complex enrichment is needed.
Cloud Data Fusion for visual data integration is a fully managed, code-free service that builds pipelines through a graphical interface. A user selects sources, transformations, and destinations from an open-source library of preconfigured connectors and transformations, and the service runs the resulting pipeline. Data flows from the source system through the visual pipeline steps into the destination such as BigQuery, with transformation happening as part of the load. Choose Cloud Data Fusion when integration speed and visual development matter more than custom code, such as combining rows, joining datasets, or cleaning files before analysis. The tradeoff is less control than Dataflow for very complex or highly custom processing graphs.
Exam tip: BigQuery Data Transfer Service moves data into BigQuery, including scheduled loads and backfills, but it cannot be used to transfer data out of BigQuery.
Matching tools to migration and integration needs depends on coding preference, transformation depth, and starting point. For an initial warehouse load, the common pattern is to export schema and data to vendor-neutral files such as Avro, ORC, or Parquet, upload those files to a Cloud Storage (opens in a new tab) bucket in the same location as the BigQuery dataset, and then load through the transfer service, Dataflow, or Cloud Data Fusion. Use Avro, ORC, or Parquet to keep schema embedded with the data, while CSV requires the schema to be specified separately at load time or auto-detected.
A decision tree branching on operational DB migration, scheduled loads, visual cleaning, and custom batch-streaming transform to pick Database Migration Service, BigQuery Data Transfer Service, Cloud Data Fusion, or Dataflow into BigQuery.
| Service | How you build it | Choose when |
|---|---|---|
| BigQuery Data Transfer Service | No-code scheduled transfer | Data comes from a supported SaaS, marketing, storage, Google service, or warehouse source and needs recurring loads into BigQuery |
| Cloud Data Fusion | Visual, code-free pipeline with preconfigured connectors | You need ETL or ELT with joins, cleaning, and integration across systems without writing code |
| Dataflow | Apache Beam code, serverless execution | You need complex transformations and enrichment at large scale, including streaming files from Cloud Storage to BigQuery |
| Database Migration Service | Managed database migration | You need to migrate an operational database and its related apps, not just warehouse export files |
Cloud Data Fusion builds fully managed ETL and ELT pipelines graphically with preconfigured connectors and transformations that clean and reshape data during loading, while BigQuery Data Transfer Service runs automated scheduled no-code loads from known sources without code. Both write to destination datasets in BigQuery, but they differ in how work is defined and when transformation happens.
Choose Dataflow when the load needs complex transformation and enrichment for batch, streaming, or both, especially for large or continuous volumes where local compute would limit throughput. Dataflow is a serverless and fully managed service that runs Apache Beam pipelines with virtually limitless capacity, with data flowing from Cloud Storage into Dataflow for transformation and then directly to BigQuery.
No, BigQuery Data Transfer Service loads data only into BigQuery and cannot transfer data out of BigQuery. It automates scheduled loads and backfills into the destination BigQuery dataset and also supports dataset copies and scheduled queries inside BigQuery.
Use Database Migration Service when you need to migrate an operational database and its related apps, not just warehouse export files. Dataflow, Cloud Data Fusion, and BigQuery Data Transfer Service load and transform data into BigQuery, including warehouse exports staged as Avro, ORC, or Parquet files in Cloud Storage.
A company wants to ingest data regularly from several external Software-as-a-Service (SaaS) applications, such as Salesforce and Google Ads, directly into BigQuery for reporting. The analytics team wants a fully managed, scheduled solution that requires no custom code or complex pipeline development.
Which Google Cloud service should the team select to meet these requirements?
Storage Transfer Service
Cloud Data Fusion
Database Migration Service
BigQuery Data Transfer Service
Last updated