Prepare and test your skills
Prepare and test your skills
Differentiating tool architectures and processing models means comparing how five Google Cloud tools run transformation work. The five tools are Dataproc, Dataflow (opens in a new tab), Cloud Data Fusion, Cloud Composer, and Dataform. The choice depends on whether the workload needs managed clusters, parallel graph execution, or orchestration of other services.
The tools separate into distinct architectural roles based on where compute runs and what the service controls:
Dataproc, also listed as Managed Service for Apache Spark, is a managed-cluster service for Apache Hadoop workloads. Teams provision Hadoop clusters through Dataproc instead of building and maintaining cluster infrastructure themselves. Those clusters then connect to underlying analytic data stores, so storage and compute stay separate while processing runs on the cluster. Dataproc is the fit when the workload needs Hadoop or Spark compatibility with control over cluster-based processing.
Dataflow graph execution expresses work as a graph of steps that run in parallel across workers. Dataflow is a graph-based execution system for data processing and simulations. This model fits compute-intensive simulations such as Monte Carlo simulations, where many calculations can run at the same time. HSBC used Dataflow to run risk calculations 16 times faster than its previous approach.
Cloud Composer, also listed as Managed Service for Apache Airflow, is a managed workflow orchestration service. Cloud Composer does not transform data itself; it schedules, orders, and coordinates tasks that run in other systems. Data flows from one orchestrated task to the next, and the Composer environment manages dependencies between those tasks. Cloud Composer is the fit when the need is to orchestrate a pipeline across multiple tools rather than to run the transformation compute directly.
Cloud Data Fusion and Dataform transformation options complete the set of five tools for simple data pipelines. Cloud Data Fusion and Dataform are listed alongside Dataproc, Dataflow, and Cloud Composer as portfolio options. Selection among all five tools is based on business requirements, including the desired interface, processing model, and operational characteristics. Neither tool replaces cluster provisioning or cross-service orchestration; they add interface choices for specific transformation needs.
Exam tip: Dataproc provisions Hadoop clusters, Dataflow runs graph-based execution, and Cloud Composer orchestrates workflows across other services.
Analyzing business requirements and data transformation needs means matching what the business must achieve to the operational strengths of Google Cloud tools. The analysis looks at data volume and scale, processing speed and latency, ease of integration, and budget limits including time to market. Dataflow, Dataproc, Cloud Data Fusion, Cloud Composer, and Dataform are part of a larger group of pipeline and orchestration options for different combinations of needs.
Processing speed requirements describe how fast transformed data must be ready and whether very low latency is truly required. Dataflow (opens in a new tab) is described as a graph-based execution system that can be much more performant for simulations such as Monte Carlo simulations. HSBC uses Dataflow to run risk calculations 16 times faster compared to its previous approach, which shows how the execution model directly affects speed for heavy calculations.
Conversations with Google Cloud customers show that the 80/20 Pareto principle applies to performance requirements for markets and trading applications. More than 80% of trading applications do not need extremely low latency, but they gain significant benefits from the resilience, security, and elasticity of the cloud. The remaining applications, less than 20%, need low latency of less than a millisecond, plus determinism and fairness in message delivery, and they conventionally run in rigid and expensive colocated facilities.
Exam tip: More than 80% of trading applications do not need extremely low latency, while less than 20% need less than a millisecond with determinism and fairness.
Data volume and elasticity requirements describe how much data must be processed and whether capacity must grow and shrink on demand. BigQuery is a fully managed, petabyte-scale, and cost-effective analytics data warehouse that lets teams run analytics over vast amounts of data in near real time. Pub/Sub is an asynchronous messaging tool for data integration pipelines that is designed to ingest and distribute data like server events and user interactions, and it can also be used for parallel processing and data streaming from Internet of Things (IoT) devices.
Dataproc, described as Managed Service for Apache Spark, lets teams provision Apache Hadoop clusters and connect to underlying analytic data stores. For high-performance computing workloads, running in the cloud or bursting on-premises workloads to the cloud can provide higher elasticity, improved security posture, and extensive monitoring and governance capabilities. For certain numerical modeling use cases like simulation of investment strategies or XVA modeling, combining Kubernetes with Kueue might offer a more powerful solution.
Integration ease describes how simply a transformation tool connects sources, microservices, and analytic stores into one working pipeline. Pub/Sub supports integration between decoupled microservices that only need to be activated when certain events occur, and data can flow from Pub/Sub through Dataflow into BigQuery for streaming analysis. Eventarc is an event-driven tool that manages the flow of state changes throughout a data pipeline, with use cases including automated error remediation, resource labeling, and image retouching.
Cloud Composer, listed as Managed Service for Apache Airflow, is a managed workflow orchestration service for coordinating pipeline steps. BigQuery sharing can be used to build an analysis-ready data-as-a-service (DaaS) platform, so data flows from central datasets to many consumers without rebuilding separate integration points. Additional pipeline and scheduling options named together for these workloads include:
Budget analysis compares the cost of running and changing the pipeline against the business value of faster delivery and clearer costs. Adopting a data-as-a-service approach supports faster time to market and cost transparency, especially for organizations that grow through mergers and acquisitions and must manage duplicate data vendors, licenses, and integration points. Cloud elasticity helps avoid paying for rigid and expensive colocated capacity when more than 80% of applications do not need sub-millisecond latency.
Future-proofing also affects budget because proprietary technologies such as custom pub-sub systems and message brokers have become commodities through open source and cloud services. BidFX, a foreign exchange multi-dealer platform, uses the cloud to launch new products quickly and to significantly increase availability and footprint without increasing resources. Choosing managed and cloud-based transformation options reduces the need to maintain custom infrastructure while keeping resilience, security, and elasticity on the cloud side of the boundary.
Assessing business requirements and tool compatibility is the process of matching what a business needs from its data to the Google Cloud tool that can deliver it with the least friction. A business requirement describes the data volume, speed, skills, budget, and systems involved, and tool compatibility describes how well a service fits those needs. This match prevents overpaying for power that is not needed or choosing a tool that cannot scale or connect.
Clarifying business requirements means writing down what the pipeline must do before comparing services. The team identifies the data sources and destinations, whether data arrives in batches or as a continuous stream, and how quickly results are needed. The team also notes who will build and maintain the pipeline, including existing Spark or Hadoop code, SQL skills, need for a visual interface, and rules for security and governance. Clear requirements make the later choice between processing, integration, and orchestration tools straightforward.
Evaluating tools on common criteria means checking each candidate against the same four business measures. Scalability asks whether the service grows automatically with data volume and concurrent jobs without manual resizing. Cost-efficiency asks whether the team pays only for what it uses and avoids idle clusters or heavy operations work. Ease of integration asks how simply the tool connects to sources, to BigQuery, and to other Google Cloud services. Performance asks whether the tool meets latency and throughput goals for batch, streaming, or SQL-based transforms.
These four criteria work together during selection:
A tool that scores well on all four for one use case may score poorly for another, so the criteria must be applied to the specific workload.
Selecting a compatible transformation tool means choosing the service whose strengths line up with the clarified requirements. Dataflow (opens in a new tab) is a serverless, graph-based execution system for batch and streaming data processing that scales automatically. Dataproc, the Managed Service for Apache Spark, runs existing Spark and Hadoop workloads without managing the underlying infrastructure details. Cloud Data Fusion provides visual integration for building extract, transform, load (ETL) pipelines, Cloud Composer orchestrates workflows across systems, and Dataform develops SQL-based transformations inside BigQuery.
A decision tree starting from a data transformation workload, branching on whether the job needs orchestration, existing Spark or Hadoop code, SQL in BigQuery, a visual interface, or continuous parallel processing to recommend Cloud Composer, Dataproc, Dataform, Cloud Data Fusion, or Dataflow respectively.
| Tool | Primary use | Choose when |
|---|---|---|
| Dataflow | Serverless batch and streaming processing | Data flows continuously or in large parallel batches and must scale automatically |
| Dataproc | Managed Apache Spark and Hadoop | The team has existing Spark code and needs a managed Spark platform |
| Cloud Data Fusion | Visual data integration | Builders prefer a visual interface with prebuilt connectors for ETL |
| Cloud Composer | Workflow orchestration | The work coordinates scheduled tasks and dependencies across multiple systems |
| Dataform | SQL transformations in BigQuery | Transformations are defined in SQL and run directly in the warehouse |
Data flows from sources through the chosen processing or integration tool to the destination such as BigQuery, and Cloud Composer can sit above that flow to schedule and depend on each step in order. A graph-based workload such as large-scale simulations can run much faster on Dataflow, while routine warehouse modeling fits naturally in Dataform SQL.
Exam tip: Choose Dataflow for auto-scaling batch and streaming transforms, and choose Dataform when the transform is SQL that runs inside BigQuery.
Dataproc provisions managed Hadoop and Spark clusters that connect to separate analytic data stores, Dataflow provides serverless graph-based batch and streaming execution that scales automatically, and Cloud Composer provides managed workflow orchestration that schedules and coordinates tasks running in other systems without running transformation compute itself. Choose Dataproc when you have existing Spark or Hadoop code, Dataflow when data flows continuously or in large parallel batches such as simulations, and Cloud Composer when the work coordinates scheduled tasks and dependencies across multiple systems.
Use Dataform when transformations are defined in SQL and run directly inside BigQuery. Dataform fits routine warehouse modeling and getting started with transformed data workflows, while Dataflow fits auto-scaling batch and streaming transforms and Dataproc fits existing Spark and Hadoop workloads on managed clusters.
More than 80% of trading applications do not need extremely low latency and gain significant benefits from the resilience, security, and elasticity of the cloud, while less than 20% need less than a millisecond with determinism and fairness and conventionally run in rigid and expensive colocated facilities. For heavy calculations, Dataflow as a graph-based execution system can be much more performant for simulations such as Monte Carlo simulations, and HSBC used Dataflow to run risk calculations 16 times faster than its previous approach.
A financial analytics company needs to modernize its data pipeline to process high-volume, real-time streaming transaction records. The solution must support continuous stream ingestion, execute in-flight transformations with sub-second latency, scale automatically during unpredictable traffic spikes, and load the enriched output into BigQuery without requiring manual cluster management.
Which Google Cloud service should the company select to meet these business and technical requirements?
Last updated