An enterprise is establishing a data analytics platform on Google Cloud and needs to assign the appropriate processing engine for three distinct pipeline workloads: Workload 1: Migrating an existing portfolio of legacy Apache Spark jobs and Hadoop dependencies with minimal code changes while leveraging ephemeral clusters to optimize compute costs. Workload 2: Processing real-time unbounded event streams alongside bounded batch datasets using a fully managed serverless runtime with exactly-once processing semantics. Workload 3: Enabling business analysts to visually design, deploy, and manage ETL/ELT pipelines using a point-and-click graphical interface backed by open-source CDAP. Which service mapping should the enterprise select for Workload 1, Workload 2, and Workload 3, respectively?

Workload 1: Cloud Dataproc; Workload 2: Cloud Dataflow; Workload 3: Cloud Data Fusion: ### What the Solution Is This architecture pairs **Cloud Dataproc**, **Cloud Dataflow**, and **Cloud Data Fusion** with their optimal operational paradigms across open-source migration, serverless stream/batch processing, and visual data integration. ### How It Meets Each Requirement - **Workload 1 (Cloud Dataproc)**: Dataproc is Google Cloud's managed **Apache Spark** and **Apache Hadoop** service. It enables lift-and-shift migration of legacy jobs without rewriting business logic, while supporting **ephemeral clusters** configured and deleted on-demand for individual batch jobs to reduce cost. - **Workload 2 (Cloud Dataflow)**: Dataflow provides a fully managed, **serverless execution environment** for **Apache Beam** pipelines. It natively supports unified batch and streaming models, dynamic auto-scaling of worker pools, and automated exactly-once processing guarantees without requiring cluster provisioning. - **Workload 3 (Cloud Data Fusion)**: Built on the open-source **CDAP** (Cask Data Application Platform) framework, Cloud Data Fusion provides a web-based, graphical point-and-click interface that lets non-programmers and analysts build, configure, and monitor complex data pipelines. ### Key Advantages and Capabilities - **Minimized Migration Risk**: Dataproc runs native Spark/Hadoop binaries and ecosystem tools like Hive and Flink directly. - **Zero Infrastructure Overhead**: Dataflow provisions, manages, autoscales, and re-allocates worker compute resources autonomously. - **Democratized ETL/ELT**: Cloud Data Fusion offers an extensive visual library of connectors, transformations, and sinks for rapid integration. - **Cost Efficiency**: Ephemeral clusters in Dataproc ensure compute is billed strictly for execution time rather than persistent idle state. ### Why This Is Optimal This mapping assigns each execution engine to its core design strength, avoiding costly pipeline rewrites for Spark workloads while giving analysts a no-code visual workbench and streaming workloads a self-optimizing serverless backbone.

Select Dataproc Dataflow Data Fusion Workloads | professional-cloud-data-engineer

AdvancedProfessional Cloud Data Engineer

An enterprise e-commerce platform is designing a real-time event ingestion architecture on Google Cloud for order processing and inventory updates. The system must satisfy the following architectural and operational requirements: Ordering Guarantees: Ensure strict in-order message delivery per merchant account (merchant_id). Operational Overhead: Eliminate cluster provisioning, broker patching, ZooKeeper/KRaft maintenance, and manual partition capacity planning. Elastic Scalability: Automatically scale ingestion throughput up or down to handle sudden traffic spikes without manual partition rebalancing. Delivery Semantics: Provide built-in retry mechanisms, dead-letter support, and granular message acknowledgment. Which messaging framework should the enterprise implement?

Identifying the services (e.g., Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Apache Spark, Hadoop ecosystem, and Apache Kafka)

AdvancedProfessional Cloud Data Engineer

An enterprise financial organization is designing an automated ingestion and processing architecture to stream high-volume transaction records from Pub/Sub into BigQuery. The architecture must satisfy the following technical requirements: Ingest records directly into BigQuery storage while strictly guaranteeing exactly-once processing semantics. Isolate and extract individual malformed or schema-mismatched rows into a dead-letter collection without failing the entire streaming pipeline. Maintain high throughput without relying on legacy streaming buffer inserts. Which ingestion pathway and processing approach should the organization implement?

AdvancedProfessional Cloud Data Engineer

A data engineering team is architecting two interdependent workflows in Google Cloud Composer: an upstream ingestion workflow (DAG_Ingest) and a downstream transformation workflow (DAG_Analytics). The team must fulfill the following architectural and operational requirements: DAG_Analytics must begin processing only after DAG_Ingest completes and publishes an event artifact notification. Prevent automated scheduling of historical backfill runs when DAG_Analytics is uploaded or unpaused. Prevent resource starvation of core Airflow components (schedulers and web servers) caused by CPU- and memory-heavy transformation tasks. Ensure robust task failure recovery without whole-workflow failure. Which combination of Airflow configurations, operators, and task relationships should the team implement?

AdvancedProfessional Cloud Data Engineer

A financial analytics firm is designing an infrastructure architecture on Google Cloud to handle two distinct analytical workloads: Workload A (Interactive Analytics): Financial analysts execute ad-hoc, low-latency SQL queries against multi-terabyte curated datasets to feed near-real-time executive dashboards, requiring consistent query response times and instant data recovery from accidental modifications within a 7-day window. Workload B (Scheduled Batch Processing): A nightly regulatory compliance pipeline executes complex, multistage Directed Acyclic Graph (DAG) transformations across large volumes of raw point-of-sale (POS) files in Cloud Storage, requiring custom Apache Spark transformations, cost-optimized ephemeral infrastructure, and automated orchestration. Which architectural pattern best meets the latency, fault tolerance, and operational requirements for both workloads?

AdvancedProfessional Cloud Data Engineer

A data engineering team operates a critical streaming pipeline running on Google Cloud. During unexpected ingestion spikes, the team needs to be notified immediately if worker compute resources experience severe CPU exhaustion while processing latency simultaneously increases on that specific workload. The alerting policy must trigger only when both conditions are satisfied concurrently for the exact same underlying resource, preventing false alerts caused by unrelated compute instances. How should the data engineer configure the alerting policy combiner in Cloud Monitoring?

IntermediateProfessional Cloud Architect

An enterprise runs a critical web application on Compute Engine instances distributed across multiple zones, consisting of both Debian Linux and Windows Server virtual machines (VMs). To maintain compliance and patch vulnerabilities, the security team requires an automated patching solution that satisfies these criteria: Automatically execute patch updates on a recurring schedule targeting only critical and security updates. Roll out updates zone-by-zone to minimize blast radius and avoid downtime. Ensure no more than 20% of instances in any given zone are disrupted at any time during the patching process. Automatically halt the rollout across remaining zones if the disruption threshold is exceeded due to failed patch operations. Run custom pre-patch backup scripts and post-patch application health verification scripts on each instance. Which solution should you implement to satisfy these requirements?

Professional Cloud Data Engineer

Select Dataproc Dataflow Data Fusion Workloads

Related questions