professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
An enterprise data engineering team maintains an ETL pipeline built with Cloud Data Fusion running on a managed Cloud Dataproc Spark cluster. The pipeline ingests multi-terabyte datasets, performs several complex relational joins and aggregations across multiple tables, and then executes custom feature-engineering transformations in Spark before writing outputs to downstream analytics sinks.
During peak runs, the team observes severe performance degradation and high compute costs driven by massive data shuffle operations during the distributed Spark joins.
Which strategy should the data engineering team implement to optimize pipeline performance and reduce compute costs?
Transformation Pushdown is an optimization mechanism in Cloud Data Fusion that offloads supported pipeline execution stages—such as multi-table relational JOIN operations and group-by aggregations—directly to BigQuery SQL engines instead of computing them on an Apache Spark execution engine running within Cloud Dataproc. The BigQuery Storage Read API is a high-throughput, parallel data ingestion API that streams processed rows directly from BigQuery storage into distributed compute frameworks like Spark using gRPC.
StudyCerts
StudyCertsJOIN and aggregation transformations are performed serverlessly inside BigQuery, the Dataproc cluster no longer requires oversized worker nodes or large memory footprints. Organizations can substantially downsize their Dataproc cluster footprint.Pushing relational operations down into BigQuery eliminates heavy Spark shuffles while maximizing execution speed. Utilizing the BigQuery Storage Read API ensures that subsequent Spark-based feature engineering steps receive clean, partitioned streams with minimal latency.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.