Etl Vs Elt Bigquery Transformation Pushdown

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data analytics team is redesigning their batch processing architecture. Every night, multi-terabyte structured datasets from operational databases and Cloud Storage are ingested, requiring complex multi-table joins and analytical aggregations before being delivered to reporting dashboards in BigQuery.

The existing pipeline runs programmatic ETL transformations entirely on Apache Spark within Cloud Dataproc clusters. However, resource-intensive shuffle operations during large-scale joins frequently cause cluster memory bottlenecks and lead to inflated compute costs.

Which architectural approach should the team implement to optimize performance and reduce compute costs?

Scale up the Cloud Dataproc cluster worker nodes to high-memory machine types and cache intermediate joined DataFrames in HDFS

Stash intermediate raw records in Cloud Spanner and execute transformations using Spanner-to-Cloud Storage templates before loading into BigQuery

Adopt an ELT design pattern by loading raw datasets directly into BigQuery staging tables and executing heavy joins and transformations declaratively using BigQuery SQL

Migrate the batch ETL pipeline to Cloud Dataflow using the STORAGE_API_AT_LEAST_ONCE write mode to stream joined records into BigQuery

Professional Cloud Data Engineer

Etl Vs Elt Bigquery Transformation Pushdown

Related questions