A data engineering team maintains a nightly batch Apache Beam pipeline running on Cloud Dataflow that aggregates multi-terabyte transactional logs. The pipeline groups data by merchant ID using a GroupByKey transform and writes aggregated metrics into BigQuery. Monitoring shows the following bottlenecks during job execution: The GroupByKey transform suffers from severe stragglers due to extreme key skew caused by a few high-volume merchants, leaving most worker vCPUs idle while a single worker runs out of memory. Worker nodes encounter severe local Persistent Disk I/O bottlenecks and disk thrashing during intermediate shuffle phases. Compute costs for the nightly batch window are exceeding the allocated budget. Which combination of optimization strategies should the team implement to resolve the performance bottlenecks and minimize execution costs?

Replace the GroupByKey transform with Combine.perKey using an associative and commutative CombineFn, enable the service-based Dataflow Shuffle, and configure FlexRS with Spot VMs.: ### What this solution is This solution addresses key skew, shuffle infrastructure bottlenecks, and compute pricing simultaneously by combining algorithmic pipeline improvements with managed **Dataflow Shuffle** service capabilities and **Flexible Resource Scheduling (FlexRS)**. ### How it meets the requirements - **Mitigating Data Skew:** Replacing `GroupByKey` with `Combine.perKey` allows Apache Beam to perform **combiner lifting**. Intermediate partial aggregations occur in-memory on the mapper workers before the data is shuffled across the network. Because the combining function is associative and commutative, records for skewed merchant keys are collapsed locally into compact partial results, eliminating hot-key stragglers. - **Eliminating Disk I/O Bottlenecks:** Enabling service-based **Dataflow Shuffle** moves the shuffle operations off worker-local persistent disks and into a dedicated, managed Cloud Dataflow shuffle infrastructure backend. This prevents local disk thrashing and reduces worker CPU and memory overhead. - **Minimizing Compute Costs:** Leveraging **FlexRS** with **Spot VMs** (or preemptible VMs) allocates lower-cost compute capacity for non-time-critical nightly batch workloads, significantly reducing the total pipeline execution cost. ### Key advantages - Reduces network and disk data transfer volume exponentially through local partial combining. - Eliminates the need to over-provision large worker Persistent Disks to accommodate shuffle operations. - Provides seamless autoscaling without disk serialization overhead. - Lowers vCPU and memory billing hours by utilizing discounted compute instances. ### Why this is optimal This approach resolves the root cause of the data skew algorithmically while delegating the heavy shuffle I/O to Google's optimized backend infrastructure, yielding the highest throughput at the lowest operational cost.

Dataflow Batch Optimization Skew Shuffle Costs | professional-cloud-data-engineer

Dataflow Batch Optimization Skew Shuffle Costs

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team maintains a nightly batch Apache Beam pipeline running on Cloud Dataflow that aggregates multi-terabyte transactional logs. The pipeline groups data by merchant ID using a GroupByKey transform and writes aggregated metrics into BigQuery.

Monitoring shows the following bottlenecks during job execution:

The GroupByKey transform suffers from severe stragglers due to extreme key skew caused by a few high-volume merchants, leaving most worker vCPUs idle while a single worker runs out of memory.
Worker nodes encounter severe local Persistent Disk I/O bottlenecks and disk thrashing during intermediate shuffle phases.
Compute costs for the nightly batch window are exceeding the allocated budget.

Which combination of optimization strategies should the team implement to resolve the performance bottlenecks and minimize execution costs?

Increase the worker Persistent Disk size to 1 TB SSD per worker, disable autoscaling by setting maxNumWorkers equal to numWorkers, and assign high-memory n2-highmem-16 worker machines.

Apply a Reshuffle transform immediately before GroupByKey, set the worker disk type to pd-standard HDD, and run the job on standard on-demand compute instances.

Add a Window.into transform with fixed one-second windows prior to GroupByKey, configure custom Cloud Monitoring alerts on vCPU usage, and increase the maximum autoscaling workers.

Replace the GroupByKey transform with Combine.perKey using an associative and commutative CombineFn, enable the service-based Dataflow Shuffle, and configure FlexRS with Spot VMs.

Professional Cloud Data Engineer

Dataflow Batch Optimization Skew Shuffle Costs

Related questions