Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team maintains a nightly batch Apache Beam pipeline running on Cloud Dataflow that aggregates multi-terabyte transactional logs. The pipeline groups data by merchant ID using a GroupByKey transform and writes aggregated metrics into BigQuery.
Monitoring shows the following bottlenecks during job execution:
GroupByKey transform suffers from severe stragglers due to extreme key skew caused by a few high-volume merchants, leaving most worker vCPUs idle while a single worker runs out of memory.Which combination of optimization strategies should the team implement to resolve the performance bottlenecks and minimize execution costs?
This solution addresses key skew, shuffle infrastructure bottlenecks, and compute pricing simultaneously by combining algorithmic pipeline improvements with managed Dataflow Shuffle service capabilities and Flexible Resource Scheduling (FlexRS).
GroupByKey with Combine.perKey allows Apache Beam to perform combiner lifting. Intermediate partial aggregations occur in-memory on the mapper workers before the data is shuffled across the network. Because the combining function is associative and commutative, records for skewed merchant keys are collapsed locally into compact partial results, eliminating hot-key stragglers.This approach resolves the root cause of the data skew algorithmically while delegating the heavy shuffle I/O to Google's optimized backend infrastructure, yielding the highest throughput at the lowest operational cost.
This option attempts to resolve memory and I/O bottlenecks by vertically scaling worker instances and provisioning static, over-allocated hardware resources without modifying the underlying pipeline code.
GroupByKey. A single worker must still process an entire unbalanced key group sequentially, meaning the pipeline will still suffer from straggler tasks and unutilized cluster capacity.n2-highmem-16 instances with attached 1 TB SSD persistent disks while disabling autoscaling significantly inflates compute and storage costs, directly violating the budget requirement.The workload had perfectly uniform data distribution, strict low-latency guarantees that prohibited VM startup overhead, and an unrestricted budget.
This option introduces an explicit Reshuffle step to redistribute elements across workers and utilizes standard spinning hard disks (pd-standard) with on-demand compute instances.
Reshuffle redistributes data across worker partitions randomly, but downstream GroupByKey operations still group all elements with identical merchant keys onto the same target worker. Consequently, the skewed key will still overload a single worker instance.pd-standard) have substantially lower IOPS and throughput compared to SSDs or dedicated external shuffle services. Relying on local HDDs for shuffle exacerbates I/O bottlenecks.Reshuffle adds an unnecessary additional serialization and deserialization phase.The goal was simply to prevent fusion of non-grouped steps and the input data had no duplicate grouping keys or key skew.
This option applies fine-grained fixed time windowing to a batch pipeline, sets up monitoring alerts, and raises the autoscaling worker limit.
GroupByKey step; the straggler task remains restricted to a single worker while other autoscaled nodes sit idle.The pipeline was an unbounded streaming workload requiring real-time sub-second windowed aggregations with elastic scaling requirements.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated