Optimize Dataflow Storage Locality Egress Flexrs

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise processes terabytes of analytics data daily using batch Apache Beam pipelines on Dataflow. The primary input dataset currently resides in a Cloud Storage multi-region bucket located in the us multi-region, while the pipeline's temporary and staging files (gcpTempLocation) are written to a bucket in us-east1. The Dataflow worker compute instances are provisioned to execute in us-central1.

The cloud operations team reports substantial inter-region network egress charges and increased execution wall time during pipeline shuffle and staging phases. Additionally, the batch workloads are non-time-sensitive.

Which architectural strategy should the data engineering team implement to minimize network egress costs, improve I/O performance, and optimize economic efficiency?

Change the default storage class of the multi-region bucket to Archive storage and keep worker VMs running continuously in us-central1 across all availability zones.

Retain the multi-region storage and cross-region worker configuration, but deploy a cross-region internal Application Load Balancer to dynamically route intermediate shuffle traffic across regions.

Migrate the input data and configure the temporary staging bucket in a regional Cloud Storage bucket located in us-central1, execute Dataflow workers in us-central1, and enable Flexible Resource Scheduling (FlexRS).

Maintain the worker execution in us-central1 and staging in us-east1, but increase the worker VM thread count and maximum workers to mask network transfer latency.

Professional Cloud Data Engineer

Optimize Dataflow Storage Locality Egress Flexrs

Related questions