Optimize Dataproc Spark Grpc Broadcast Join

What the solution is

This solution combines connector-level transport optimization with Spark execution plan tuning. Enabling the gRPC storage client configures the Cloud Storage connector to use Google's high-performance gRPC API instead of the standard JSON HTTP API for read operations. In parallel, wrapping the small lookup dataset in a broadcast() hint instructs Spark to perform a Broadcast Hash Join (BHJ) instead of a distributed Sort-Merge Join (SMJ).

How it meets the requirements

• Cloud Storage read performance: Passing --properties=spark.hadoop.fs.gs.client.type=STORAGE_CLIENT at job submission enables gRPC read requests on the Cloud Storage connector specifically for this job without requiring cluster recreation or cluster-wide restarts.
• Eliminating shuffle overhead: The 15 MB metadata table is sufficiently small to fit in worker memory. Broadcasting it distributes a local copy of the dimension table to every executor, enabling each mapper to perform joins locally against partition data and completely eliminating the cross-network shuffle stage.
• Job-level isolation: Using spark.hadoop. property prefixes during gcloud dataproc jobs submit ensures the configuration applies strictly to this execution without modifying the underlying Dataproc cluster properties.

Key advantages

• Higher read throughput: gRPC utilizes HTTP/2 multiplexing and streaming, significantly reducing latency and CPU overhead on high-throughput Cloud Storage reads.
• Zero shuffle data transfer: Broadcast joins bypass wide transformations, saving substantial network I/O and disk serialization overhead across executors.
• Targeted deployment: Allows fine-tuned optimization on a per-job basis without impacting other workloads sharing the Dataproc cluster.

Why this is optimal

Alternative join strategies like sort-merge or shuffle-hash require partitioning both datasets by join key across all worker nodes. For asymmetric joins between large fact tables and tiny dimension tables, broadcast joins eliminate shuffle entirely. Pairing this with gRPC maximizes end-to-end data throughput from storage ingestion to final processing.