Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team runs a PySpark batch transformation pipeline on an existing Google Cloud Dataproc cluster. The job reads several terabytes of raw logs stored in Google Cloud Storage and joins this massive dataset with a 15 MB dimension table containing account metadata.
During job execution, the team observes two major performance bottlenecks:
How should the team optimize this pipeline for a specific job submission without altering cluster-wide settings?
--properties=spark.sql.sources.gcs.grpc.enabled=true, and set spark.sql.autoBroadcastJoinThreshold=-1 to force bucketed sort-merge joins.This configuration attempts to use a non-existent Spark SQL property (spark.sql.sources.gcs.grpc.enabled) and sets spark.sql.autoBroadcastJoinThreshold to -1, which explicitly disables broadcast joins in Spark SQL.
fs.gs.* (specifically spark.hadoop.fs.gs.client.type=STORAGE_CLIENT), not internal spark.sql.sources.* flags.spark.sql.autoBroadcastJoinThreshold to -1 forces Spark to use a Sort-Merge Join or Shuffle-Hash Join even when tables are tiny, guaranteeing that massive shuffle network traffic will occur between cluster workers.Both datasets were extremely large (hundreds of gigabytes each), pre-bucketed, sorted by the join key, and exceeded available executor memory, making broadcast joins impossible.
--properties=core:fs.gs.client.type=STORAGE_CLIENT to enable gRPC, and invoke repartition() on the dimension table before joining.The property core:fs.gs.client.type=STORAGE_CLIENT is intended for cluster creation rather than Spark job submission. Calling repartition() on the dimension table increases or redistributes its partitions across the cluster.
spark.hadoop. prefix (e.g., spark.hadoop.fs.gs.client.type=STORAGE_CLIENT) to inject Hadoop filesystem configurations directly into the Spark runtime environment. Using core: at job submission is ignored by Spark.repartition() on a small 15 MB table does not prevent a shuffle during the subsequent join; in fact, it adds an extra unnecessary shuffle stage before the join occurs without resolving the standard sort-merge join shuffle bottleneck.core: property prefixes are parsed by Dataproc during cluster initialization (gcloud dataproc clusters create), not dynamically during individual PySpark job runs.repartition() forces a full data shuffle across executor nodes, which adds overhead rather than eliminating it.The goal was to configure default gRPC access globally at cluster creation time (gcloud dataproc clusters create --properties=core:fs.gs.client.type=STORAGE_CLIENT) and balance partitions for two equally massive tables suffering from partition skew.
--properties=spark.hadoop.fs.gs.client.type=HTTP_CLIENT, and persist both the raw logs and dimension table in memory using cache().Setting spark.hadoop.fs.gs.client.type=HTTP_CLIENT forces the connector to use the standard JSON API over HTTP/1.1 instead of gRPC. Applying cache() materializes and persists both DataFrames in executor memory after their initial evaluation.
HTTP_CLIENT is the legacy default client type; it does not leverage high-performance gRPC streaming, missing the opportunity to accelerate multi-terabyte Cloud Storage reads.OutOfMemoryError failures.cache() is only beneficial for datasets that are reused multiple times across iterative transformations or actions; caching a massive dataset for a single join operation wastes memory.The dataset fit comfortably within cluster memory, was repeatedly queried across multiple iterative machine learning loops, and required fallback to standard HTTP proxies incompatible with gRPC.
--properties=spark.hadoop.fs.gs.client.type=STORAGE_CLIENT to enable gRPC for Cloud Storage reads, and apply a broadcast join (broadcast()) on the dimension table in the PySpark logic.This solution combines connector-level transport optimization with Spark execution plan tuning. Enabling the gRPC storage client configures the Cloud Storage connector to use Google's high-performance gRPC API instead of the standard JSON HTTP API for read operations. In parallel, wrapping the small lookup dataset in a broadcast() hint instructs Spark to perform a Broadcast Hash Join (BHJ) instead of a distributed Sort-Merge Join (SMJ).
--properties=spark.hadoop.fs.gs.client.type=STORAGE_CLIENT at job submission enables gRPC read requests on the Cloud Storage connector specifically for this job without requiring cluster recreation or cluster-wide restarts.spark.hadoop. property prefixes during gcloud dataproc jobs submit ensures the configuration applies strictly to this execution without modifying the underlying Dataproc cluster properties.Alternative join strategies like sort-merge or shuffle-hash require partitioning both datasets by join key across all worker nodes. For asymmetric joins between large fact tables and tiny dimension tables, broadcast joins eliminate shuffle entirely. Pairing this with gRPC maximizes end-to-end data throughput from storage ingestion to final processing.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated