Home / Data Engineer Associate / Objective 3.5

Domain 3: Data Transformation and Modeling · 22% of the exam · Full exam outline

3.5 Understand the basic tuning parameters (spark.sql.shuffle.partitions, spark.default.parallelism, spark.executor/driver.memory, spark.sql.autoBroadcastJoinThreshold) and re-measure the performance.

Practice questions

Question 1easy

Which Spark property, when set to auto, lets Databricks choose the number of shuffle partitions for joins and aggregations based on the query plan and input data size?

Included in Master: master every domain.See plans
Question 2easy

A team migrating Spark jobs to Databricks classic compute plans to copy spark.executor.memory=6g from its old cluster into every job's compute definition. What does Databricks recommend?

Included in Master: master every domain.See plans