If I use the same deterministic expression twice or more in a query, will spark know to optimize and not-recalculate it?
I saw this question before, but the answer is to check the plan, and I don't really understand the plan to answer my question
Given this dataframe:
df = spark.createDataFrame([{'word': 'hello'}, {'word': 'goodbye'}])
+-------+
| word|
+-------+
| hello|
|goodbye|
+-------+
Let's say I want to add a column, that concatenate -world to the word column, but only if the result is hello-world
concatenated = F.concat_ws('-', 'word', F.lit('world'))
df.withColumn('result', F.when(concatenated == F.lit('hello-world'), concatenated))
The plan is:
== Parsed Logical Plan ==
'Project [word#5678, CASE WHEN (concat_ws(-, 'word, world) = hello-world) THEN concat_ws(-, 'word, world) END AS result#5680]
+- LogicalRDD [word#5678], false
== Analyzed Logical Plan ==
word: string, result: string
Project [word#5678, CASE WHEN (concat_ws(-, word#5678, world) = hello-world) THEN concat_ws(-, word#5678, world) END AS result#5680]
+- LogicalRDD [word#5678], false
== Optimized Logical Plan ==
Project [word#5678, CASE WHEN (concat_ws(-, word#5678, world) = hello-world) THEN concat_ws(-, word#5678, world) END AS result#5680]
+- LogicalRDD [word#5678], false
== Physical Plan ==
*(1) Project [word#5678, CASE WHEN (concat_ws(-, word#5678, world) = hello-world) THEN concat_ws(-, word#5678, world) END AS result#5680]
+- *(1) Scan ExistingRDD[word#5678]
So I can't really figure out if concat_ws(-, word#5678, world) gets recalculated
Another example that is much more complex Add another column, that has all of the numbers doubled, where the doubled number is > 3, but only if the size of the resulting list is over 2
df = spark.createDataFrame([{'numbers': [1,2,3,4]}, {'numbers': [1,2]}])
filterd_list = F.filter(
F.transform('numbers', lambda x: x * 2),
lambda j: j > 3
)
df.withColumn('abc',
F.when(
F.size(
filterd_list
) >= 3,
filterd_list
)
).explain(True)
== Parsed Logical Plan ==
'Project [numbers#5648, CASE WHEN (size(filter(transform('numbers, lambdafunction((lambda 'x_52 * 2), lambda 'x_52, false)), lambdafunction((lambda 'x_53 > 3), lambda 'x_53, false)), true) >= 3) THEN filter(transform('numbers, lambdafunction((lambda 'x_52 * 2), lambda 'x_52, false)), lambdafunction((lambda 'x_53 > 3), lambda 'x_53, false)) END AS abc#5650]
+- LogicalRDD [numbers#5648], false
== Analyzed Logical Plan ==
numbers: array<bigint>, abc: array<bigint>
Project [numbers#5648, CASE WHEN (size(filter(transform(numbers#5648, lambdafunction((lambda x_52#5651L * cast(2 as bigint)), lambda x_52#5651L, false)), lambdafunction((lambda x_53#5653L > cast(3 as bigint)), lambda x_53#5653L, false)), true) >= 3) THEN filter(transform(numbers#5648, lambdafunction((lambda x_52#5652L * cast(2 as bigint)), lambda x_52#5652L, false)), lambdafunction((lambda x_53#5654L > cast(3 as bigint)), lambda x_53#5654L, false)) END AS abc#5650]
+- LogicalRDD [numbers#5648], false
== Optimized Logical Plan ==
Project [numbers#5648, CASE WHEN (size(filter(transform(numbers#5648, lambdafunction((lambda x_52#5651L * 2), lambda x_52#5651L, false)), lambdafunction((lambda x_53#5653L > 3), lambda x_53#5653L, false)), true) >= 3) THEN filter(transform(numbers#5648, lambdafunction((lambda x_52#5652L * 2), lambda x_52#5652L, false)), lambdafunction((lambda x_53#5654L > 3), lambda x_53#5654L, false)) END AS abc#5650]
+- LogicalRDD [numbers#5648], false
== Physical Plan ==
*(1) Project [numbers#5648, CASE WHEN (size(filter(transform(numbers#5648, lambdafunction((lambda x_52#5651L * 2), lambda x_52#5651L, false)), lambdafunction((lambda x_53#5653L > 3), lambda x_53#5653L, false)), true) >= 3) THEN filter(transform(numbers#5648, lambdafunction((lambda x_52#5652L * 2), lambda x_52#5652L, false)), lambdafunction((lambda x_53#5654L > 3), lambda x_53#5654L, false)) END AS abc#5650]
+- *(1) Scan ExistingRDD[numbers#5648]
These are just some made-up examples, but I come across this a lot dealing with structs and lists, sometimes with several more repeated expressions.
If the answer is the it does get re-calculated, is the way to overcome this is to use several withColumn expressions, dropping the intermediate ones in the end?