What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?

spark.default.parallelism is the default number of partition set by spark which is by default 200. and if you want to increase the number of partition than you can apply the property spark.sql.shuffle.partitions to set number of partition in the spark configuration or while running spark SQL.

Normally this spark.sql.shuffle.partitions it is being used when we have a memory congestion and we see below error: spark error:java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE

so set your can allocate a partition as 256 MB per partition and that you can use to set for your processes.

also If number of partitions is near to 2000 then increase it to more than 2000. As spark applies different logic for partition < 2000 and > 2000 which will increase your code performance by decreasing the memory footprint as data default is highly compressed if >2000.

In case someone who might want to know, there's indeed a special situation when the setting spark.sql.shuffle.partitions might become invalid. When you restart a structured streaming spark application with the same checkpoint location, changing this term does not take effect. See more at https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration