Can Spark change number of executors during runtime?
Example, In an Action(Job), Stage 1 runs with 4 executor * 5 partitions per executor = 20 partitions in parallel.
If I repartition with .repartition(100), Which is Stage 2 now (because of repartition shuffle), Can in any case Spark increases from 4 executors to 5 executors (or more)?
If I cache some data in Stage 1 which was running with 4 executors * 5 partition per executor = 20 partitions, then the cached data must be in the RAM of 4 machines. If I repartition with .repartition(2), in this case, definitely there will be <= 2 executor machines involved. Will spark shuffle my cached data to the active tasks?