I was wondering if there is any way to simulate some workers in my local machine, I do know all spark can be set up locally and all can be fully tested. However, I was wondering if it is possible to actually simulate workers, with the idea to see how the repartition of work, loads and the DAG dynamics work.
I also can think about ways that can help me out, for example debugging and tracing data transformations. What I want to do is to develop an optimal way do my program in a testing fashion, I do not wanna rely in the big theory behind shuffling and expensive operations. Or are we dommed to try to test this up in a trial and error in real clusterS? thanks!