I've written a program that will run on 10,000 different physical nodes in a distributed fashion where each node is its own computing device and am trying to simulate it before I move onto the hardware side of my project. I've been researching this for quite some time and am unable to find anything. I can't spin up 10,000 threads and I can't spin up 10,000 processes (which would be ideal as that would be the closest to real-life). Any ideas on how to get this done?
1 Answers
Creating many system threads/processes is very expensive as it would not only slow to create/destroy but also put a lot of pressure on the scheduler that is not meant to operate on so many tasks (even though it can be quite Ok for the scheduler if only a tiny fraction of the tasks is active). Not to mention there are system limitation (that can be tweak if you have admin rights on the target machine).
One efficient solution is to use green threads like fibers. Fibers are basically a user-level thread with its own stack and registers. The scheduling of fiber is done cooperatively (as opposed to pre-emption for system threads). This means that waiting fibers must call a yield function causing a context-switch to another fiber on the same thread. In order to maximize performance, it is better to use multiple threads with multiple fibers in each thread (so for the application to run on multiple cores). There is no limit to the number of fibers that can be created except the amount of memory needed for their stack. The default stack size if of few Mo by default which mean that 10K threads or processes would require dozens of GiB of stack which is not reasonable. The size of the stack can be tuned so to take only dozens of KiB. However, this means that the executed operation should not make an intensive use of the stack (ie. no deep recursion, nor stack-allocated arrays).
Alternatively, you may be interested in academic projects like Distem or SimGrid that are related to this subject.
Related