I have a big tasks which can be seperated into several small tasks.
For Example,
The Tasks load data from input file, based on those data, calculate some results, then dump into output file.
each task handle different file, wont have any conflicits.
Assume I have 100 small tasks, each task will cost 1s, if i run them in one machine by single thread, it will cost me 100s.
I tried multi-threading, i have 20 cores, even i used thread_pool, it only can reduce time to 50s( and 3 cores-20 cores make no difference), I spend a lot of time on multi-threading, but can make it better. I think it may be caused by false-sharing or some other complex problems, i cant understand yet.
So, I am thinking if there is some easy way i can use to assign these problems into different machines in my ethernet.
In my imagination, the good ways is:
every machine installed same software, and in my control machine, i just run a command to assign tasks to those machine. they compute, when finished, the control machine collect result back, so, it speed up, and got same result with single machine.
if i have 10 machines, i will expect the time can be reduced to 10s(10% of 100s).
Is there any easy ways can be used for a rockie?