How to use parallelism in Julia to speed up combinatorial checking?

user_11391560 0
Viewed 39

Given a bunch of nested for loops, I wish to find the best combination of values according to some metric, and I want to keep track of the best combination and the best result.

I gave an example (below) of how a very basic version of what I want would look like sequentially. However, I realize that this problem can be easily parallelized, especially if I can generate all combinations and then run the check function for each combination at the same time.

In my specific use case, the check function takes the majority of the time, so it is a clear bottleneck in my algorithm. My intuition is that if I were to create an array of tasks, I could run these tasks all at once and achieve a considerable speedup.

Given my requirements, how would I implement this in Julia? In Java, I know I could use something like ForkJoinPool and add a slew of tasks that way, but I am unsure of the Julia equivalent.

function check(parameters)
    return value(parameters)
end

bestCheck = 0 
bestParameters = []
for a in 1:30
   for b in 1:50
       for c in 1:90
           temp = check(a,b,c)
           if temp > bestCheck
               bestCheck = temp
               bestParameters = [a b c]
           end
        end
    end
end

println(bestCheck)
println(bestParameters)
julia combinations parallel-processing java
1 Answers
using Distributed
addprocs(16) # whatever CPU / vCPU you have
@everywhere function check(a,b, c)
    a^2 - 2b - 45c + c^2
end

best = @distributed max for (a,b,c) in vec(collect(Iterators.product(1:30, 1:50, 1:90)))
    (check(a,b,c), a,b,c)
end

julia> best
(4948, 30, 1, 90)

As you can see in Julia this kind of code i short. Distributed is an equivalent of multiprocessing in Python but much more powerful. @everywhere defines function in all nodes around the cluster. @distributed in this scenario uses an aggregator function which is max. We distribute computing of a Cartesian Product of all parameters - in distributed scenario this is usually better than nested loops. Also note that the order of parameter matters when computation times depend on parameter values. The loop yields a tuple that goes to the max aggregator function.

user_99577100
Related