Clarification on benfefits of threads useage and process

user_3281849 0
Viewed 156

I'm new in using Julia and after some courses about numeric analysis programming became a hobby of mine. I ran some tests with all my cores and did the same with threads to compare. I noticed that doing heavier computation went better with the threaded loop than with the process, But it was about the same when it came to addition. (operations were randomly selected for example) After some research its all kinda vague and I ultimately want some perspective from someone that is using the same language if it matters at all.

Some technical info: 8 physical cores, julia added vector of 16 after addprocs() and nthreads() is 16

using Distributed
addprocs()
@everywhere using SharedArrays;
@everywhere using BenchmarkTools;

function test(lim)
    r = zeros(Int64(lim / 16),Threads.nthreads())
    Threads.@threads for i in eachindex(r)
        r[Threads.threadid()] = (BigInt(i)^7 +5)%7; 
    end
    return sum(r)
end
@btime test(10^4)   # 1.178 ms (240079 allocations: 3.98 MiB)

@everywhere function test2(lim)
    a = SharedArray{Int64}(lim);
    @sync @distributed for i=1:lim
        a[i] = (BigInt(i)^7 +5)%7;
    end
    return sum(a)
end
@btime test2(10^4)  # 3.796 ms (4413 allocations: 189.02 KiB)
julia
1 Answers

Note that your loops do very different things.

  1. Int the first loop each thread keeps updating the same single cell the Array. Most likely since only a single memory cell is update in a single thread, the processor caching mechanism can be used to speed up things. On the other hand the second loop each process is updating several different memory cells and such caching is not possible.
  2. The first Array holds Float64 values while the second holds Int64 values

After correcting those things the difference gets smaller (this is on my laptop, I have only 8 threads):

julia> @btime test(10^4)
  2.781 ms (220037 allocations: 3.59 MiB)
29997

julia> @btime test2(10^4)
  4.867 ms (2145 allocations: 90.14 KiB)
29997

Now the other issue is that when Distributed is used you are doing inter-process communication which does not occur when using Threads. Basically, the inter-process processing does not make sense to be used for jobs lasting few milliseconds. When you try to increase the processing volumes the difference might start to diminish.

So when to use what - it depends.. General guidelines (somewhat subjective) are following:

  1. Processes are more robust (threads are still experimental)
  2. Threads are easier as long as you do not need to use locking or atomic values
  3. When the parallelism level is beyond 16 threads become inefficient and Distributed should be used (this is my personal observation)
  4. When writing utility packages for others use threads - do not distribute code inside a package. Explanation: If you add multi-threading to a package it's behavior can be transparent to the user. On the other hand Julia's multiprocessing (Distributed package) abstraction does not distinguish between parallel and distributed - that is your workers can be either local or remote. This makes fundamental difference how code is designed (e.g. SharedArrays vs DistributedArrays), moreover the design of code might also depend on e.g. number of servers or possibilities of limiting inter-node communication. Hence normally, Distributed-related package logic should be separated from from standard utility package while the multi-threaded functionality can just be made transparent to the package user. There are of course some exceptions to this rule such as providing some distributed data processing server tools etc. but this is a general rule of thumb.
  5. For huge scale computations I always use processes because you can easily go onto a computer cluster with them and distribute the workload across hundreds of machines.
user_99577100
Related