Julia parallel function inside module

Question

I have a function inside a module in which I want to dynamically change the number of processes (addproc). After I spin up these new procs, I would like to add a function to be executed on each of the workers. However, I am having difficulty defining this function using @everywhere while inside the module. I want to spaw up these new processes only in the function do_work. I do not want to spawn new processes before importing/using the module (this would be an ideal job for multithreading, but my function does a lot of I/O and I currently get a segfault in Julia 1.0).

Here's a simple working example of my issue. Is there a way to define foo on the worker processes (ideally without calling @everywhere include("somefile_with_foo.jl"))

module Example

using Distributed

function do_work()
    total_procs = nprocs()
    # Make sure we have as many works are cores
    if total_procs < Sys.CPU_THREADS
        addprocs(Sys.CPU_THREADS - total_procs); 
    end

    @everywhere function foo(x::Integer)
        println(x)
    end

    remotecall(foo, 2, 2)

    while nprocs() > total_procs
        rmprocs(procs()[end])
    end
end
end

user_9957710 · Answer

You cannot use @everywhere inside a function because the function has its own naming scope and this scope does not exist on remote processes.

One option (and in most cases it is the recommended approach) is to load the module on all workers:

@everywhere using Example

And now you can use its functions on the remote workers.

However, if you really want to be able define and run remote functions you could use metaprogramming to send code to a remote process. Consider my Julia session below:

julia> using Distributed

julia> addprocs(1)
1-element Array{Int64,1}:
 2

julia> function do_work(w)
    f_foo = quote 
        function foo(x::Int64)
           println("hello ",x," from ",myid())
        end
    end
    Main.eval(f_foo)
    @fetchfrom w Main.eval(f_foo)
    @fetchfrom w foo(9999)
end
do_work (generic function with 1 method)

julia> do_work(2)
      From worker 2:    hello 9999 from 2

Julia parallel function inside module

1 Answers