Calling Julia from Python on thread

Viewed 474

I've been playing around with calling Julia functions from Python which has all worked fine until I try and call from a thread. To illustrate, here is a short example:

Julia code:

function add(x, y)
    return x + y
end

Python code:

import threading
import time
import julia
from julia import Main     # Move these two lines to add_numbers_from_julia() to call from thread
Main.include('library.jl') # ..but it will only work *once* :|

def add_numbers_from_julia():
    z = Main.add(4, 5)
    print(f"result = {z}")

def main():
    i = -1
    while i != 0:
        print("Options:")
        print("0 - Exit")
        print("1 - Add 4 and 5")
        print("2 - Add 4 and 5 from thread")

        i = int(input())

        if i == 1:
            add_numbers_from_julia()
        elif i == 2:
            thread = threading.Thread(target = add_numbers_from_julia, args = ())
            thread.start()
            time.sleep(5)
        print()
            

if __name__ == "__main__":
    main()

In the above code, if the user chooses option 1, we call the Julia add(x,y) function on the current thread and it all works fine.

If the user chooses option 2 we attempt to call add(x,y) from a thread, which causes an access violation. If we remove the from julia import Main and Main.include('library.jl') lines from the top of the file and put them into the add_numbers_from_julia() function, then we can successfully call add(x,y) but only once. Attempting to hit 2 again will cause another access violation.

Can someone tell me whats going wrong and where I should be importing the Julia code if I want to call it from a thread?

Many TIA

1 Answers

Here is an interesting discussion about this (independent from python): https://discourse.julialang.org/t/embedding-julia-into-multithreading-apps/20122/21

In particular, this part of yuyichao explain the problem that you are seeing:

julia does not support being initialize and finalize multiple times so make sure you don’t contruct that class more than once in the lifetime of the program.

As I understand, you could rather use a long-lived "compute" thread and calling "from julia import Main" only once in this thread. You can use queues to send data to this compute thread.

Related