Use python's multiprocessing library in Rust

Question

I use Rust to speed up a data processing pipeline, but I have to run some existing Python code as-is, which I want to parallelize. Following discussion in another question, creating multiple Python processes is a possible approach given my project's specific constraints. However, running the code below gives an infinite loop. I can't quite understand why.

use cpython::Python;

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    py.run(r#"
import sys
from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    print('start')
    sys.argv=['']
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
    "#, None,None).unwrap();
}

Output (continues until Ctrl-C):

start
start
start
start
start
start
start
start

EDIT

As mentioned in the comments below, I gave up on trying to create processes from the Python code. The interference between Windows, the Python multiprocessing module, and how processes are created with Rust are too obscure to manage properly. So instead I will create and manage them from Rust. The code is therefore more textbook:

use std::process::Command;
fn main() {
    let mut cmd = Command::new("python");
    cmd.args(&["-c", "print('test')"]); 
    let process = cmd.spawn().expect("Couldn't spawn process.");
    println!("{:?}", process.wait_with_output().unwrap());
}

user_14637 · Answer

I can't reproduce this; for me it just prints start and then hello bob as expected. For whatever reason, it seems that in your case, __name__ is always equal to "__main__" and you get this infinite recursion. I'm using the cpython crate version v0.4.1 and Python 3.8.1 on Arch Linux.

A workaround is to not depend on __name__ at all, but to instead define your Python code as a module with a main() function and then call that function:

use cpython::{Python, PyModule};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let module = PyModule::new(py, "bob").unwrap();
    py.run(r#"
import sys
from multiprocessing import Process

def f(name):
    print('hello', name)

def main():
    print('start')
    sys.argv=['']
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
    "#, Some(&module.dict(py)), None).unwrap();
    module.call(py, "main", cpython::NoArgs, None).unwrap();
}

Use python's multiprocessing library in Rust

1 Answers