Multiprocessing: append to 2 lists simultanously

Viewed 1132

I have this code:

from multiprocessing import Pool, Manager
import numpy as np

l = Manager().list()

def f(args):
    a, b = args
    l.append((a, b))


data = [(1,2), (3,4), (5,6)]
with Pool() as p:
    p.map(f, data)
x, y = np.transpose(l)

# do something with x and y...

In reality, data is an array with lots of values, and the transpose operation is long and memory consuming.

What I would like is to append "a" and "b" directly to lists x and y to avoid the transpose operation. It is important that the output keeps the correspondence in data and looks like so: [[1,3,5], [2,4,6]]

What would be a smart way to do that?

1 Answers
Related