Generating unique set of values using multiprocessing in python

Viewed 98

I am trying to generate unique set of values for example set of (int, int). I tried using multiprocessing and faker. Each process generates a unique list of length 125000, but at last when it gets appended the unique vales are just 125000 i.e each process has generated the same set of values. As we expect to have 1000000 unique values, Is there a better way to generate unique values ?

Code :

from faker import Faker

faker = Faker()

def get_row():
    return [faker.unique.pyint(min_value=0, max_value=100000000, step=1), faker.unique.pyint(min_value=0, max_value=100000000, step=1)]

def get_rpwss(n):
    return np.unique([get_row() for x in range(n)], axis=0).tolist()

t0 = time.time()
n_threads = 8
total = 1000000
n_per_task = total // n_threads
result = []
with multiprocessing.Pool() as p:
    for batch in p.imap_unordered(get_rows,[n_per_task for x in range(n_threads)],):
        result = result + batch

old = result
result = np.unique(result, axis=0).tolist()

t1 = time.time()

print(len(result))
print(t1-t0)
print(len(old))

Output :

125000
21.0101299338833
1000000
0 Answers
Related