I am trying to generate unique set of values for example set of (int, int). I tried using multiprocessing and faker. Each process generates a unique list of length 125000, but at last when it gets appended the unique vales are just 125000 i.e each process has generated the same set of values. As we expect to have 1000000 unique values, Is there a better way to generate unique values ?
Code :
from faker import Faker
faker = Faker()
def get_row():
return [faker.unique.pyint(min_value=0, max_value=100000000, step=1), faker.unique.pyint(min_value=0, max_value=100000000, step=1)]
def get_rpwss(n):
return np.unique([get_row() for x in range(n)], axis=0).tolist()
t0 = time.time()
n_threads = 8
total = 1000000
n_per_task = total // n_threads
result = []
with multiprocessing.Pool() as p:
for batch in p.imap_unordered(get_rows,[n_per_task for x in range(n_threads)],):
result = result + batch
old = result
result = np.unique(result, axis=0).tolist()
t1 = time.time()
print(len(result))
print(t1-t0)
print(len(old))
Output :
125000
21.0101299338833
1000000