PYTHON How to generate 20 million unrepeatable random numbers

user_14069441 0
Viewed 227

Need to generate 20 million unrepeatable random numbers with 8 characters length and save it in an array. I try with multiprocessing,threading but it stays slow.

Try with multiprocessing:

from numpy.random import default_rng
from multiprocessing import Process,Queue
import os,time
import numpy as np
rng = default_rng()
f=np.array([],dtype=np.int64)
def generate(q,start,stop):    
    numbers=[rng.choice(range(start,stop),replace=False) for _ in range(1000)]    
    q.put(numbers)       

if __name__ == '__main__':
    timeInit = time.time()
    for x in range(20000):
        q=Queue()
        p = Process(target=generate,args=(q,11111111,99999999,))    
        p.start()        
        f=np.append(f,q.get())
        p.join()
    print(f)
    timeStop = time.time()
    print('[TIME EXECUTED] ' + str(timeStop-timeInit) +' segs')
numpy multithreading multiprocessing python
3 Answers

This took less than 30 secs on my personal laptop, if it works for you:

import random
candidates = list(range(10**7, 10**8)) # all numbers from 10000000 to 99999999
random.shuffle(candidates)
result = candidates[:20* 10**6] # take first 20 million
user_68436750

You haven't explained why you're doing all of that overhead. I simply took a random sample from the candidate numbers:

from random import sample

result = sample(
    list(range(10**7, 10**8)),
    2*10**7
)

51 seconds on my laptop, with interference from other jobs.

I just ran a more controlled test on both solutions. The one in this post took 48.5 seconds; the one from naicolas took 81.6 seconds, likely due to the extra list creation.

user_47851850

I hope I got your idea. The random numbers that you are trying to generate are actually a bit tricky. Basically we are looking for a set of unique (non-repeatable) but random numbers. In this case, we can not draw random numbers from uniform distribution, because there is no guarantee that numbers are unique.

There are 2 possible algorithms. The first one is to generate A LOT of possible random numbers, and remove those repeated ones. For instance,

import numpy as np


N = 20_000_000
L0 = 11_111_111  # legitimate int in Python
L1 = L0 * 9

not_enough_unique = True

while not_enough_unique:
    X = np.random.uniform(L0, L1, int(N * 2)).astype(int)
    X_unique = np.unique(X)  # remove repeated numbers
    not_enough_unique = len(X_unique) < N

random_numbers = X_unique[:N]
np.random.shuffle(random_numbers)

There is also another more "physics" approach. We can start with equal–spaced numbers, and move each number a little bit. The result will not be as random as the first one, but it is much faster and purely fun.

import numpy as np

N = 20_000_000
L0 = 11_111_111  # legitimate int in Python
L1 = L0 * 9

lattice = np.linspace(L0, L1, N)  # all numbers have equal spacing
pertubation = np.random.normal(0, 0.4, N)  # every number move left/right a little bit
random_numbers = (lattice + pertubation).astype(int)

# check if the minimum distance between two successive numbers
# i.e. all numbers are unique
min_dist = np.abs(np.diff(random_numbers)).min()
print(f"generating random numbers with minimum separation of {min_dist}")
print("(if it is > 1 you are good)")

np.random.shuffle(random_numbers)

(Both algorithms generate the result within 10s on my laptop)

user_41165380
Related