Converting NumPy array to a set takes too long

user_7272130 0
Viewed 3147

I'm trying to execute the following

from numpy import *
x = array([[3,2,3],[711,4,104],.........,[4,4,782,7845]])  # large nparray
for item in x:
    set(item)

and it takes very long compared to:

x = array([[3,2,3],[711,4,104],.........,[4,4,782,7845]])  # large nparray
for item in x:
    item.tolist()

Why does it take much longer to convert a NumPy array to a set than to a list? I mean basically both have complexity O(n)?

set numpy arrays python performance
2 Answers

Here is a way to speed things up: avoid the loop and use a multiprocessing pool.map trick

from multiprocessing.dummy import Pool as ThreadPool
import multiprocessing

pool = ThreadPool(multiprocessing.cpu_count()) # get the number of CPU
y = pool.map(set,x) # apply the function to your iterable
pool.close()
pool.join()
user_76488810
Related