Why is the one-hot encoder provided by sklearn so slow?
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
categories = np.expand_dims(np.array(list(map(chr, np.arange(97,123)))), axis=1)
enc.fit(categories)
vec = np.expand_dims(list(map(chr, np.random.randint(97,123,1000))), axis=1)
%timeit enc.transform(np.array([['b']]))
%timeit (categories == 'b')[:,0] *1
output:
569 µs ± 3.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
3.23 µs ± 58.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
100x slower than the naive way of doing this, while it actually could be faster by using the knowledge that elements of categories are unique.