I was trying to improve the performance of the func function and I found that a simple change in how the aX list is generated improves the performance quite a bit:
import timeit
import numpy as np
def func(a, b):
return [_ for _ in a if _ not in b]
Na, Nb = 10000, 5000
b = list(np.random.randint(1000, size=Nb))
# Ordered list of Na integers
a1 = [_ for _ in range(Na)]
# Random list of Na integers
a2 = list(np.random.randint(Na, size=Na))
# Ordered list of Na integers generated with numpy
a3 = list(np.arange(Na))
start_time = timeit.default_timer()
ab1 = func(a1, b)
abt1 = timeit.default_timer() - start_time
print("Time ab1", abt1)
start_time = timeit.default_timer()
ab2 = func(a2, b)
abt2 = timeit.default_timer() - start_time
print("Time ab2", abt2)
start_time = timeit.default_timer()
ab3 = func(a3, b)
abt3 = timeit.default_timer() - start_time
print("Time ab3", abt3)
print("Ratio 1/2:", abt1 / abt2)
print("Ratio 1/3:", abt1 / abt3)
In Python 2.7.13 this results in:
('Time ab1', 5.296088933944702)
('Time ab2', 1.5520200729370117)
('Time ab3', 1.5581469535827637)
('Ratio 1/2:', 3.412384302428827)
('Ratio 1/3:', 3.3989662667998095)
In Python 3.5.2 the difference is even larger:
Time ab1 6.758207322000089
Time ab2 1.5693355060011527
Time ab3 1.5148192759988888
Ratio 1/2: 4.306413317073784
Ratio 1/3: 4.461395117608107
I need to process an ordered list integers (i.e: a1 or a3), so my question is:
Why is the random list processed so much faster than the ordered list not generated with numpy?