Remove all values within one list from another list in less time (optimizing times)

user_7635726 0
Viewed 51

I present the following example. I have two lists, one total and one with processed elements. The idea is, as the example shows, to eliminate the elements that have already been processed. The problem is that with the real data, that takes me 13 minutes! Surely it can be optimized in some way. Please help.

processed = ['a','b','c']

total = [(1, 'a', 'alaksdl'),
         (2, 'x', 'asda'),
         (3, 'b', 'asda'),
         (4, 'c', 'wef'),
         (5, 'e', 'asaaa'),         
         (5, 'j', 'asd')                     
]

start = time.time()
result = [x for x in total if x[1] not in processed]
end = time.time()
print("Time: ",round(end - start,5), " segundos")

Time:  0.00014  segundos
[(2, 'x', 'asda'), (5, 'e', 'asaaa'), (5, 'j', 'asd')]
result
list optimization python
3 Answers
s = set(processed)
list(filter(lambda x: x[1] not in s, total))
user_31250080

Just convert to an intermediate dict, remove the processed items and convert back to result

import time
processed = ['a','b','c']

total = [(1, 'a', 'alaksdl'),
         (2, 'x', 'asda'),
         (3, 'b', 'asda'),
         (4, 'c', 'wef'),
         (5, 'e', 'asaaa'),         
         (5, 'j', 'asd')                     
]

start = time.time()
inter = {x[1]:x for x in total}
for item in processed:
    del inter[item]
result = list(inter.values())
end = time.time()

This should be more like O(n)

quamrana_48340

You can use pandas for this, like below. We will use a set of processed for the isin method, as it 's much faster in general to look for values in a set compared with list. The following code takes some seconds for a list of 6,000,000 items:

import pandas as pd

df=pd.DataFrame(total)
s=set(processed)
df2=df[-df[1].isin(s)]

res=[tuple(i) for i in df2.values]

>>> print(res)

[(2, 'x', 'asda'), (5, 'e', 'asaaa'), (5, 'j', 'asd')]
user_82285580
Related