What's the fastest (most efficient) way to iterate through a list and find a fitting string pattern in Python?

user_13845970 0
Viewed 91

So I have a list of string which looks something like that -

['https://images.website.com/images/data/X/source1',
'https://images.website.com/articles/data/main_data/source2',
'https://images.website.com/harmony/data/Y/source3',
'https://images.website.com/files/data/Z/source4',
'https://images.website.com/pictures/data/T/source5']

I need to find the item which has main_data in it.

I thought of something like.


def func():
    for item in list:
        if item.index('/main_data/'):   # not -1 
            return item

return -1  # item not found

Is it the fastest way when I have a list of 100-1000 and maybe more items?

python
3 Answers

If you only have 100-1000 items as you mentioned, I would imagine that the method you already posted for us completes instantly, so it's hard to imagine perceivable gains from speeding things up. However, if you have many more items, it is usually faster to use a builtin function over a for loop if possible.

def func(l):
    return next(filter(lambda s: '/main_data/' in s, l))
    # raises StopIteration if no string contains /main_data/
user_146449410

There are many points to be made:

  • If performance is your concern, don't use Python for that bit.
  • If you can use several cores, do so. The task is embarrassingly parallel, although you likely need to keep the threads in a pool, because there are so few items.
  • If you roughly know where the substring is within a string, then you can avoid traversing the entire string. Same for guessing at which position the string usually is within the list.
  • If you have information about some property that allows you to cut the search down, use it.
  • If you need to search several times, perhaps for different terms, you are likely able to build a better data structure or an index rather than performing the naive search every time.
user_93053980

Your function is fastest

def func_1(path_list):
     for item in path_list:
         if '/main_data/' in item:
             return item
     return -1

For example, for 100 elements, the time processing is 0:00:00.000048 and thats is good.

path_list = 99*['https://images.website.com/images/data/X/source1']
path_list.append('https://images.website.com/articles/data/main_data/source2')

from datetime import datetime

now = datetime.now()

func_1(path_list)

end = datetime.now()
print(end-now)

0:00:00.000048

user_109862090
Related