Find 4k words in a 100k len list of words

user_15501146 0
Viewed 103

I have a list of words in a txt file and thousands of words in a list. How can I search for them and add them in a new list if they are found

The words.txt file contain 100k words. my_list contain 4k words.

Currently I do this.

    my_list = [hello, hi, hey, ho, wow, .....] 

    with open("words.txt") as f:
        lines = [line.rstrip() for line in f]
    
    words_in_lines = []

    for i in my_list: 
        if i in lines:
            words_in_lines.append(i)

This is non ending, it's not executing as there are too many words in the words list.

data-structures algorithm python
1 Answers
  1. Convert my_list from list to set for faster lookups
  2. Instead of looking up words from my_list in lines, search for word in line inside my_list
my_list = set([hello, hi, hey, ho, wow, .....])
words_in_lines = []

with open("words.txt") as f:
    for line in f:
        words = line.strip().split()
        for word in words:
            if word in my_list:
                words_in_lines.append(word)

This should have a time complexity of O(number of words in file)

Edit: as pointed out by @greybeard, this appproach

  1. Does not have the same order of words in words_in_lines
  2. Modifies my_list from list to set
  3. Skips the creation of a 100k word list
user_93507200
Related