combining thousands of list strings in python

Question

I have a .txt file of "Alice in the Wonderland" and need to strip all the punctuation and make all of the words lower case, so I can find the number of unique words in the file. The wordlist referred to below is one list of all the individual words as strings from the book, so wordlist looks like this


    ["Alice's", 'Adventures', 'in', 'Wonderland', "ALICE'S", 
    'ADVENTURES', 'IN', 'WONDERLAND', 'Lewis', 'Carroll', 'THE', 
    'MILLENNIUM', 'FULCRUM', 'EDITION', '3.0', 'CHAPTER', 'I', 
    'Down', 'the', 'Rabbit-Hole', 'Alice', 'was', 'beginning', 
    'to', 'get', 'very', 'tired', 'of', 'sitting', 'by', 'her', 
    'sister', 'on', 'the', 'bank,'

The code i have for the solution so far is


from string import punctuation
def wordcount(book):

    for word in wordlist:
        no_punc = word.strip(punctuation)
        lower_case = no_punc.lower()
        newlist = lower_case.split()
        print(newlist)

This works for stripping punctuation and making all words lowercase, however the newlist = lower_case.split() makes an individual list of every word, so I cannot iterate over one big list to find the number of unique words. The reason I did the .split() is so that when iterated over, python does not count ever letter as a word, rather each word is kept intact since it is its own list item. Any ideas on how I can improve this or a more efficient approach? Here is a sample of the output


    ['down']
    ['the']
    ['rabbit-hole']
    ['alice']
    ['was']
    ['beginning']
    ['to']
    ['get']
    ['very']
    ['tired']
    ['of']
    ['sitting']
    ['by']
    ['her']

user_410975 · Answer

Here is a modification of your code with outputs

from string import punctuation

wordlist = "Alice fell down down down!.. down into, the hole."

single_list = []
for word in wordlist.split(" "):
    no_punc = word.strip(punctuation)
    lower_case = no_punc.lower()
    newlist = lower_case.split()
    #print(newlist)
    single_list.append(newlist[0])

print(single_list)
#to get the unique
single_list_unique = set(single_list)
print(single_list_unique)
print(len(single_list_unique))

and that produces:

['alice', 'fell', 'down', 'down', 'down', 'down', 'into', 'the', 'hole']

and the unique set:

{'fell', 'alice', 'down', 'into', 'the', 'hole'}

and the length of the unique:

(This may not be the most efficient approach but it is close to your current code and will suffice for that book of thousands of elements. If this was a backend process serving multiple requests you would optimize it with improvements)

EDIT----------

You may be importing from file using a library that passes in a list, in which case you produce an error AttributeError: 'list' object has no attribute 'split', or you might see the error IndexError: list index out of range because of an empty string. In which case you use this modification:

from string import punctuation

wordlist2 = ["","Alice fell down down down!.. down into, the hole.", "There was only one hole for Alice to fall down into"]


single_list = []
for wordlist in wordlist2:
    for word in wordlist.split(" "):
        no_punc = word.strip(punctuation)
        lower_case = no_punc.lower()
        newlist = lower_case.split()
        #print(newlist)
        if(len(newlist) > 0):
            single_list.append(newlist[0])

print(single_list)
#to get the unique
single_list_unique = set(single_list)
print(single_list_unique)
print(len(single_list_unique))

producing:

['alice', 'fell', 'down', 'down', 'down', 'down', 'into', 'the', 'hole', 'there', 'was', 'only', 'one', 'hole', 'for', 'alice', 'to', 'fall', 'down', 'into']
{'there', 'fall', 'fell', 'alice', 'for', 'down', 'was', 'into', 'the', 'to', 'only', 'hole', 'one'}
13

combining thousands of list strings in python

1 Answers