If statement to check if an element is found in a counter (Python)

user_4352328 0
Viewed 937

I'm writing a program where I have a counter (collections.counter) named vocabulary which is the top 10,000 most frequent of a counter named wordFrequency made from counting instances of words read from a text file. I'm stuck on trying to make an if statement that checks if an element is found within that counter. What I have is:

vocabulary = wordFrequency.most_common(10000)

[...]

for line in trainReader2:
    if len(line) == 10 and line[5] != "_":
        if wordPosition < matrixWidth:
            word = line[1]
            if word in vocabulary:
                sentenceRow[wordPosition] = word
            else:
                sentenceRow[wordPosition] = "[unknown]"
            wordPosition += 1
    elif wordPosition != 0:
        trainingMatrix.append(sentenceRow)
        print("sentence row:", sentenceRow)
        wordPosition = 0
        sentenceRow = ["[padding]"] * matrixWidth

I thought for sure if word in vocabulary: would work, but the condition never seems to be met and the sentence rows always consist entirely of [unknown] and [padding]. What if statement am I supposed to use for this situation?

counter collections python
1 Answers

The problem is that wordFrequency.most_common(10000) returns a list of tuples, not a dictionary. You need to make it a dictionary or a set in order to find words in it.

vocabulary  = dict(wordFrequency.most_common(10000))

or 

vocabulary  = set(w for w,_ in wordFrequency.most_common(10000))

You could also use the last value in most_common as a threshold and compare the value of the word's frequency to it instead of creating a second word list (given that the last of the most common words may have the same frequency as other words not included in the list)

minFreq =  wordFrequency.most_common(10000)[-1][1]

...  

if wordFrequency.get(word,0) >= minFreq:
user_52375600
Related