Python 3.5 - Get counter to report zero-frequency items

user_4610654 0
Viewed 3895

I am doing textual analysis on texts that due to PDF-to-txt conversion errors, sometime lump words together. So instead of matching words, I want to match strings.

For example, I have the string:

mystring='The lossof our income made us go into debt but this is not too bad as we like some debts.'

And I search for 

key_words=['loss', 'debt', 'debts', 'elephant']

The output should be of the form:

Filename Debt Debts Loss Elephant
mystring  2    1     1    0

The code I have works well, except for a few glitches: 1) it does not report the frequency of zero-frequency words (so 'Elephant' would not be in the output: 2) the order of the words in key_words seems to matter (ie. I sometimes get 1 count each for 'debt' and 'debts', and sometimes it reports only 2 counts for 'debt', and 'debts is not reported. I could live with the second point if I managed to "print" the variable names to the dataset... but not sure how.

Below is the relevant code. Thanks! PS. Needless to say, it is not the most elegant piece of code, but I am slowly learning. 

bad=set(['debts', 'debt'])

csvfile=open("freq_10k_test.csv", "w", newline='', encoding='cp850', errors='replace')
writer=csv.writer(csvfile)
for filename in glob.glob('*.txt'):

    with open(filename, encoding='utf-8', errors='ignore') as f:
      file_name=[]
      file_name.append(filename)

      new_review=[f.read()]
      freq_all=[]
      rev=[]

      from collections import Counter

      for review in new_review:
        review_processed=review.lower()
        for p in list(punctuation):
           review_processed=review_processed.replace(p,'')
           pattern = re.compile("|".join(bad), flags = re.IGNORECASE)
           freq_iter=collections.Counter(pattern.findall(review_processed))           

        frequency=[value for (key,value) in sorted(freq_iter.items())]
        freq_all.append(frequency)
        freq=[v for v in freq_all]

    fulldata = [ [file_name[i]] + freq  for i, freq in enumerate(freq)]  

    writer=csv.writer(open("freq_10k_test.csv",'a',newline='', encoding='cp850', errors='replace'))
    writer.writerows(fulldata)
    csvfile.flush()
word-frequency counter python
4 Answers

A sleek solution is to use regex:

import regex
mystring='The lossof our income made us go into debt but this is not too bad as we like some debts.'
key_words=['loss', 'debt', 'debts', 'elephant']
print ({k:len(regex.findall(k,mystring,overlapped=True)) for k in key_words})

results to:

{'loss': 1, 'debt': 2, 'debts': 1, 'elephant': 0}
user_5089070

Counting the occurrences can be done in a simple one-liner:

counts = {k: mystring.count(k) for k in key_words}

Putting that together with a csv.DictWriter results in:

import csv

mystring = 'The lossof our income made us go into debt but this is not too bad as we like some debts.'
key_words = ['loss', 'debt', 'debts', 'elephant']

counts = {k: mystring.count(k) for k in key_words}
print(counts) # {'loss': 1, 'debt': 2, 'debts': 1, 'elephant': 0}

# write out
with open('out.csv', 'w', newline='') as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=counts, delimiter=' ')
    # key_words
    writer.writeheader()
    # counts
    writer.writerow(counts)

# out.csv:
# loss debt debts elephant
# 1 2 1 0
user_129982050
Related