counting words with or Whitespace with collections.counters

user_14510454 0
Viewed 49

I'm doing a word count of a source code, and for example, I want to know how many for there are in a txt, for now it does well, but in some cases programmers write this way: for( or for (. In my case my code only counts the for ( with the space but not without the space, how could I solve this?. Also in some cases, programmers put for example, for(xxx or for (xxx or for ( xxx, how could I make only for ?

from collections import Counter

words_to_keep = {"for", "setup()", "loop()"}

def word_count(filename):
    with open('hello.txt', 'r') as f: # use `filename`
        return Counter(w for w in f.read().split() if w in words_to_keep)

counter = word_count('hola.txt')

for i in counter:
    print (i, ":", counter [i])
python
3 Answers

As you observed, the problem with split is that you need your words to be surrounded with spaces but that not always true with code. Probably regex is your best option to handle the more general case of string matching.

This first compiles a pattern by OR-ing your accepted words (after escaping them), finds all matches of the pattern, then counts the matches of the regex on the file:

import re
from collections import Counter

words_to_keep = {"for", "setup()", "loop()"}

pattern = re.compile('|'.join(re.escape(word) for word in words_to_keep))
# in this case, pattern = "for|setup\(\)|loop\(\)"

def word_count(filename):
    with open(filename, 'r') as f:
        words_found = pattern.findall(f.read())
        return Counter(words_found)

for word, count in word_count('test.txt').items():
    print (word, ":", count)

If the file is big and you don't want to read it all at once, you can use the benefits of adding Counters:

def word_count(filename):
    counter = Counter()
    with open(filename, 'r') as f:
        for line in f:
            counter += Counter(pattern.findall(line))
        return counter
user_60458000

You can replace the optional space(s) and the bracket using re.sub:

from collections import Counter
import re

words_to_keep = {"for", "setup()", "loop()"}

def word_count(filename):
    with open('hello.txt', 'r') as f: # use `filename`
        return Counter(w for w in re.sub(r'for(\s*\().*','',f.read()).split() if w in words_to_keep)

counter = word_count('hola.txt')

for i in counter:
    print (i, ":", counter [i])
user_122698570

Without Counter:

import re

def word_count(filename, words):
    res = {x: 0 for x in words}
    pattern = re.compile('|'.join(re.escape(word) for word in words))
    with open(filename, 'r') as f:
        for a in re.finditer(pattern, f.read()):
            res[a.group(0)] += 1 
    return res

words = ("for", "setup()", "loop()")
result = word_count('hola.txt', words)
print(result)
user_92102550
Related