MapReduce WordCount with prop nouns

Alex V. 0
Viewed 19

I'm trying to make a MapReduce WordCount that get a large article and counts proper nouns. Here's requirements:

  1. Starts with a capital letter and has never been found in the text with a small letter
  2. Has length between 2 and 7 letters
  3. Sort in descending order

Looks like a typical WordCount mapreduce, but I couldn't do this. How to get rid of all the punctuation marks? What's the right way to construct mapper and reducer?

import sys
import re

for line in sys.stdin:
    article_id, text = line.strip().split('\t', 1)
    text = re.sub('\W', ' ', text).split(' ')
    for word in text:
        if len(word) >= 2 and len(word) < 7:
            key = "".join(sorted(word.lower()))
            print("{}\t{}\t{}".format(key, word.lower(), 1))
mapreduce python
1 Answers

If you are only looking for words, since you already imported re, you can use re.compile (one solution) :

re.compile('\w+').findall(text)

This way you remove all the punctuation in the string, keeping only letters and numbers.

If you take the string below :

text = "Looks like a typical WordCount mapreduce, but I couldn't do this. How to get rid of all the punctuation marks"

you quickly obtain :

liste = ['Looks', 'like', 'a', 'typical', 'WordCount', 'mapreduce', 'but', 'I', 'couldn', 't', 'do', 'this', 'How', 'to', 'get', 'rid', 'of', 'all', 'the', 'punctuation', 'marks']

On which you can run your for loop in the same way.

L4ur3nt0
Related