Calculate number of rating records provided in each year-month

user_19346807 0
Viewed 103

My mapper is:

import sys

for line in sys.stdin:
    # split line into the four fields
    fields = line.strip().split("\t")

    value = fields[2] #rating
    key = fields[3] #timestamp in unix seconds
    
    print(key, value, sep="\t")

My reducer is:

import sys
                                     
(last_key , count) = (None, 0)

for line in sys.stdin:

    (key, value) = line.strip().split("\t")
   
    if  (last_key  and last_key  !=key):
          print(last_key, count, sep="\t")
          count=0
    
    last_key  = key
    count += int(value)
    
          
print(last_key, count, sep="\t")

How to I get the number of ratings? The mapper works fine. And when should I convert the timestamp (last_key in this case)

Output should be (year-month \t number of rating records)

mapreduce timestamp python
1 Answers

If you want to reduce (group by) the year-month string, that needs to be the key of the mapper, rather than having second-precision.

After that, you just need to count the number of values for that key in the reducer. (The mapper's value can just be 1, not the actual value, if you only need to count the ratings, then the reducer can sum the values)

I'd recommend using mrjob library rather than manually reading from sys.stdin, or you can re-write your code in PySpark and do the same operation in less lines.

user_23086830
Related