Coursera Bigdata Grader and how to set Hadoop Streaming number of reducers?

user_9659033 0
Viewed 44

I'm trying to pass the course task on the Coursera, but fail at some unit test with the following error:

RES1_6 description: The first job should have more than 1 reducer or shouldn't have them at all. Please set the appropriate number in -D mapreduce.job.reduces. It can be 0 or more than 1.

BUT, I use NUM_REDUCERS=4 in the following script!

yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar \
    -D mapreduce.job.name="somename" \
    -D mapreduce.job.reduces=${NUM_REDUCERS}\
    -files mapper.py,reducer.py,somesupplimentary.txt \
    -mapper "python3 mapper.py" \
    -reducer "python3 reducer.py" \
    -input someinput.txt \
    -output ${OUT_DIR} > /dev/null 2> $LOGS

And when I read the logs, I see the following:

Job Counters 
    Killed reduce tasks=1
    Launched map tasks=2
    Launched reduce tasks=9
    Data-local map tasks=2

So I feel myself stupid and totally do not understand what does the grader want from me. That simply doesn't make a consecutive picture for me. It seems that I use MORE than one reducer and the log seem to approve it. Why does the unit test fail? Or am I not understanding some homespun truth?

hadoop-streaming hadoop-yarn hadoop mapreduce bash
1 Answers

I have needed to delete "2> $LOGS" part. That was the grader issue, since it have implied that I would not catch the logs and would not write it into the file, at least I shouldn't do it myself.

user_96590330
Related