Hadoop MapReduce function is giving an error. Streaming Command Failed

user_12903350 0
Viewed 171

Saved mapper.py ,reducer.py, count_word_data.txt files in C:\BigData\Hadoop-3.2.2 directory.

Initial commands given

hadoop-3.2.2\bin\>hdfs fs -mkdir input

hadoop-3.2.2\bin\>hdfs fs -copyFromLocal C:\BigData\Hadoop-3.2.2\count_word_data.txt /input

mapper.py

#!/usr/bin/python -O

import sys

for line in sys.stdin:
    line = line.strip()
    keys = line.split()
    for key in keys:
        value = 1
        print("{}".format(key)," {} ".format(value))

reducer.py

#!/usr/bin/python -O

import sys

last_key = None
running_total = 0

for input_line in sys.stdin:
   input_line = input_line.strip()
   this_key, value = input_line.split("\t", 1)
   value = int(value)

   if last_key == this_key:
       running_total += value
   else:
       if last_key:
           print( "%s\t%d" % (last_key, running_total) )
       running_total = value
       last_key = this_key

if last_key == this_key:
   print( "%s\t%d" % (last_key, running_total) )

To run these python files, I have used Hadoop command

Hadoop jar C:/BigData/hadoop-3.2.2/share/Hadoop/tools/lib/hadoop-streaming-3.2.2.jar -mapper “python C:/BigData/hadoop-3.2.2/mapper.py” -reducer  “C:/BigData/hadoop-3.2.2/reducer.py”  -input “input/count_word_data.txt”  -output “input/output”

After that it is giving me this error

Though I have specified all the paths(Python,Hadoop,JAVA,SPARK) correctly in system environment variables.I dont know why I am facing this error.

2022-02-10 18:33:09,065 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
2022-02-10 18:33:09,065 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
2022-02-10 18:33:09,066 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2022-02-10 18:33:09,075 ERROR streaming.PipeMapRed: configuration exception
java.io.IOException: Cannot run program "python": CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
        at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
        at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
hadoop-streaming hadoop python
1 Answers

The system cannot find the file specified

This problem is not directly related to Hadoop.

Either:

  1. The error indicates that python is not installed (or on your Windows PATH). To fix this, make sure you can run python -V and it shows you the version that is installed.
  2. It is referring to your mapper/reducer scripts, which need to be uploaded to HDFS, where the command will actually be ran. You'd do this with more options -files C:\path\to\mapper.py,C:\path\to\reducer.py.

You also do not need quotes on your input or output paths.

I will also point out that PySpark is arguably better/easier way to do word-count, and it doesn't require HDFS.

user_23086830
Related