How to pass input path to MRJob.mapper_raw?

user_7516298 0
Viewed 201

I'm running a mapreduce job with mrjob package and I am trying to use mapper raw as below

class MRJOB(MRJob):
    def mapper_raw(self, input_path, input_uri):
          import csv
          print(input_path, input_uri)
          with open(input_path) as f:
              reader = csv.reader(f)
              for line in reader:
                  if line:
                      yield (0, line)

    def steps(self):
          return [
              MRStep(mapper=self.mapper_raw)
          ]  # , reducer=self.reducer)]


if __name__ == "__main__":
      MRJOB().run()

however, if I run this as python mrjobfile.py inputfile.csv I get the following error:

TypeError: expected str, bytes or os.PathLike object, not NoneType

How to tell mrjob to treat input as a string with the filename? It seems to just pass in the first line of the file.

mrjob mapreduce
1 Answers

You have to declare mapper_raw instead of mapper in your MRStep. So, how it will look like:

    def steps(self):
          return [
              MRStep(mapper_raw=self.mapper_raw)
          ]  # , reducer=self.reducer)]
user_135164910
Related