Issue with scripts not running in a basic MapReduce in Bash

user_8990847 0
Viewed 41

I am trying to create a basic MapReduce function in bash (I am very new at it). I have two scripts at the moment, job_master.sh and map_function.sh. I am trying to run the map function from the job master to cut from a data file, and if it doesn't exist send it to key, but if it does send it to the file of that name. Nothing is happening when I run the job_master script or the map_function script on its own with a file as an argument. It was working before I added the if statements into the map_function.

I have included both codes below if anyone is able to spot why they are not running. I tried including echo statements to test and it is not entering the loop in the job_master script, or doing anything at all in the map_function script.

MAP_FUNCTION

#!/bin/bash

while IFS="," read -r date prod remainder; do
        if [ ! -e "$prod" ];
        then
                echo $prod >> keys
        else
                echo $prod >> $prod
        fi
done

JOB_MASTER

#!/bin/bash

files=$(ls | egrep 'sales_a*')

for elem in $files ; do
        ./map_function.sh $elem
done
mapreduce bash
1 Answers

The map function script is waiting for input. Look at this:

while IFS="," read -r date prod remainder; do
    # ...
done

Where will read get its input? It's waiting on stdin, but you're not passing it anything.

On the other hand, I see that you're calling the map function script like this:

./map_function.sh $elem

That is, you are passing a command line argument. But the map function script doesn't use that argument. It seems you want to redirect the content of $elem to the stdin of the script. Write like this:

for elem in sales_a*; do
    ./map_function.sh < "$elem"
done

This also fixes some other issues you had in the script. files=$(ls | grep 'sales_a*') looks buggy, and it's an inappropriate way to iterate over files.

user_6419550
Related