Yet another command-line argument with spaces question

Question

I am executing a spark job, via spark-submit using a wrapper script and a parameter file. The parameter file contains some config information in addition to a set of set of command line arguments needed by the spark job.

The parameter file, params.sh, look like this:

MODE="query"
QUERYCOND="Id==1 and customer==2"
PACKAGE="com.etl"
CLASS="ExecSparkETL"
DRIVER_CLASS_PATH="/home/jars/ojdbc8.jar"
JARS="/home/jars/ojdbc8.jar"
JAR="/home/jars/etl-scala_2.11-1.0.0.jar"
ARGS="$MODE \"$CUSTOM\""

The wrapper script, exec.sh, contains this:

source params.sh
spark2-submit \
--class ${PACKAGE}.${CLASS} \
--queue root.me \
--deploy-mode client \
--master yarn \
--driver-class-path $DRIVER_CLASS_PATH \
--jars $JARS \
--conf spark.network.timeout=1200000 \
$JAR \
$ARGS

If I echo $ARGS from the script the result is what I should be passing to the scala jar as two args, one of them having spaces in the arg which is wrapped in double-quotes:

query "Id==1 and customer==2"

But when I execute and print those two args from within the scala program I get:

arg1: query
arg2: "Id==1

So it is somehow not parsing my double-quote wrapped argument correctly.

Now if I execute the program without assigning the program paramters to the script variable ARGS, it works as expected:

source params.sh
spark2-submit \
--class ${PACKAGE}.${CLASS} \
--queue root.me \
--deploy-mode client \
--master yarn \
--driver-class-path $DRIVER_CLASS_PATH \
--jars $JARS \
--conf spark.network.timeout=1200000 \
$JAR \
query "Id==1 and customer==2"

Results...

arg1: query
arg2: "Id==1 and customer==2"

So it doesn't seem to like the fact that the two parameters passed to the scala program are stored in a variable, like it is dropping the double-quote wrap from the parameter that contains spaces.

Any ideas on how I can fix that?

user_7552 · Answer

I'm trying to put a command in a variable, but the complex cases always fail! -- the solution is to use an array:

source params.sh

spark_args=(
    --class "${PACKAGE}.${CLASS}"
    --queue root.me
    --deploy-mode client
    --master yarn
    --driver-class-path "$DRIVER_CLASS_PATH"
    --jars "$JARS"
    --conf spark.network.timeout=1200000
    "$JAR"
    "$MODE"
    "$CUSTOM"
)

spark2-submit "${spark_args[@]}"

Note how all the variables are quoted.

Arrays are available in bash/zsh/ksh but not plain sh.

Yet another command-line argument with spaces question

1 Answers