I wrote a simple python backend that accepts an HTTP request, connects via SSH to the cluster and executes the "spark-submit" command like this:
spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 8g \
--driver-cores 4 \
--executor-memory 8g \
--executor-cores 4 \
--num-executors 6 \
--conf spark.yarn.submit.waitAppCompletion=false \
--name "Spark_Test" \
--driver-class-path "./" \
--jars .../test_jar.jar \
--class ...test_class .../test_jar.jar \
--testoutpath "hdfs://..."
It doesn't seem to be a good practice. I assume that it is better to do a remote launch via pyspark. For example:
import pyspark
from pyspark.sql import SparkSession
conf = pyspark.conf.SparkConf()
conf.set("spark.executor.cores", "4")
conf.set("spark.executor.memory", "8g")
conf.set("spark.executor.instances", "6")
conf.set("spark.driver.memory", "8g")
conf.set("spark.driver.cores", "4")
conf.set("spark.yarn.submit.waitAppCompletion", "false")
conf.set("spark.jars", ".../test_jar.jar")
conf.set("spark.yarn.jars", "hdfs://...")
conf.setMaster('yarn')
# conf.set("spark.submit.deployMode", "cluster")
conf.set("spark.driver.extraClassPath", "...test_class .../test_jar.jar")
spark = SparkSession.builder.appName("Spark_Test").config(conf=conf).enableHiveSupport().getOrCreate()
sc = spark.sparkContext
Is it possible to pass custom command-line arguments via pyspark, such as "--testoutpath"?