Best practice to create SparkSession object in Scala to use both in unittest and spark-submit

user_5103857 0
Viewed 20714

I have tried to write a transform method from DataFrame to DataFrame. And I also want to test it by scalatest.

As you know, in Spark 2.x with Scala API, you can create SparkSession object as follows:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.bulider
     .config("spark.master", "local[2]")
     .getOrCreate()

This code works fine with unit tests. But, when I run this code with spark-submit, the cluster options did not work. For example,

spark-submit --master yarn --deploy-mode client --num-executors 10 ...

does not create any executors.

I have found that the spark-submit arguments are applied when I remove config("master", "local[2]") part of the above code. But, without master setting the unit test code did not work.

I tried to split spark (SparkSession) object generation part to test and main. But there is so many code blocks needs spark, for example import spark.implicit,_ and spark.createDataFrame(rdd, schema).

Is there any best practice to write a code to create spark object both to test and to run spark-submit?

spark-submit apache-spark scala
4 Answers

The spark-submit command with parameter --master yarn is setting yarn master. And this will be conflict with your code master("x"), even using like master("yarn").

If you want to use import sparkSession.implicits._ like toDF ,toDS or other func, you can just use a local sparkSession variable created like below:

val spark = SparkSession.builder().appName("YourName").getOrCreate()

without setting master("x") in spark-submit --master yarn, not in local machine.

I advice : do not use global sparkSession in your code. That may cause some errors or exceptions.

hope this helps you. good luck!

user_101641890
Related