Difference between spark.config & spark.conf.set

Viewed 465

I am using Spark 2.1.0. In this version getting a SparkSession is through using a builder, e.g. SparkSession.builder.app_name().config("some-property", "something-value").

Earlier versions used spark.conf.set("property", "value").

Are this SparkSession.config("property", "value") and spark.conf.set("property", "value") the same? If not, what is the difference?

I also tried:

SparkSession.builder.app_name()
  .config("some-property1", "something-value1")
  .config("some-property2", "something-value2")

Is this same as the below?

spark.conf.set("property1", "value1")
spark.conf.set("property2", "value2") 
0 Answers
Related