pyspark boolean configs issue with capitalization

user_3868051 0
Viewed 101

Probably a little silly question, but i don't want to do trial-error anymore... when you set pyspark config in cluster, which of the following 2 is correct? ->

spark.hadoop.validateOutputSpecs: true

or

spark.hadoop.validateOutputSpecs: True

I know scala has boolean values in lowercase, so does spark. But just a bit confused as python has boolean values as 'True/False', so does pyspark also have same capitalization of 1st letter? Or are spark configs case insensitive?

Thanks!

pyspark boolean python
2 Answers

Pyspark is also lowercase for the config. And I understand your frustration lol. If you look at the pyspark docs, they show it lowercase. Also, all of my pyspark scripts use lowercase for config and they display as lowercase when I view them on the cluster.

user_90700010

Depends completely the way you write. Having said that, both of the below implementations work just fine -

spark.conf.set("spark.hadoop.validateOutputSpecs", True)

spark.conf.set("spark.hadoop.validateOutputSpecs", "true")
user_151125630
Related