Spark: Unable to create database default as failed to create its directory

user_15389447 0
Viewed 790

I want to put my Java Spark application in a container Docker. To do this I use Maven to create a Fat Jar. When I run the application from Intellij it works fine, but when I run the Jar from terminal using java -jar <name_jar> it crashes after a few second with the following error: org.apache.spark.SparkException: Unable to create database default as failed to create its directory file: ... /spark-warehouse.

The only difference between the stacktraces is that just before crashing in terminal stacktrace appear this WARN: WARN SharedState: URL.setURLStreamHandlerFactory failed to set FsUrlStreamHandlerFactory.

I searched for information on the Internet, but I found nothing. Anyone knows what's the problem?

P.S.: I used (in both cases) openjdk version "1.8.0_292"

apache-spark maven java
1 Answers

Spark creates a folder called as spark-warehouse to save table's data. We can specify the path for that folder using configuration -spark.sql.warehouse.dir.

// While creating SparkSession
SparkSession.builder.config("spark.sql.warehouse.dir", "/path/to/warehouse").getOrCreate()

// Or after creating SparkSession
spark.conf.set("spark.sql.warehouse.dir", "/path/to/warehouse")

// Or while launching application
spark-submit --conf spark.sql.warehouse.dir="/path/to/warehouse"

If you don't specify the folder path then it will be created in the current directory from where you launched spark application. If you have permission issues to create folder then you will get errors.

NOTE: Use spark-submit command to launch spark applications.

user_87733090
Related