pyspark.sql.utils.IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuild in windows 10

user_1080145 0
Viewed 20800

I have installed spark 2.2 with winutils in windows 10.when i am going to run pyspark i am facing bellow exception

pyspark.sql.utils.IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder'

I have already tried permission 777 commands in tmp/hive folder as well.but it is not working for now

winutils.exe chmod -R 777 C:\tmp\hive

after applying this the problem remains same. I am using pyspark 2.2 in my windows 10. Her is spark-shell env enter image description here

Here is pyspark shell enter image description here

Kindly help me to figure out Thankyou

pyspark apache-spark
8 Answers

I had the same problem using the command 'pyspark' as well as 'spark-shell' (for scala) in my mac os with apache-spark 2.2. Based on some research I figured its because of my JDK version 9.0.1 which does not work well with Apache-Spark. Both errors got resolved by switching back from Java JDK 9 to JDK 8.

Maybe that might help with your windows spark installation too.

user_70964110

Try this . It worked for me!. Open up a command prompt in administrator mode and then run the command 'pyspark'. This should help open a spark session without errors.

user_10557040

I also come across the error in Unbuntu 16.04:

raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder'

this is because I have already run ./bin/spark-shell

So, just kill that spark-shell, and re-run ./bin/pyspark

user_85824800

I also come across the error in MacOS10, and I solved this by use Java8 instead of Java9.

When Java 9 is the default version getting resolved in the environment, pyspark will throw error below and you will see name 'xx' is not defined error when trying to access sc, spark etc. from shell / Jupyter.

more details you can see this link

user_67922470

You must have hive-site.xml file in the spark configuration directory. Change the port from 9000 to 9083 resolved the problem for me.

Please ensure that the property is updated in both the hive-site.xml files which would be placed under hive config and spark config directory.

<property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>   </property>

For me in ubuntu, the location for hive-site.xml are:

/home/hadoop/hive/conf/

and

/home/hadoop/spark/conf/

user_91035050
Related