Pyspark session error: "requirement failed: key cannot be null"

Question

I try to start a Spark Session on Jupyter Notebook (Spark 3.2., Hadoop 3.2., Jupyter NB 6.1.4., Python 3.8.5).

I set up the environment:

%env
SPARK_HOME = "C:/Spark/spark-3.2.0-bin-hadoop3.2"
PATH = "C:/Spark/spark-3.2.0-bin-hadoop3.2/bin"
PYSPARK_PYTHON = "C:/Users/xxx/anaconda3/python.exe"
PYSPARK_DRIVER_PYTHON = "C:/Users/xxx/anaconda3/Scripts/jupyter.exe"
PYSPARK_DRIVER_PYTHON_OPTS = "notebook"

And then try to start my session

import sys
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from pyspark.sql.types import ArrayType, DoubleType, BooleanType
spark = SparkSession.builder.appName("Test").config().getOrCreate()

But I have this error:

---------------------------------------------------------------------------
IllegalArgumentException                  Traceback (most recent call last)
 in 
      5 from pyspark.sql.types import StructType, StructField, StringType, IntegerType
      6 from pyspark.sql.types import ArrayType, DoubleType, BooleanType
----> 7 spark = SparkSession.builder.appName("Test").config("spark.some.config.option","some-value").getOrCreate() 

C:\Spark\spark-3.2.0-bin-hadoop3.2\python\pyspark\sql\session.py in getOrCreate(self)
    231                     session = SparkSession(sc)
    232                 for key, value in self._options.items():
--> 233                     session._jsparkSession.sessionState().conf().setConfString(key, value)
    234                 return session
    235 

C:\Spark\spark-3.2.0-bin-hadoop3.2\python\lib\py4j-0.10.9.2-src.zip\py4j\java_gateway.py in __call__(self, *args)
   1307 
   1308         answer = self.gateway_client.send_command(command)
-> 1309         return_value = get_return_value(
   1310             answer, self.gateway_client, self.target_id, self.name)
   1311 

C:\Spark\spark-3.2.0-bin-hadoop3.2\python\pyspark\sql\utils.py in deco(*a, **kw)
    115                 # Hide where the exception came from that shows a non-Pythonic
    116                 # JVM exception message.
--> 117                 raise converted from None
    118             else:
    119                 raise

IllegalArgumentException: requirement failed: key cannot be null

Any idea?

Pyspark session error: "requirement failed: key cannot be null"

0 Answers