Problem while creating SparkSession using pyspark

Question

I am new to Spark. I am trying to create a Spark session from pyspark.sql in order to load a .csv file. However, every time I am trying to execute the 2nd line (as shown below), the command keeps on executing for hours & never seems to generate the other lines of the code. The code is as follows:

from pyspark.sql import SparkSession
sp = SparkSession.builder.appName("solution").config("spark.some.config.option", "some-value").getOrCreate()
df = sp.read.csv('walmart_stock.csv', header= True, inferSchema= True)
df.columns

Also, if I kill the kernel after waiting for a long time, the following exception appears:

 in 
----> 1 sp = SparkSession.builder.appName("solution").config("spark.some.config.option", "some-value").getOrCreate()

~\anaconda3\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    184                             sparkConf.set(key, value)
    185                         # This SparkContext may be an existing one.
--> 186                         sc = SparkContext.getOrCreate(sparkConf)
    187                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    188                     # by all sessions.

~\anaconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    369         with SparkContext._lock:
    370             if SparkContext._active_spark_context is None:
--> 371                 SparkContext(conf=conf or SparkConf())
    372             return SparkContext._active_spark_context
    373 

~\anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    126                 " is not allowed as it is a security risk.")
    127 
--> 128         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    129         try:
    130             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

~\anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    318         with SparkContext._lock:
    319             if not SparkContext._gateway:
--> 320                 SparkContext._gateway = gateway or launch_gateway(conf)
    321                 SparkContext._jvm = SparkContext._gateway.jvm
    322 

~\anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
    100             # Wait for the file to appear, or for the process to exit, whichever happens first.
    101             while not proc.poll() and not os.path.isfile(conn_info_file):
--> 102                 time.sleep(0.1)
    103 
    104             if not os.path.isfile(conn_info_file):

Can you kindly suggest what is the problem?

user_16204064 · Answer

I had same issue and even I wasn't able to create a spark context so through some research I got to know that our installed Spark version should match with our Pyspark version which is pretty weird but it is true and I did the same in my laptop and surprisingly it worked.

To check the spark version you have enter(in cmd):

spark-shell --version

And, to check Pyspark version enter(in cmd):

pip show pyspark

After that,
Use the following code to create SparkContext :

conf = pyspark.SparkConf() 
sqlcontext = pyspark.SparkContext.getOrCreate(conf=conf)
sc = SQLContext(sqlcontext)

after that run :

df.columns

This way it worked for me I hope it does for you too.

Problem while creating SparkSession using pyspark

1 Answers