How to set Configuration of SparkContext correctly in Jupyter Notebook?

user_14868005 0
Viewed 2875

I am new with spark, and I try to configurate SparkContext, but unfortunately I got error messages..

I wrote this code:

from pyspark import SparkConf,SparkContext
from pyspark.streaming import StreamingContext
from pyspark.sql import Row,SQLContext
import sys
import requests

# create spark configuration

conf = SparkConf()
conf.setAppName("TwitterStreamApp")  

# create spark context with the above configuration
sc = SparkContext(conf=conf)

And I got this error:

Py4JError                                 Traceback (most recent call last)
<ipython-input-97-b0f526d72e5a> in <module>
      1 # create spark context with the above configuration
----> 2 sc = SparkContext(conf=conf)

~\anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    133             # If an error occurs, clean up in order to allow future SparkContext creation:
    134             self.stop()
--> 135             raise
    136 
    137     def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

~\anaconda3\lib\site-packages\pyspark\context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    211         self.pythonVer = "%d.%d" % sys.version_info[:2]
    212 
--> 213         if sys.version_info < (3, 6):
    214             with warnings.catch_warnings():
    215                 warnings.simplefilter("once")

~\anaconda3\lib\site-packages\py4j\java_gateway.py in __getattr__(self, name)
   1528                     answer, self._gateway_client, self._fqn, name)
   1529         else:
-> 1530             raise Py4JError(
   1531                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1532 

Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does not exist in the JVM

Also, in system ENV I added JAVA_HOME, SPARK_HOME.. but it is not working.

jupyter-notebook pyspark apache-spark python
1 Answers

I think the way you're setting up you end up running multiple SparkContexts at once.

Try this simple setup instead:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('TwitterStreamApp').getOrCreate()

if you are NOT streaming, assuming you are reading a csv file with headers

staticDF = spark.read.csv('source/file/path/here', header = True, inferSchema = True)

if you ARE streaming, say csv format again

streamingDF = spark.readStream \
   .schema(provide schema here)
   .option('....') // whatever your options are
   .csv('source/file/path/here')

Your might want to get used to the idea of providing or building a schema when you read your data, it helps your processing speed than when spark tries to infer it.

user_148872700
Related