Adding custom jars to pyspark in jupyter notebook

user_2299864 0
Viewed 28475

I am using the Jupyter notebook with Pyspark with the following docker image: Jupyter all-spark-notebook

Now I would like to write a pyspark streaming application which consumes messages from Kafka. In the Spark-Kafka Integration guide they describe how to deploy such an application using spark-submit (it requires linking an external jar - explanation is in 3. Deploying). But since I am using Jupyter notebook I never actually run the spark-submit command, I assume it gets run in the back if I press execute.

In the spark-submit command you can specify some parameters, one of them is -jars, but it is not clear to me how I can set this parameter from the notebook (or externally via environment variables?). I am assuming I can link this external jar dynamically via the SparkConf or the SparkContext object. Has anyone experience on how to perform the linking properly from the notebook?

spark-streaming jupyter-notebook pyspark apache-kafka python-3.x
5 Answers

Indeed, there is a way to link it dynamically via the SparkConf object when you create the SparkSession, as explained in this answer:

spark = SparkSession \
    .builder \
    .appName("My App") \
    .config("spark.jars", "/path/to/jar.jar,/path/to/another/jar.jar") \
    .getOrCreate()
user_41137700

In case someone is the same as me: I tried all above solutions and none of them works for me. What I'm trying to do is to use Delta Lake in the Jupyter notebook.

Finally I can use from delta.tables import * by calling SparkContext.addPyFile("/path/to/your/jar.jar") first. Though in the spark official docs, it only mentions adding .zip or .py file, but I tried .jar and it worked perfectly.

user_41018420

for working on jupyter-notebook with spark you need to give the location of the external jars before the creation of sparkContext object. pyspark --jars youJar will create a sparkcontext with location of external jars

user_25302670
Related