Import custom scala object in jupyter notebook with sparkContext.addFile

user_4967382 0
Viewed 593

I've a Jupyter Notebook where i'm trying to import a keys as a string from an outside function. In pyspark all work without problems with this code

Config.py

credentials = {"item1": "prueba"}

def get_key(storage_account):
    return credentials[storage_account]

Call in the notebook:

spark.sparkContext.addFile("/home/joaco/data-experiment/notebooks_fpatito/readExternalFilesNotebook/pySpark/Config.py")
import Config
key = Config.get_key("item1")
print(key)

enter image description here

But, when I'm trying to do the same in Scala Spark the notebook throw me a not found error

"not found: value testMap"

testMap.scala

object testMap {

  val mapExample = Map(
    "item1" -> "item1value",
    "item2" -> "item2value",
    "item3" -> "item3value",
  )

}

I'm doing the same sparkContext.addfile:

spark.sparkContext.addFile("/home/joaco/data-experiment/notebooks_fpatito/readExternalFilesNotebook/Spark/testMap.scala")

And this import: import testMap._

But the class is not found

enter image description here

jupyter-notebook apache-spark scala
1 Answers

Python is an interpreted language but Scala is not, hence the Notebook throws Error when you try to load a Scala file like you did for a Python file. You need to compile the Scala file to a class and make it available as a JAR. Then you can add the JAR to your path as follows:

  1. --jars option on spark commandline
  2. spark.sparkContext.addJar(/path/to/JAR/file)
user_57426620
Related