How can I pass a RDD from scala to python?

user_135135 0
Viewed 93

I'm trying to pass a Spark RDD from Scala to Python so that I can then call functions on it from PySpark.

My scala code returns rdd.toJavaRDD(), which I then read in python via:

scala_res = sc._jvm.com.package.Class.method()
from pyspark.rdd import RDD
python_rdd = RDD(scala_res, sc._sc)

This initially appears to work, but if I call functions on this RDD, like python_rdd.count(), it will start a Spark job to get the result but error out with the error: org.apache.spark.SparkException: Unexpected element type class java.lang.Long.

It seems maybe I'm missing some conversion step from java to python RDD?

I'm using Scala 2.11, and Spark 2.2.0.

apache-spark scala python
1 Answers

You need to use some gateway from JVM generated code to Python like Py4J gateway:

  • Build a jar file with a code that yields your RDD.
  • Include this jar file to classpath w\ --driver-class-path for spark-shell or spark-submit
  • Instantiate a JVM instance and reuse necessary Scala code:
jvm = sc._jvm
rdd_result = jvm.foo.Bar.baz()
user_68606820
Related