I am creating a spark job that requires a column to be added to a dataframe using a function written in python. The rest of the processing is done using Scala.
I have found examples of how to call a Java/Scala function from pyspark:
- https://community.hortonworks.com/questions/110844/is-it-possible-to-call-a-scala-function-in-pythonp.html
- http://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html
The only examples I have found to send data the other way is using pipe
Is it possible for me to send the entire dataframe to a python function, have the function manipulate the data and add additional columns and then send the resulting dataframe back to the calling Scala function?
If this isn't possible my current solution is to run a pyspark process and call multiple Scala functions to manipulate the dataframe, this isn't ideal.