Execute PySpark code from a Java/Scala application

Viewed 101

Is there a way to execute PySpark code from a Java/Scala application on an existing SparkSession?

Specifically, given a PySpark code that receives and returns pyspark dataframe, is there a way to submit it to Java/Scala SparkSession and get back the output dataframe:

String pySparkCode = "def my_func(input_df):\n" +
    "    from pyspark.sql.functions import *\n" +
    "    return input_df.selectExpr(...)\n" +
    "            .drop(...)\n" +
    "            .withColumn(...)\n"

SparkSession spark = SparkSession.builder().master("local").getOrCreate()

Dataset inputDF = spark.sql("SELECT * from my_table")

outputDf = spark.<SUBMIT_PYSPARK_METHOD>(pySparkCode, inputDF)

0 Answers
Related