Spark context not invoking stop() and shutdown hook not called when saving to s3

Viewed 726

I'm reading data from parquet files, processing it, and then saving the result to S3. The problem is relevant only to the last (saving) part.

When running locally, after saving the data, the sparkContext's stop() is not invoked, and the shutdown hook is not called. I need to manually invoke/call them by clicking the IDE's (IntelliJ) stop button.

When saving to a local folder, the process finishes correctly. When running on EMR, the process finishes correctly.

Changing the format/header/etc. doesn't solve the problem. Removing any/all of the transformations/joins doesn't solve the problem.

Problem occurs when using both dataframes and datasets.

EDIT: Following comments, tried using both s3:// and s3a:// - as well as separating the commands, but the issue remains.

Example code:

package test

import org.apache.spark.sql.SparkSession

object test {
  def main(args: Array[String]): Unit = {
    SparkSession
      .builder
      .appName("s3_output_test")
      .master("local[*]")
      .getOrCreate
      .read
      .parquet("path-to-input-data")
      // .transform(someFunc)
      // .join(someDF)
      .coalesce(1)
      .write
      .format("csv")
      .option("header", true)
      .save("s3a://bucket-name/output-folder")

    sparkSession.stop()
  }
}

Any ideas on a solution or how to further debug the issue will be greatly appreciated, thanks!

0 Answers
Related