I'm reading data from parquet files, processing it, and then saving the result to S3. The problem is relevant only to the last (saving) part.
When running locally, after saving the data, the sparkContext's stop() is not invoked, and the shutdown hook is not called. I need to manually invoke/call them by clicking the IDE's (IntelliJ) stop button.
When saving to a local folder, the process finishes correctly. When running on EMR, the process finishes correctly.
Changing the format/header/etc. doesn't solve the problem. Removing any/all of the transformations/joins doesn't solve the problem.
Problem occurs when using both dataframes and datasets.
EDIT: Following comments, tried using both s3:// and s3a:// - as well as separating the commands, but the issue remains.
Example code:
package test
import org.apache.spark.sql.SparkSession
object test {
def main(args: Array[String]): Unit = {
SparkSession
.builder
.appName("s3_output_test")
.master("local[*]")
.getOrCreate
.read
.parquet("path-to-input-data")
// .transform(someFunc)
// .join(someDF)
.coalesce(1)
.write
.format("csv")
.option("header", true)
.save("s3a://bucket-name/output-folder")
sparkSession.stop()
}
}
Any ideas on a solution or how to further debug the issue will be greatly appreciated, thanks!