Say I have a Spark job that does
val df1 = spark.read.csv(file1)
val df2 = spark.read.csv(file1)
df1.filter(...).join(df2, df1('id') === df1('fkey'),"inner"))
I know that I could stop the job with spark.stop() but is it possible to pause such a Spark job on a given action, say file read df1, and later resume?