Killing the spark.sql

Viewed 36

I am new to scala and spark both .

I have a code in scala which executes quieres in while loop one after the other.

What we need to do is if a particular query takes more than a certain time , for example # 10 mins we should be able to stop the query execution for that particular query and move on to the next one

for example

    do {
        var f = Future(
       spark.sql("some query")) 
       )
    
    f onSucess {
      case suc - > println("Query ran in 10mins")
    }
    
    f failure {
     case fail -> println("query took more than 10mins")
    }
}while(some condition)

var result = Await.ready(f,Duration(10,TimeUnit.MINUTES))

I understand that when we call spark.sql the control is sent to spark which i need to kill/stop when the duration is over so that i can get back the resources

I have tried multiple things but I am not sure how to solve this. Any help would be welcomed as i am stuck with this.

0 Answers
Related