I'm developing application which should access data from several different kind of data stores such as HBase or ElasticSearch. My plan is to use Spark SQL through Spark DataFrame.
Now here's the problem. Sometimes system user will request to run long time query to transform those data and in that case (for example longer than 1 min?) I want to handle that request in a different way. But for that, I need to know how long the query seems to run.
Is there any way to 'estimate' running time of Spark SQL query before actually running it?