I have tried every possible approach on stackoverflow but not able to solve this. I am trying to find the time for the following operation:
long startTime=System.currentTimeMillis()
Dataset<Row> readDataFrame= spark.read()
.format("jdbc")
.option("url", drivingUrl)
.option("dbtable", drivingQuery)
.option("driver", driver)
.load();
long endTime=System.currentTimeMillis();
This gives wrong results due to lazy execution of spark. I can't have a separate action for this, since I already have one at the end of the application. Adding more will increase the performance cost.
Approaches I tried
1) I have tried other approaches like spark.time() etc, but they are giving incorrect results due to the lazy execution of spark.
2) I was looking into the various methods in listener as well, but most were for stages/jobs. There is onOtherEvent but it gives me all the events that happen inside it and that too stage to stage. Also, I tried using SQL Listener, but you would need to know executor id for that, which I don't think we can know since it's spark who allocates executors.
3) I have tried using QueryExecutionListener but it gives me the entire stage time, which I don't want. I want the time only for loading the above.
Can anyone help me out here? I am very new to the concept of listeners and I am not able to find any good resources online for my purpose.