Using listeners for calculating datatbase load time in spark

Viewed 50

I have tried every possible approach on stackoverflow but not able to solve this. I am trying to find the time for the following operation:

long startTime=System.currentTimeMillis()
Dataset<Row> readDataFrame= spark.read()
                .format("jdbc")
                .option("url", drivingUrl)
                .option("dbtable", drivingQuery)
                .option("driver", driver)
                .load();
long endTime=System.currentTimeMillis();

This gives wrong results due to lazy execution of spark. I can't have a separate action for this, since I already have one at the end of the application. Adding more will increase the performance cost.

Approaches I tried

1) I have tried other approaches like spark.time() etc, but they are giving incorrect results due to the lazy execution of spark.

2) I was looking into the various methods in listener as well, but most were for stages/jobs. There is onOtherEvent but it gives me all the events that happen inside it and that too stage to stage. Also, I tried using SQL Listener, but you would need to know executor id for that, which I don't think we can know since it's spark who allocates executors.

3) I have tried using QueryExecutionListener but it gives me the entire stage time, which I don't want. I want the time only for loading the above.

Can anyone help me out here? I am very new to the concept of listeners and I am not able to find any good resources online for my purpose.

0 Answers
Related