I'd like to test the performance of various approaches in Databricks and am wondering how this is done. Is timeit still the best way? I'm not sure if Databricks being distributed changes anything?
For example:
def test(col1):
#do a thing
def test2(col1):
#do a different thing
df = spark.sql('''select test(col1) as test from table'''
df1 = spark.sql('''select test2(col1) as test from table'''
How can I test which is the more efficient approach?