I have a Notebook running in databricks cluster and it has below piece of code.
FluidConsumptionValues_Table = pd.DataFrame(
{
'Units': [fluid_units, fluid_units,bsfc_units,'$/hr', '$/hr', '$/hr'],
'Mean': [avg_fuelling_mean, avg_def_mean,bsfc_mean,DieselCost_mean,DefCost_mean, FuelCost_mean],
'StdDev': [avg_fuelling_StdDev, avg_def_StdDev,bsfc_StdDev,DieselCost_StdDev, DefCost_StdDev, FuelCost_StdDev],
'Median': [avg_fuelling_Median, avg_def_Median,bsfc_Median,DieselCost_Median,DefCost_Median, FuelCost_Median],
'Max': [avg_fuelling_Max, avg_def_Max,bsfc_Max,DieselCost_Max,DefCost_Max, FuelCost_Max],
'Min': [avg_fuelling_Min, avg_def_Min,bsfc_Min,DieselCost_Min,DefCost_Min, FuelCost_Min],
'Total': [avg_fuelling_Total, avg_def_Total,bsfc_Total,DieselCost_Total,DefCost_Total, FuelCost_Total],
'20th Percentile': [avg_fuelling_20_Percentile, avg_def_20_Percentile,bsfc_20_Percentile,DieselCost_20_Percentile,DefCost_20_Percentile, FuelCost_20_Percentile],
'80th Percentile': [avg_fuelling_80_Percentile, avg_def_80_Percentile,bsfc_80_Percentile,DieselCost_80_Percentile,DefCost_80_Percentile, FuelCost_80_Percentile],
'GDC': [avg_fuelling_GDC, avg_def_GDC,bsfc_GDC,DieselCost_GDC,DefCost_GDC, FuelCost_GDC]
},
index=['Diesel Consumption', 'DEF Consumption', 'BSFC', 'Diesel Cost*', 'DEF Cost*', 'Total Cost*'])
Values mentioned in Mean,Stddev,Total are calculated(prior to this step) from a dataframe which has 800 million records.
As soon the code block reaches this step mentioned above, driver memory exceeds and throws an out of memory exception. Is there a way I can mitigate this? How can I distribute this workload across executors instead of driver? Your help is much appreciated.