I have the following scenario : a backend application written in Python that reads from a Cassandra database, then it does some processing and then it sends the data to the front end.
The problem : the operation of initializing a Pandas DataFrame from a Python dict takes too long.
The code that is taking too long and causing some performance issues:
st = time.time()
dataframe = pd.DataFrame(data_to_df)
dataframe = dataframe.astype(DOWNCAST['power'])
print_statistics('orjson', '()', 'data_to_df_perf.py', time.time() - st)
In this case data_to_df is a list, containing around 70.000 entries like :
{"year": 2022, "month": 2, "quarter": 1, "value": 156.99, "day": 16, "hour": 0, "week": 7, "wk_year": 2022, "is_peak": 0}
{"year": 2022, "month": 2, "quarter": 1, "value": 155.38, "day": 16, "hour": 1, "week": 7, "wk_year": 2022, "is_peak": 0}
{"year": 2022, "month": 2, "quarter": 1, "value": 132.78, "day": 16, "hour": 2, "week": 7, "wk_year": 2022, "is_peak": 0}
...
The initial input is a str which gets passed as an input parameter to my function and then gets transformed into the list :
data = orjson.loads(orjson.loads(data))
data_to_df = data['data']
dataframe = pd.DataFrame(data_to_df)
dataframe = dataframe.astype(DOWNCAST[commodity])
In my Python app, when it is deployed in AWS, the operation of transforming data_to_df takes between 1.5 seconds and 2 seconds and this causing a performance problem as this part of the code is called several times.
Output on localmachine :
<class 'list'
orjson () data_to_df_perf.py Execution time: 0.13002800941467285156 s
Does anyone know how to reduce this costly operation ?
Anyone has something different in mind ? Like transforming the list to another data type and then initializing the Pandas DataFrame ?
DataFrame constructor says :
Parameters ---------- data : ndarray (structured or homogeneous), Iterable, dict, or DataFrame Dict can contain Series, arrays, constants, dataclass or list-like objects. If data is a dict, column order follows insertion-order. If a dict contains Series which have an index defined, it is aligned by its index.
Other things that I tried :
st = time.time()
data = pd.read_csv('tests/HPFC_power_data_to_df_2.txt', header = None)
print_statistics('from file', '()', 'test_data_to_df_perf.py', time.time() - st)
st = time.time()
data = dataframe = pd.DataFrame.from_records(data_to_df)
print_statistics('records', '()', 'test_data_to_df_perf.py', time.time() - st)
st = time.time()
dataframe = pd.DataFrame(data_to_df)
print_statistics('default', '()', 'test_data_to_df_perf.py', time.time() - st)
dataframe = dataframe.astype(DOWNCAST['power'])
st = time.time()
df = pd.read_json(json_data_list, orient ='records')
print_statistics('records', '()', 'test_data_to_df_perf.py', time.time() - st)
Output :
from file () test_data_to_df_perf.py execution 0.10300374031066894531
records () test_data_to_df_perf.py execution 0.11999869346618652344
default () test_data_to_df_perf.py execution 0.11696839332580566406
records () test_data_to_df_perf.py execution 0.26199913024902343750
Thanks !