Efficiently unpacking many Json files and combining into multiple dataframes

Question

I have a massive number (I believe in the tens of thousands) of json files structured as follows:

{'data': 
{
"1":{"A":123,"B":456, "C":789},
"2": {"A":3423,"B":356, "C":549},
 ..., 
"4000":{"A":765,"B":355, "C":321}
}, 
"timestamp":timestamp1}

My end goal is three pandas dataframes structured such that each parameter denoted as a letter above gets its own dataframe like this:

dfA =

	1	2	...	4000
timestamp1	123	3432	...	765
[timestamp from next json]	[from next json]	[from next json]	...	[from next json]

dfB =

	1	2	...	4000
timestamp1	456	356	...	355
[timestamp from next json]	[from next json]	[from next json]	...	[from next json]

dfC =

	1	2	...	4000
timestamp1	789	549	...	321
[timestamp from next json]	[from next json]	[from next json]	...	[from next json]

I have code which I believe works and achieves this, however I'm not very good with pandas and I know it is extremely slow compared to what it could be.

For each json my code looks something like this:

with open(path) as json_file:
     data = json.load(json_file)
df = pd.DataFrame.from_dict(data['data']).T
A_series = df['A']
B_series = df['B']
C_series = df['C']
    
A_series.name = json_timestamp #timestamp is known before reading each json, but is also present inside each json
B_series.name = json_timestamp #adding these lines make json_timestamp the index in the df
C_series.name = json_timestamp


Adf = Adf.append(A_series.to_dict(),ignore_index=true)
Bdf = Bdf.append(B_series.to_dict(),ignore_index=true)
Cdf = Cdf.append(C_series.to_dict(),ignore_index=true)

This code is looking like its going to take about a week to run- are there changes I can make to make this more efficient or elegant?

user_12939557 · Answer

The main bottleneck of your code lies in the append. Indeed, pandas return a full copy of the input dataframe with a new added line. This means that the complexity is O(n * n) where n is the number of line of the final dataframe -- i.e. the execution time is quadratic to the number of json files added. Copying the dataframe for each newly added frame is very expensive since the dataframe takes at least several hundreds of MiB in memory.

You can solve that by appending the lines to a list and then call pd.join. This performs the operation in linear time. I expect the computation time to be about 3 order of magnitude faster regarding your input.

Efficiently unpacking many Json files and combining into multiple dataframes

1 Answers