IPython Kernel dies when trying to merge large list of dataframes

Viewed 2951

I'm trying to merge over 100 dataframes into a single dataframe using Pandas. When I try to merge them all, the iPython kernel dies.

I've already tried uninstalling and re-installing ipython and jupyter notebook as suggested here. I've run this code in Spyder and Jupyter Notebook and the kernel has died in both. I installed both through Anaconda Navigator on Mac OSX.

In the terminal, the only error I get is:

[I 09:09:34.341 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports kernel 89dadad3-97c9-42c6-bcc9-9cedc4a6b6ff restarted

The code that I'm using to merge all the dataframes is

mega_df = reduce((lambda x, y: pd.merge(x, y, how = 'outer', on = col)), df_list)

I'd expect the result to be a dataframe with around 4500 rows and 12000 columns.

0 Answers
Related