Pandas concat function timing out when combining large dataframes

user_13746024 0
Viewed 125

I am attempting to combine three dataframes that total over 120,000 features and 206,000 rows. All three of the dataframes are document-term matrices. Here is the code: 

new_df = pd.concat([df_1, df_2], axis = 1, sort = False)

As you can see I am using the concat function to combine the three dataframes across the horizontal axis. Currently, Jupyter Notebook times out and restarts the kernel after a while of this code running (I assume because of the number of features). Any insight into ways to reduce the size of these dataframes or combine them would be very much appreciated.

pandas dataframe nltk python
3 Answers

Did you check the error message? Does it say memory error? There is a chance that you do not have enough memory to hold all three + the combined dataframe. Besides, You do not need that many features. I highly recommend using PCA, SVD or any other dimentionality reduction algorithms and only take the first let's say 50 or so components to use in the model.

user_58547310

Try SFrame from https://github.com/apple/turicreate:

pip install -U turicreate

Instead of 

import pandas as pd
df = pd.read_csv(...)

Do this with SFrame, https://turi.com/products/create/docs/generated/graphlab.SFrame.html#graphlab.SFrame:

import turicreate as tc
sf1 = tc.SFrame.read_csv(...)
sf2 = tc.SFrame.read_csv(...)

Then to concat the SFrames:

new_sf = sf1.append(sf2)

Q: Why does SFrame work for large no. of rows of data but not pandas?

A: https://www.youtube.com/watch?v=wUOMaKYtaMY

Q: Then why is Pandas more popular than SFrame?

A: *Shrugs.

user_6105690

Firstly, your example has two dataframes. Secondly, your dataframe will have 20 BILLION entries, and I assume each one is more than a char, so we are talking about 100GB of data at least. So, there is really not much you can do, other than (a) getting a machine with a lot of memory or (b) doing dimensionality reduction as suggested in the comments.

user_29772560
Related