CSV to Feather in Pandas with slicing Rows

user_4186517 0
Viewed 7896

I am processing a huge dataset (50 million rows) in CSV. I am trying to slice it and save it as Feather Format in order to save some memory while loading the feather format later.

As a workaround, I loaded the data in chunks as CSV file and later merged it into one data frame.

This is what I have tried so far:

df[2000000:4000000].to_feather('name')

I have got the following error:

ValueError: feather does not support serializing a non-default index for the index; you can .reset_index() to make the index into column(s)

Then I tried to reset the index but still, I get the same error.

feather pandas python
2 Answers

Try with .loc :

df.loc[2000000:4000000].reset_index().to_feather("./myfeather.ftr")

You'll have to reset the indexes to save the datataframe to feather format. Works for me.

user_152303100

Save the required slice of the data to CSV df.to_csv(), load the data again from the CSV and then save to feather format. This method worked for me

user_128609660
Related