how to run a large feather R database

Admin 1
Viewed 91

I am trying to use the code below to import a 4GB database (about 9,000,000 obs and 100 variables) into R using a windows 10 with 8GB RAM

library(feather)
memory.limit(size=99999)

rais_transp = read_feather('rais_transp.feather')

but every time I try to run it I get the following error message

"r encountered a fatal error: the session was terminated"

I have tried to download only one column but still I get the same message and the session is restarted

rais_transp = read_feather('rais_transp.feather', columns=c('black'))

I used to be able to handle this database on my computer, but now I can't run it anymore.

Somebody to help me?

Thanks

feather r
1 Answers

I had a similar problem with a .csv and I followed the next steps:

As you are using windows you should install cygwin to preprocess the file. Once installed you can split your file in smaller chuncks using the cygwin shell and writting on it:

split -b100m rais_transp.csv

You will have to convert it to a csv as danh pointed. The command -b100m means that your new file chunks will have a size of 100MB. As to feather size is smaller than a csv file maybe you need to make smaller chuncks. You can get chuncks of 1MB for example with -b 1024k.

You can find useful information related in the points 5.3.2 and 6.6 of the book Efficient R Programming.

Here you can find the link to check this points: https://csgillespie.github.io/efficientR/data-carpentry.html#working-with-databases

user_141521350
Related