I have a code that generates several very large dataframes in a loop. Each one has around 300 million rows so I run out of memory before the loop is over. I am trying to export each dataframe once it is constructed within the loop and then remove it to free up space in my R environment before I start constructing the next.
The issue is how to export these very large datasets. I tried using fwrite from the data.table package but when I open the csv file I get an empty csv file called Book1 instead. I also tried saving it as a dta file using write.dta from the foreign package but Stata tells me it is corrupted when I try opening it.