Your problem is that your code is dumping everything into the global environment (your Rmd's environment). When I work with larger data I tend to wrap my analysis into a function inside of the chunk, instead of writing it as if it were an R script. I'll give a simple example to illustrate:
Imagine the following as a script:
r <- load_big_data()
train <- r[...]
test <- r[...]
fit <- lm(x ~ y, data = train)
summary(fit)
If this is your chunk, all of these variables are left in the environment when your model run is completed. However, if you encapsulate your work in a function, once the function is done the interim variables are typically released from memory.
r <- load_big_data()
myFun <- function(r) {
train <- r[...]
test <- r[...]
fit <- lm(x ~ y, data = train)
return(summary(fit))
}
Now, instead of having test, train, and fit in the workspace as the Rmd is knit, you only have r in your workspace (and myFun, which is practically costless)
Bonus: You'll find you can reuse these functions the longer your analysis gets!
Updates
RE: cache = TRUE
To answer your subsequent question. cache=TRUE will load from an RDS file instead of re-running the code chunk. It could be effective as a tool to minimize memory usage -- but you'll still need to remember to remove data from the workspace as it loads from the cache rather than running. You should think of this as saving time, rather than saving memory unless you manually clean up.
RE: gc()
gc, or "garbage collection" is a trigger for a process that R runs frequently by itself to collect and dump memory that it has held temporarily but is no longer using. Garbage collection in R is quite good, but using gc can help release memory in more stubborn situations. Hadley does a good job of summarizing here: http://adv-r.had.co.nz/memory.html. With that said, it's rarely ever the silver bullet and typically, if you feel like you need to use it you either need to rethink your approach or rethink your hardware, or both.
re: External resources
This may sound a bit flippant, but sometimes loading up another machine that's much larger than yours to finish the work is wildly less expensive (time == $) than fixing a memory leak. Example: an R5 with 16 cores and 128GB of RAM is $1 per hour. The calculus on your time is often quite lucrative.
