I have an issue with the R system() function (for running an OS command from within R) that only arises when the R session uses up more than some fraction of the available RAM (maybe ~75% in my case), even though there is plenty of RAM available (~15GB in my case) and the same OS command can be easily run at the same time from a terminal.
System info:
64GB RAM PC (local desktop PC, not cloud-based or cluster)
Ubuntu 18.04.1 LTS - x86_64-pc-linux-gnu (64-bit)
R version 3.5.2 (executed directly, not e.g. via docker)
This example demonstrates the issue. The size of the data frame d needs to be adjusted to be as small as possible and still provoke the error. This will depend on how much RAM you have and what else is running at the same time.
ross@doppio:~$ R
R version 3.5.2 (2018-12-20) -- "Eggshell Igloo"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> n <- 5e8
> d <- data.frame(
+ v0 = rep_len(1.0, n),
+ v1 = rep_len(1.0, n),
+ v2 = rep_len(1.0, n),
+ v3 = rep_len(1.0, n),
+ v4 = rep_len(1.0, n),
+ v5 = rep_len(1.0, n),
+ v6 = rep_len(1.0, n),
+ v7 = rep_len(1.0, n),
+ v8 = rep_len(1.0, n),
+ v9 = rep_len(1.0, n)
+ )
> dim(d)
[1] 500000000 10
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 260857 14.0 627920 33.6 421030 22.5
Vcells 5000537452 38151.1 6483359463 49464.2 5000559813 38151.3
> system("free -m", intern = FALSE)
Warning messages:
1: In system("free -m", intern = FALSE) :
system call failed: Cannot allocate memory
2: In system("free -m", intern = FALSE) : error in running command
The call to gc() indicates R has allocated ~38GB out of 64 GB RAM and running free -m in a terminal at the same time (see below) shows that the OS thinks there is ~16GB free.
ross@doppio:~$ free -m
total used free shared buff/cache available
Mem: 64345 44277 15904 461 4162 18896
Swap: 975 1 974
ross@doppio:~$
So free -m can't be run from within R because memory cannot be allocated, but free -m can be run at the same time from a terminal, and you would think that 15GB would be enough to run a light-weight command like free -m.
If the R memory usage is below some threshold then free -m can be run from within R.
I guess that R is trying allocate an amount of memory for free -m that is more than actually needed and depends on the amount of memory already allocated. Can anyone shed some light on what is going on here?
Thanks