In my R code, I use the MixedModels Julia package.
I integrate Julia in R using the JuliaCall package.
I work with very large datasets (~1 GB, ~4x10^6 observations) and at the modeling step (mixed models) a lot of RAM is allocated (~80 GB), most of it does not return to the system after the end of calculations.
How can I close the connections to Julia from under R (or maybe close Julia) after obtaining the modeling results in the R environment? At the same time, the R session should not be interrupted, since after calculating the mixed models, other functions should be executed in pipeline.
This action can help release certain amount of memory to the system (I ran code on pure Julia and after Julia exit about 70 GB was returned to OS. When I run code on R, RAM occupied by Julia is not returned to OS after pipeline calculations are completed)
(Platform: x86_64-pc-linux-gnu).
Close connections to Julia from under R
Viewed 96
1 Answers
If you called the Julia code from R as a function, you can return a tuple of the parameters you have calculated to R without the data. If so, then, prior to returning from Julia, you can set all of the references to the data and other large memory referencing variables to 0 and then collect garbage, eg. if
a = rand(1000, 1000, 100)
a = 0 # destroy reference since Julia reference counts
then call the garbage collector manually:
GC.gc()
This assumes that you don't need the internal data that the Julia variables reference any more, which is likely if all you wanted was the model's parameters.
Related