Who's managing the memory when an external program calls `Rf_allocXXX` via R's C API interface?

user_5715571 0
Viewed 134

I'm writing an R package in Rust, which communicates with R via its C API interface.

One fundamental question seems tricky to me is the memory management.

So first let me briefly explain how my Rust program communicates with R.

First, on R side, it calls a C dynamic library with .Call(). The C library then is linked to the Rust static library with C compatible ABI.

R scripts is doing some simple works like input validation, and decide which C function to call based on inputs. Then C program passes the calls to underlying Rust functions, which are exposed as C-compatible functions.

It's pretty clear so far, then it gets tricky once Rust has finished computation and needs to send result back.

One option is to directly call Rf_allocXXX functions on Rust side, and store results within it. Then passes the raw pointer of the R object back to C, and then to R.

But it's unclear to me, whether this will result in memory leak.

It seems to me that if Rust calls Rf_allocXXX, the new R objected (SEXP) is created on Rust program's heap memory. When passing the raw pointer out, Rust will not destroy the object. But then what will happen?

Note that the SEXP created by Rust in this way is directly passed back to C and R. There's no re-allocation. So it seems very unclear to me whether this SEXP would be correctly freed by R's GC.

A related question,

What I noticed from Rcpp's source code is, it seems like just calling R's C API for creating its various vectors, which are wrappers of SEXP.

But I'm not yet clear how it deal with memory management. Does it just simply turns the SEXP object back and R will correctly handle GC?

rcpp r
2 Answers

It's all (as always) in Writing R Extensions (and as always not the easiest to find....)

In a nutshell, when you call an R extension package are you essentially required to call R's API and its Calloc() and Free() routines (and variants). Why? Because anything you return to R becomes an R object and is indistinguishable from all other R object and behaves identical. Including, very importantly, when it comes to garbage collection.

And the only way to do that is via R's own allocator. So Rcpp uses it. And creates objects that are in fact indistinguishable. Which makes it all work.

R> Rcpp::cppFunction("IntegerVector foo() { 
+                           IntegerVector v = {1, 2, 3}; return v; }") 
R> foo()
[1] 1 2 3 
R> identical(foo(), c(1L, 2L, 3L))  
[1] TRUE  
R> identical(foo(), 1:3)    
[1] TRUE    
R>

But as a first step, you could just do your thing in Rust to compute results, and then pay the one-time conversion cost (from your object into the SEXP R expects as a result of a .Call()) . "Walk before you run" and all that. Rust bindings would be cool. I presume you know Jeroen and others had already done some work?

user_1433050

Update to this question:

Based on Dirik's answer and after reading Memory Allocation section in Writing R Extensions, now I realised that

  • If a foreign program, potentially in other languages, created a SEXP object via R_allocXXX interface or the equivalence in Rcpp, then R is responsible to free the memory,
    • At the end of the call of .C(), .Call(), and .External()
    • Or when an error is raised
  • But if an R object is manually created via malloc or similar interface, user is responsible to free the memory, regardless of whether this object is created in C, Rcpp, or other foreign programs.

So if we assume R doesn't panic before freeing the memory, then in most cases we should use R_allocXXX bindings to create SEXP, and there will be NO MEMORY LEAK. Once the object is turned to R, it's safe.

But still caution must be made when dealing with Foreign Function Interface (FFI) in general,

  • If we manually allocate the memory and returns a C object to C interface, then we need to make sure neither side of FFI panics before converting the C object to SEXP. Otherwise the memory is not freed.
  • Similarly this applies to when we manually allocated the memory in C program, and sends an object to a foreign program
  • In any cases, manually allocated memory needs to be manually freed.

So in general, whenever possible, we should use R_allocXXX interface for memory safety.

user_57155710
Related