How should I free a C# byte[] allocated in Rust?

user_1191181 0
Viewed 556

I have a Rust function that passes a byte array to C#:

#[no_mangle]
pub extern "C" fn get_bytes(len: &mut i32, bytes: *mut *mut u8) {
    let mut buf : Vec<u8> = get_data();
    buf.shrink_to_fit();

    // Set the output values
    *len = buf.len() as i32;
    unsafe {
        *bytes = buf.as_mut_ptr();
    }

    std::mem::forget(buf);
}

From C#, I can call it without crashing. (In lieu of a crash, I assume this is correct but am not 100% sure):

[DllImport("my_lib")] static extern void get_bytes(ref int len, 
    [MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 0)] ref byte[] bytes);

void test()
{
   int len = 0;
   byte[] bytes = null;
   get_bytes(ref len, ref bytes);
}

I then make use of bytes, but I understand that this memory needs to be freed by Rust. So I have another Rust function to free it:

#[no_mangle]
pub extern "C" fn free_bytes(len: i32, bytes: *mut *mut u8) {
    // also tried with: -------------- bytes: *mut u8
    assert!(len > 0);

    // Rebuild the vec
    let v = unsafe { Vec::from_raw_parts(bytes, len as usize, len as usize) };

    //println!("bytes to free: {:?}", v);

    drop(v); // or it could be implicitly dropped
}

And the corresponding C#. Making the call crashes my app:

[DllImport("my_lib")] extern void free_bytes(int len, ref byte[] bytes);

void test()
{
   int len = 0;
   byte[] bytes = null;
   get_bytes(ref len, ref bytes);

   // copy bytes to managed memory
   bytes[] copy = new byte[len];
   bytes.CopyTo(copy, 0);
   // free the unmanaged memory
   free_bytes(len, ref bytes); // crash occurs when executing this function
}

I see that Vec::from_parts_raw "is highly unsafe". Because "capacity needs to be the capacity that the pointer was allocated with.", I also tried passing the capacity between Rust & C# without shrink_to_fit to preserve length & capacity. That also crashed.

I assume from_parts_raw recovers the existing memory on the heap, but I noticed that the byte content in C# (shown in Visual Studio) doesn't match the contents in Rust (via the 'bytes to free' println). So is my error in how I'm recovering the Vec<u8> to be freed, in the types Rust is accepting (eg, *mut u8 vs *mut *mut u8), in my C# DllImport, somewhere else?

rust ffi c#
1 Answers

The Main Problem

A byte*/*mut u8 and a byte[] are different kinds of object. The latter must point to memory managed by the .NET GC. So while it is possible to view a byte[] as a byte* (while it is pinned), you cannot view an arbitrary byte* as a byte[].

I'm not entirely sure what the marshaller is doing in your case, but it is probably something like this:

  • Allocate a pointer-sized space, initialized to the null pointer.
  • Call the rust method with a pointer to this space as the second parameter.
  • Interpret the updated contents of that space as a pointer to a C-style byte array.
  • Copy the contents of this array to a newly-allocated managed array.
  • Place that managed array in the C# local bytes.

As you can see, the array you get in bytes is a fresh managed array, with no lasting relation to the pointer written to *bytes by Rust. So of course attempting to call free_bytes on bytes will fail, since it will be marshalled as a pointer to memory managed by the .NET GC and not Rust.

A secondary problem

If you intend to free the memory via P/Invoke, there is no way to get around passing the capacity to C# and keeping it around. This is because Vec::shrink_to_fit is not guaranteed to reduce capacity to len, as indicated by the documentation. And you must have the correct capacity in order to call Vec::from_raw_parts.

Solution

The only reasonable way to pass ownership of a Vec to other code is with functions like these on the Rust side.

#[no_mangle]
pub unsafe extern "C" fn get_bytes(len: *mut i32, capacity: *mut i32) -> *mut u8 {
    let mut buf: Vec<u8> = get_data();

    *len = buf.len() as i32;
    *capacity = buf.capacity() as i32;

    let bytes = buf.as_mut_ptr();
    std::mem::forget(buf);
    return bytes;
}

#[no_mangle]
pub unsafe extern "C" fn free_bytes(data: *mut u8, len: i32, capacity: i32) {
    let v = Vec::from_raw_parts(bytes, len as usize, capacity as usize);
    drop(v); // or it could be implicitly dropped
}

And on the C# side, you would have something like this:

[DllImport("my_lib")] 
static extern IntPtr get_bytes(out int len, out int capacity);

[DllImport("my_lib")] 
static extern void free_bytes(IntPtr bytes, int len, int capacity);

void test()
{
   int len, capacity;
   IntPtr ptr = get_bytes(out len, out capacity);
   // TODO: use the data in ptr somehow
   free_bytes(ptr, len, capacity);
}

You have a few different options for what to put in place of the TODO.

  • Use the IntPtr as-is, reading data from the array with methods like Marshal.ReadIntPtr. I don't recommend this because it is verbose and error-prone, and would prevent the use of most APIs targeting arrays.
  • Convert the IntPtr to a byte* with (byte*)ptr.ToPointer() and use the raw byte* directly. This might be marginally less verbose than the above, but it's just as error-prone, and many useful APIs do not accept raw pointers.
  • Copy data form the IntPtr into a managed byte[]. This is a little inefficient, but you will have all the advantages of a real managed array, and it will be safe to use the array even after calling free_bytes on the original memory. However, if you want to modify the array and have these modifications be visible to Rust, you will have to perform another copy. For this solution, replace the comment with:
byte[] bytes = new byte[len];
Marshal.Copy(ptr, bytes, 0, len);
  • If you are in C# 7.2 or above, you can avoid copying memory with the new Span<T> type, which can represent a range of managed or unmanaged memory. Depending on what you plan to do with bytes, a Span<byte> may be sufficient, as many APIs have been updated to accept spans in recent versions of C#. As the span refers directly to the memory allocated by Rust, any mutations to it will be reflected on the Rust side, and you must not attempt to use it after that memory has been freed by the call to free_bytes. For this solution, replace the comment with:
Span<byte> bytes = new Span<byte>(ptr.ToPointer(), len);

A note about safety

Note that the Rust function get_bytes is marked as unsafe. This is because the as operator is used to cast the length and capacity of the vec to i32s. This will panic if they do not fit in the range of i32, and to the best of my knowledge panicking over an FFI boundary such as that introduced by P/Invoke is still undefined behavior. In production code, get_bytes could be modified to handle such errors some other way, perhaps by returning a null pointer, and C# would need to detect this situation and act accordingly.

user_18258600
Related