Calling through pointer returned by Marshal.GetFunctionPointerForDelegate causes Access Violation

Question

I'm calling native x64 code from C# using Marshal.GetDelegateForFunctionPointer(). I pass a pointer into the native code as a parameter. I get the pointer from Marshal.GetFunctionPointerForDelegate() passing in a C# delegate. When executing in the native code I attempt to callback into C# using the passed pointer. This results in an Access Violation. I believe it's because the native code is not setting the stack up properly before attempting the callback, but I can't determine how this should be done. I've condensed this down to the following repo:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;

namespace AsmCallbackRepo
{
  unsafe class Program
  {
    [DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)]
    static extern IntPtr VirtualAllocEx(IntPtr hProcess, IntPtr lpAddress,
    uint dwSize, AllocationType flAllocationType, MemoryProtection flProtect);

    [Flags]
    public enum AllocationType
    {
      Commit = 0x1000,
      Reserve = 0x2000,
      Decommit = 0x4000,
      Release = 0x8000,
      Reset = 0x80000,
      Physical = 0x400000,
      TopDown = 0x100000,
      WriteWatch = 0x200000,
      LargePages = 0x20000000
    }

    [Flags]
    public enum MemoryProtection
    {
      Execute = 0x10,
      ExecuteRead = 0x20,
      ExecuteReadWrite = 0x40,
      ExecuteWriteCopy = 0x80,
      NoAccess = 0x01,
      ReadOnly = 0x02,
      ReadWrite = 0x04,
      WriteCopy = 0x08,
      GuardModifierflag = 0x100,
      NoCacheModifierflag = 0x200,
      WriteCombineModifierflag = 0x400
    }

    static readonly byte[] i64 = new byte[]
    { 
      0xcc,                                       // int 3        debug break
      0x48, 0x89, 0xC8,                           // mov rax,rcx  parm 1: call-back address
      0x48, 0xC7, 0xC1, 0x0F, 0x00, 0x00, 0x00,   // mov rcx,15   input parm for call-back
      0x48, 0x83, 0xEC, 0x20,                     // sub rsp,32   space for register home storage
      0xFF, 0xD0,                                 // call rax     call the managed call-back
      0x48, 0x83, 0xC4, 0x20,                     // add rsp,32   release register home storage space
      0xC3,                                       // ret          return to managed caller
    };

    delegate void CallBackDel(long parm);     // prototype of call-back
    delegate void NativeDel(void* arg);       // prototype of x64 native method


    static void Main(string[] args)
    {
      CallBackDel callback = new CallBackDel(CallBack);
      IntPtr memory = VirtualAllocEx(Process.GetCurrentProcess().Handle, IntPtr.Zero, 4096,
                                     AllocationType.Commit, MemoryProtection.ExecuteReadWrite);
      byte* ptr = (byte*)memory.ToPointer();

      // copy x64 native code to allocated memory segment
      for (int i = 0; i < i64.Length; ++i)        
      {
        ptr[i] = i64[i];
      }

      // wrap native code in a delegate
      NativeDel i64Action = (NativeDel)Marshal.GetDelegateForFunctionPointer(new IntPtr(ptr), typeof(NativeDel));
      Debugger.Break();

      // get pointer for call-back
      IntPtr callbackPtr = Marshal.GetFunctionPointerForDelegate(callback);

      // call native x64 copied to allocated memory passing address of call-back
      i64Action(callbackPtr.ToPointer());
    }

    static void CallBack(long parm)
    {
      Debugger.Break();
      Console.WriteLine($"CallBack was called with value {parm}");
    }
  }
}

Debugging in WinDbg I hit the Break before calling into the native code and the Break that's at the top of the native code. I can single step through the native code until executing the CALL RAX in native code. At this point I get an Access Violation trying to save a floating point register. This is intended to be compiled for 64-bit and I'm trying to make the native code adhere to x64 stack usage/calling conventions.

Any insights would be greatly appreciated- you might even save a few keyboards from being shattered :-)

user_8422330 · Answer

Within the calling function, the stack is 16-byte aligned. When it calls the native function, it pushes the return address, so the stack is now misaligned by 8 bytes. So within your function, you need to subtract an odd multiple of 8 to realign it, before making another call.

Windows also requires unused space of 32 bytes at the top of the stack before the call. (Presumably this is why the sub 32 is already there.)

So the solution is to subtract 40 from rsp instead of 32.

As you extend this function to add functionality, you may need to push registers and/or allocate additional memory on the stack. While doing so, be sure to maintain the 16-byte stack alignment and also maintain the 32 bytes of unused space at the top of the stack.

Calling through pointer returned by Marshal.GetFunctionPointerForDelegate causes Access Violation

1 Answers