How to avoid expensive array marshalling between VBA and C#?

I'm exporting this C# method through COM:

public interface IMatrixMath {
    void AddScaledRange3D(ref double[,,] acc, [In] ref double[,,] vals, double scale);
    // ...
}

Then I call it with large matrices from VBA in Excel:

Dim A(0 To 2000, 0 To 120, 0 To 500) As Double
Dim B(0 To 2000, 0 To 120, 0 To 500) As Double
' ...
MatrixMath.AddScaledRange3D A, B, 0.4

This takes about 15 seconds but only 1 second of that time is spent in C#. Most of it is overhead before entering the C# method.

I'm not sure where the overhead comes from. If Excel needs to marshal to SAFEARRAY, wouldn't it need to marshal back, too? Why does it even take so long, doesn't Excel store arrays contiguously? Could it be related to crossing certain COM boundaries?

Are there ways around this overhead? I don't mind dealing with raw pointers or non-COM interop.