Hooking virtual method on Win32 (return object larger than ptr crashes)?

Question

So trying to hook a OpenVR method "ITrackedDeviceServerDriver::GetPose" but i'm getting access violations when my custom hooked method returns out (I think something is wrong with calling convention but idk enough about what x64 asm expects).

All other methods I hook in "ITrackedDeviceServerDriver" work fine. Only "GetPose" fails.

Class thats being hooked looks like:

class ITrackedDeviceServerDriver
{
public:
    virtual EVRInitError Activate( uint32_t unObjectId ) = 0;
    virtual void Deactivate() = 0;
    virtual void EnterStandby() = 0;
    virtual void *GetComponent( const char *pchComponentNameAndVersion ) = 0;
    virtual void DebugRequest( const char *pchRequest, char *pchResponseBuffer, uint32_t unResponseBufferSize ) = 0;
    virtual DriverPose_t GetPose() = 0;
};

Object thats larger than ptr & gets returned from method that crashes

struct DriverPose_t
{
    double poseTimeOffset;
    vr::HmdQuaternion_t qWorldFromDriverRotation;
    double vecWorldFromDriverTranslation[ 3 ];
    vr::HmdQuaternion_t qDriverFromHeadRotation;
    double vecDriverFromHeadTranslation[ 3 ];
    double vecPosition[ 3 ];
    double vecVelocity[ 3 ];
    double vecAcceleration[ 3 ];
    vr::HmdQuaternion_t qRotation;
    double vecAngularVelocity[ 3 ];
    double vecAngularAcceleration[ 3 ];
    ETrackingResult result;
    bool poseIsValid;
    bool willDriftInYaw;
    bool shouldApplyHeadModel;
    bool deviceIsConnected;
};

Example of how I'm hooking the virtual methods:

typedef DriverPose_t(__thiscall* GetPose_Org)(ITrackedDeviceServerDriver* thisptr);
GetPose_Org GetPose_Ptr = nullptr;
DriverPose_t __fastcall GetPose_Hook(ITrackedDeviceServerDriver* thisptr)
{
    DriverPose_t result = GetPose_Ptr(thisptr);// works
    //result.deviceIsConnected = true;
    return result;// after return I get access violation crash
}

void TestHook(ITrackedDeviceServerDriver *pDriver)
{
    MEMORY_BASIC_INFORMATION mbi;
    ZeroMemory(&mbi, sizeof(MEMORY_BASIC_INFORMATION));
    void** vTable = *(void***)(pDriver);
    VirtualQuery((LPCVOID)vTable, &mbi, sizeof(MEMORY_BASIC_INFORMATION));
    VirtualProtect(mbi.BaseAddress, mbi.RegionSize, PAGE_EXECUTE_READWRITE, &mbi.Protect);// unlock

    Activate_Ptr = (Activate_Org)vTable[0];
    vTable[0] = &Activate_Hook;// Hook!

    Deactivate_Ptr = (Deactivate_Org)vTable[1];
    vTable[1] = &Deactivate_Hook;// Hook!

    EnterStandby_Ptr = (EnterStandby_Org)vTable[2];
    vTable[2] = &EnterStandby_Hook;// Hook!

    GetPose_Ptr = (GetPose_Org)vTable[5];
    vTable[5] = &GetPose_Hook;// Hook!

    VirtualProtect(mbi.BaseAddress, mbi.RegionSize, mbi.Protect, &mbi.Protect);// lock
}

So why is "GetPose" the only method that fails after it returns out? How can I fix the calling convention on "GetPose" to work with registers correctly?

user_6401656 · Answer

as already noted in comments

__thiscall and __fastcall are not the same calling convention

even for x64. for x86 this is obvious - __thiscall - pass first parameter (this) via Ecx register, and next parameters via stack (i skip now float/double parameters case). __fastcall - 1-first via Ecx, 2-second via Edx, and next via stack

in case x64 (again let assume that no float/double parameters) - first 4 parameters via Rcx, Rdx, R8, R9 and next in stack. so look like x64 have single and common calling convention (no difference between __stdcall, __fastcall, __cdecl, __thiscall)

but exist question about implicit parameters. say in case __thiscall - the first parameter - this is implicit parameter, always passed as first parameter via Ecx/Rcx

and now case when function try "return" object. really on any platform function can return something only via cpu registers. it finite count, and only several of it can be used for return value store.

say on x86(x64) - for return value can be used: AL, AX, EAX, RAX, DAX:EAX, RDX:RAX so function can return only something that have size - 1, 2, 4, 8 (and 16 for x64)

if function try "return" object with another size - this is impossible do direct, compiler need transform your code. for example if you write function:

DriverPose_t GetPose(ITrackedDeviceServerDriver* thisptr)
{
    DriverPose_t pos = ...;
    return pos;
}

and call

DriverPose_t pos = GetPose(thisptr);

compiler (of course this already implementation details, different compilers can do this in different ways !) can transform this to

void GetPose(DriverPose_t *ppos, ITrackedDeviceServerDriver* thisptr)
{
    DriverPose_t pos = ...;
    *ppos = pos;
}

and call to

DriverPose_t pos;
GetPose(&pos, thisptr);

so here pointer to object passed to function as first ,hidden/implicit, parameter. but in case member function, still exist another hidden/implicit, parameter - this, which always passed as first, as result hidden/implicit, parameter for return value moved to second place !

struct ITrackedDeviceServerDriver
{
    DriverPose_t GetPose()
    {
        DriverPose_t pos = ...;
        return pos;
    }
};

and call

ITrackedDeviceServerDriver obj;
DriverPose_t pos = obj.GetPose();

can be transformed to

struct ITrackedDeviceServerDriver
{
    void GetPose(DriverPose_t *ppos)
    {
        DriverPose_t pos = ...;
        *ppos = pos;
    }
};

and call to

ITrackedDeviceServerDriver obj;

DriverPose_t pos;
obj.GetPose(&pos);

so real signature of function, if "uncover" this parameter is

void ITrackedDeviceServerDriver::GetPose(ITrackedDeviceServerDriver* this, DriverPose_t *ppos)
{
    DriverPose_t pos = ...;
    *ppos = pos;
}

and call

ITrackedDeviceServerDriver obj;

DriverPose_t pos;
GetPose(&obj, &pos);

so compare

void ITrackedDeviceServerDriver::GetPose(ITrackedDeviceServerDriver* this, DriverPose_t *ppos);

and

void GetPose(DriverPose_t *ppos, ITrackedDeviceServerDriver* thisptr);

you implement

DriverPose_t GetPose_Hook(ITrackedDeviceServerDriver* thisptr);

which will be transformed by compiler to

void GetPose_Hook(DriverPose_t* ,ITrackedDeviceServerDriver* thisptr);

but really you need implement next hook:

void GetPose_Hook(ITrackedDeviceServerDriver* thisptr, DriverPose_t* );

you confuse first and second parameters. however i think not need change object vtable at all, instead need return proxy object to client. possible and very generic implementation for COM interface hooks, but it too long for post here. for your case can be done next example of hook:

struct DriverPose_t 
{
    int x, y, z;
};

struct __declspec(novtable) IDemoInterface 
{
    virtual DriverPose_t GetPose() = 0;
    virtual ULONG GetComponent( const char * pchComponentNameAndVersion ) = 0;
    virtual void Delete() = 0;
};

class DemoObject : public IDemoInterface
{
    ULONG v;
    DriverPose_t pos;

    virtual DriverPose_t GetPose()
    {
        DbgPrint("%s<%p>
", __FUNCTION__, this);
        return pos;
    }

    virtual ULONG GetComponent( const char * pchComponentNameAndVersion )
    {
        DbgPrint("%s<%p>(%s)
", __FUNCTION__, this, pchComponentNameAndVersion);
        return v;
    }

    virtual void Delete()
    {
        delete this;
    }

public:
    DemoObject(ULONG v, int x, int y, int z) : v(v) 
    { 
        pos.x = x, pos.y = y, pos.z = z; 
        DbgPrint("%s<%p>
", __FUNCTION__, this);
    }

    virtual ~DemoObject()
    {
        DbgPrint("%s<%p>
", __FUNCTION__, this);
    }
};

class Hook_DemoInterface : public IDemoInterface
{
    IDemoInterface* pItf;

    virtual DriverPose_t GetPose()
    {
        DriverPose_t pos = pItf->GetPose();
        DbgPrint("%s<%p>=<%d, %d, %d>
", __FUNCTION__, this, pos.x, pos.y, pos.z);
        return pos;
    }

    virtual ULONG GetComponent( const char * pchComponentNameAndVersion )
    {
        ULONG v = pItf->GetComponent(pchComponentNameAndVersion);
        DbgPrint("%s<%p>(%s)=%u
", __FUNCTION__, this, pchComponentNameAndVersion);
        return v;
    }

    virtual void Delete()
    {
        delete this;
    }
public:
    
    Hook_DemoInterface(IDemoInterface* pItf) : pItf(pItf) 
    {
        DbgPrint("%s<%p>
", __FUNCTION__, this);
    }

    ~Hook_DemoInterface() 
    { 
        pItf->Delete(); 
        DbgPrint("%s<%p>
", __FUNCTION__, this);
    }
};

BOOL CreateDemoIface(IDemoInterface** ppItf)
{
    if (DemoObject* pObj = new DemoObject (1, -1, 2, 3))
    {
        *ppItf = pObj;
        return TRUE;
    }
    return FALSE;
}

BOOL Hook_CreateDemoIface(IDemoInterface** ppItf)
{
    IDemoInterface* pItf;

    if (CreateDemoIface(&pItf))
    {
        if (Hook_DemoInterface* pObj = new Hook_DemoInterface(pItf))
        {
            *ppItf = pObj;
            return TRUE;
        }

        *ppItf = pItf;
        return TRUE;
    }
    return FALSE;
}

void UseIface(IDemoInterface* pItf)
{
    ULONG v = pItf->GetComponent("some string");
    DriverPose_t pos = pItf->GetPose();
    DbgPrint("v=%u, pos<%d, %d, %d>
", v, pos.x, pos.y, pos.z);
}

void test()
{
    IDemoInterface* pItf;
    
    if (CreateDemoIface(&pItf))
    {
        UseIface(pItf);
        pItf->Delete();
    }

    if (Hook_CreateDemoIface(&pItf))
    {
        UseIface(pItf);
        pItf->Delete();
    }
}

instead hook object vtable - hook (if possible) object creation CreateDemoIface - replace it to self Hook_CreateDemoIface which return proxy object.

debug output

DemoObject::DemoObject<0000012C31E08F70>
DemoObject::GetComponent<0000012C31E08F70>(some string)
DemoObject::GetPose<0000012C31E08F70>
v=1, pos<-1, 2, 3>
DemoObject::~DemoObject<0000012C31E08F70>


DemoObject::DemoObject<0000012C31E08F70>
Hook_DemoInterface::Hook_DemoInterface<0000012C31DFAA10>
DemoObject::GetComponent<0000012C31E08F70>(some string)
Hook_DemoInterface::GetComponent<0000012C31DFAA10>(some string)=1
DemoObject::GetPose<0000012C31E08F70>
Hook_DemoInterface::GetPose<0000012C31DFAA10>=<-1, 2, 3>
v=1, pos<-1, 2, 3>
DemoObject::~DemoObject<0000012C31E08F70>
Hook_DemoInterface::~Hook_DemoInterface<0000012C31DFAA10>

Hooking virtual method on Win32 (return object larger than ptr crashes)?

1 Answers