Is it possible to change calling convention in gcc for x64?

user_5081245 0
Viewed 285

I’m writing a C program where I intercept the arguments to a function called by pointer before changing them and forwarding the call to the actual function, and this forwarding is done in assembly. I know that when compiling to x86 it’s possible to mark the calling convention on a per-function basis using attribute((stdcall)) so I need to know two things:

  1. Is it possible to change the calling convention gcc uses in x64 mode, or is that only available in 32 bit programs?
  2. If it is possible, can those, or similar annotations be applied to function pointers so I can ensure the functions are called properly?

I am open to using other compilers if they support this and gcc does not.

calling-convention clang compilation gcc c
1 Answers

GCC supports two calling conventions on x86-64: the System V ABI conventions used on most Unix-like systems, and the Microsoft x64 convention used on Windows. You can choose a specific one for a function with __attribute__((ms_abi)) or __attribute__((sysv_abi)). These attributes can also be applied to function pointers. See https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gcc/x86-Function-Attributes.html#x86-Function-Attributes

Try on godbolt. Notice that foo1 and ptr1 have their arguments passed in ecx and edx per the MS convention, while foo2 and ptr2 use edi and esi as per SysV.

clang behaves the same.

user_6349190
Related