How memory allocation is done heavily depends on the operating system. You need to understand how shared libraries work in those operating systems, how C language relates to those operating systems and to the concept of shared libraries.

C, C++ and modular programming

First of all, I want to mention that C language is not a modular language e.g. it has no support for modules or modular programming. For languages like C and C++ implementation of modular programming is left to the underlying operating system. Shared libraries is an example of mechanism that is used to implement modular programming with C and C++, therefore I will refer to them as modules.

Module = shared library and executable

Linux and Unix-like systems

Initially everything on Unix systems was statically linked. Shared libraries came later. And as Unix was a starting point for the C langauge, those systems try to provide shared library programming interface that is close to what programming in C feels like.

The idea is that C code written originally without shared libraries in mind should be build and should work without changes made to the source code. As the result, provided environment usually has single process-wide symbol namespace shared by all loaded modules e.g. there can only be a single function with a name foo in the whole process, except for static functions (and some functions that are hidden in moduels using OS-specific mechanisms). Basically it is the same as with static linking where you are not allowed to have duplicate symbols.

What this means for your case is that there is always a single function named malloc in use in the whole process and every module is using it e.g. all modules share the same memory allocator.

Now if process happens to have multiple malloc functions, only a single one is picked and will be used by all modules. Mechanism here is very simple - as shared libraries do not know location of every referenced function, they will usually call them though some table (GOT, PLT) that will be filled with required addresses lazily on the first call or at load time. The same rule is applied to the module that provides original function - even internally this function will be called though the same table, making it possible to override that function even in the original module that provides it (which is the source of many ineffeciencies related to usage of shared libraries on Linux, search for -fno-semantic-interposition, -fno-plt to overcome this).

The general rule here is that the first module to introduce symbol will be the one providing it. Therefore original process executable has the highest priority here and if it defines malloc function, that malloc function will be used everywhere in the process. The same applies to functions calloc, realloc, free and others. Using this trick and tricks like LD_PRELOAD allow you to override "default memory allocator" of your application. This is not guaranteed to work thought as there are some corner cases. You should consult documentation for your library before doing this.

I want to specifically note that this means there is a single heap in the process shared by all modules and there is a good reason for that. Unix-like systems usually provide two ways of allocating memory in a process:

1. brk, sbrk syscalls
2. mmap syscall

The first one provides you an access to a single per-process memory region usually allocated directly after the executable image. Because of the fact that there is only one such region, this way of memory allocation can only be used by a single allocator in a process (and it is usually already used by your C library).

This is important to understand before you throw any custom memory allocator into your process - it either should not use brk, sbrk, or should override existing allocator of your C library.

The second one can be used to request chunk of memory directly from the underlying kernel. As kernel know the structure of your process virtual memory, it is able to allocate pages of memory without interfering with any user-space allocator. This is also the only way to have multiple fully independent memory allocators (heaps) in the process.

Windows

Windows does not rely on C runtime the same way Unix-like systems do. Instead it provides its own runtime - Windows API.

There are two ways of allocating memory with Windows API:

1. Using functions like VirtualAlloc, MapViewOfFile.
2. And heap allocation functions - HeapCreate, HeapAlloc.

The first one is an equivalent to mmap, while the second one is a more advanced version of malloc which is based internally (as I believe) on VirtualAlloc.

Now because Windows does not have the same relation to C language as Unix-likes have, it does not provide you with malloc and free functions. Instead, those are provided by C runtime library which is implemented on top of Windows API.

Another thing about Windows - it does not have a concept of single per process symbol namespace e.g. you cannot override function here the same way you do on Unix-like systems. This allows you to have multiple C runtimes co-existing in the same process, and every of those runtimes can provide its independent implementation of malloc, free etc, each operating on a separate heap.

Therefore on Windows all libraries will share a single process Windows API-specific heap (can be obtained through GetProcessHeap), at the same time they will share heap of one of C runtimes in the process.

So how do you integrate memory allocator into your program?

It depends. You need understand what you are trying to achieve.

Do you need to replace memory allocator used by everyone in your process e.g. the default allocator? This is only possible on Unix-like system.

The only portable solition here is to use your specific allocator interface explicitly. It doesn't really matter how you do this, you just need to make sure the same heap is shared by all libraries on Windows.

The general rule here is that either everything should be statically linked or everything should be dynamically linked. Having some sort of mix between the two might be really complicated and requires you to keep the whole architecture in your head to avoid mixing heaps or other data structures in your program (which is not a big problem if you don't have many modules). If you need to mix static and dynamic linking, you should build you allocator library as a shared library to make it easier having single implementation of it in a process.

Another difference between Unix-alikes and Windows is that Windows does not have a concept of "statically linked executable". On Windows every executable has dependencies on Windows-specific dynamic libraries like ntdll.dll. While with ELF executables have separate types for "statically linked" and "dynamically linked" executables.

This is mostly due to single per-process symbol namespace which makes it dangerous to mix shared and static linking on Unix-alikes, but allows Windows to mix static and dynamic linking just fine (almost, not really).

If you use one of your libraries, you should make sure you link it dynamically with dynamically linked executables. Imagine if you link your allocator statically into your shared library, but another library in your process uses the same library too - you might be using another allocator by accident, not the one you were expecting.