How to use virtual memory/implement realloc on mac osx?

Using only POSIX mmap flags, the "optimistic" strategy is what Jester suggested, using mmap without MAP_FIXED to try to allocate new pages contiguous with what you already have. (The first arg is a "hint" of where you'd like it to allocate).

Instead of failing, it will allocate somewhere else (unless virtual address space is full, but that's unlikely on 64-bit). So you need to detect that mmap's return value != your hint. Probably just munmap that untouched space and ask again with the full size you need, then copy. You could attempt to mmap the remaining space onto the end of the new pages you just got, but that could fail and then you're making even more system calls.

On Linux you'd use mmap(MAP_FIXED_NOREPLACE) to return an error if it can't allocate where you want (without overlapping / replacing existing mappings).

Of course Linux mremap is even better, avoiding ever copying the data, just mapping the same physical pages to a new virtual address if you let it (with MREMAP_MAYMOVE). If MacOS doesn't have similar functionality via any MacOS-specific function calls or mmap flags, you simply can't get that functionality.

I find it really dumb that C++ std::vector is designed so it can't easily take advantage of realloc and thus mremap even if it exists, with replaceable new being a potentially visible side effect. And the new/delete allocator API entirely lacking a try-realloc that you could use even with non-trivially-copyable types. But this overly-conservative design in some higher-level languages means that low-level features might not get much use even if they existed, so I wouldn't be surprised if MacOS lacked it.

OTOH, C realloc certainly can use mremap if the original allocation has its pages to itself, and lots of stuff is written in C, not hobbled by C++'s allocator API. So MacOS might well support something like this somehow, but I don't know MacOS-specific system call details.

I did have a look at the table of BSD system calls in the Darwin XNU kernel https://github.com/opensource-apple/xnu/blob/master/bsd/kern/syscalls.master as suggested by macOS 64-bit System Call Table

There might be other whole categories of system call, but I'd hope that any mmap-related calls would be in the BSD family of calls, using the 0x2000000 class bit.

There is a int memorystatus_control(uint32_t command, int32_t pid, uint32_t flags, user_addr_t buffer, size_t buffersize); but that returns an int, so I assume it's not what we're looking for.

I didn't see any other system calls that looked at all promising for this.

I didn't check the MacOS man page for mmap; if it has any MacOS-specific flags like MAP_FIXED_NOREPLACE, they'd hopefully be there.