This is a surprisingly complex problem to solve. I will at this point not give working code, but rather focus on a few "nifty" issues that you have - or, as you continue coding for this - will encounter.
First, why are you recursing ?
The reason for that is that while signal handlers are "execution context transfers", by default they do not have their own stack. That means if you receive a signal as a consequence of an overflown stack, the signal handler will attempt to allocate space-on-the-stack for context potentially passed to it - and that simply re-throws the same signal again.
To make sure signal handlers run on their own separate / preallocated stack, use sigaltstack() and the SA_ONSTACK flag for sigaction().
Second, depending on "how badly" the stack overruns (your test program may not trigger this but a real world program may), the memory access (attempt) that's "the overflow-effecting action" may end up with other signals but SIGSEGV.
Your example "unspecifically" catches all signals, but that may in practice be rather insufficient / rather confusing - you sending your app a SIGUSR1 or the shell/terminal sending it a SIGTTOU on being backgrounded are absolutely not indicative of a stackoverflow.
This means there's another issue - which signals are to be expected when making an "out of stack" memory access as consequence of a stack overflow ? And how can you know that a specific signal you got was due to a stack access ?
The answer to that again is more complex than first sight:
- if the stack overflow is "small enough", it's conceivable that it's within a guard page (a valid mapping, but deliberately unreadable) and it'll trigger
SIGSEGV.
- if (no guard pages are used and) the access were to an unmapped memory region, you'll receive a
SIGBUS instead.
- Even certain CPU instructions may make a difference whether access to "invalid memory address X" results in
SIGSEGV or SIGBUS (For example, on x86, certain instructions raise #GP while others #PF - for the same mem address read/write - and the Linux kernel translates one possibly to SIGBUS the other to SIGSEGV)
- if there happens to be other memory mapped where this access happens (say, you've got
char local_to_blow_stack[1ULL << 40]; memset(&local_to_blow_stack, 0, 1);) and just-so-as-it-happens something else valid is at "whatever your stack is minus a terabyte"), that access will in fact just-work. Without the compiler to create you "assist" code to identify such accesses, it's actually possible you've blown the stack and still make a number of successful / non-signaling memory accesses before eventually hitting a mem region triggering a signal.
- You may receive these signals for other invalid operations but stack access. Heap access, memory-mapped file/device access could possibly result in the same as well.
So "just catching signals", even "catching all signals that may possibly occur as a consequence of a stack overflow" is insufficient. You need, within the signal handler to decode the memory access location, and possibly the operation / cpu instruction, to verify that the memory access attempted actually was a "stack access out of bounds". It is possible for a thread to retrieve its own stack boundaries - https://man7.org/linux/man-pages/man3/pthread_getattr_np.3.html can be used for this, at least on Linux (_np implies 'non portable' - this isn't guaranteed to be available on all systems, others may have different interfaces to retrieve this information) - but ... to find the memory location that was accessed depends on the signal and accessing instruction again. Often (but not always) it's in the siginfo (the si_addr) field.
From what I remember, exactly which signals fill si_addr under exactly what circumstances, and whether the address in there is e.g. the instruction issuing the memory access or the memory location of the attempted access, is somewhat system- and hardware-dependent (Linux may behave differently from Windows or MacOSX, and different on ARM than on x86)
So you would also need to validate that "the si_addr in this siginfo_t is somewhere-near the signaled thread's stack", but possibly also validate that the instruction that caused it was actually a memory access / si_addr can be "traced back" to the instruction that faulted. That (finding the faulting instruction's address / the program counter) ... requires decoding the other argument for the signal handler, the ucontext_t ... and there you're deep deep deep [ recurse infinity here ] in HW / OS specifics.
At this point I'd like to terminate; a "simple" but not perfect solution just needs an alternate signal stack, and the handler to retrieve the current stack boundaries via pthread_getattr_np(), to compare the si_addr against. If your life or that of others depends on the correct answer, remember the above though.