How to catch a call to exit() (for unit testing)

It is UB to define another exit() in standard C (both as a function and as a macro if the header is included). In many environments, you are likely to be able to get away with it, though.

With that said, doing such a thing makes no sense here, because we are talking about exit(). The function is not expecting to continue execution after that, so that pretty much forces you to replace it with a longjmp() or go with textual replacement to convert it to a return (assuming a void return type). In both cases, it means you need to assume the function is not leaving things in a broken state (like holding some resource). That is a lot to assume, but it may be a reasonable way out if your unit testing framework is meant to be tied to this particular project.

Instead of trying to modify the behavior of the tested function, I suggest you add support for running tests in their own process to your test framework. There are many advantages apart from being able to test things like these. For instance, you get the ability of running tests in parallel for free and isolates many side-effects between them.

Another solution: use an #ifdef macro to call a different function in your debug build.

#ifdef DEBUG
 #define exit_badindex(...) my_debug_function(__VA_ARGS__)
#else
 #define exit_badindex(...) exit(__VA_ARGS__)
  #ifndef NDEBUG
   #warning NDEBUG undefined in non-DEBUG build: my_debug_function will not be called
  #endif
#endif

If it's for unit testing, you don't need to physically catch the call to exit().

Before the solution, I would like to suggest redesigning your library. You have a few alternatives:

• Having util_dyn_array include a callback that is called if out-of-bound mode is encountered, and default it to exit(1) (not exit(-1), that doesn't work well when the program is called from a shell).
• Having a global out-of-bound handler (which again defaults to exit(1)), and allow the program to change the handler at runtime by calling something like set_oob_handler(new_handler).
• Doing integration tests instead of unit tests. As suggested by multiple people here, if the library can exit or crash, this goes to the realm of integration (with the calling process/OS).

My solution:

main.c:

#include <stdio.h>

void func(void);

int main(int argc, char **argv)
{
    printf("starting\n");
    func();
    printf("ending\n");
}

something.c:

void my_exit(int status)
{
    printf("my_exit(%d)\n", status);
#ifdef UNIT_TEST
    printf("captured exit(%d)\n", status); // you can even choose to call a global callback here, only in unit tests.
#else
    exit(status);
#endif
}

void func(void) {
    my_exit(1);
}

makefile:

# these targets are MUTUALLY EXCLUSIVE!!

release:
    cc -g -c -fpic something.c
    cc -shared -o libsomething.so something.o
    cc -g -o main main.c -L. -lsomething

fortest:
    cc -DUNIT_TEST=1 -g -c -fpic something.c
    cc -shared -o libsomething.so something.o
    cc -g -o main main.c -L. -lsomething

$ make release
cc -g -c -fpic something.c
[...]
$ LD_LIBRARY_PATH=. ./main
starting
my_exit(1)
$ make fortest
cc -DUNIT_TEST=1 -g -c -fpic something.c
[...]
$ LD_LIBRARY_PATH=. ./main
starting
my_exit(1)
captured exit(1)
ending

(note that this was tested on Linux, I don't have a Mac to test on, so minor makefile modifications might be required).

The exit function is a weak symbol, so you can create your own copy of the function to catch the case where it gets called. Additionally, you can make use of setjmp and longjmp in your test code to detect a proper call to exit:

For example:

#include "file_to_test.c"

static int expected_code;    // the expected value a tested function passes to exit
static int should_exit;      // 1 if exit should have been called
static int done;             // set to 1 to prevent stubbing behavior and actually exit

static jmp_buf jump_env;

static int rslt;    
#define test_assert(x) (rslt = rslt && (x))

// stub function
void exit(int code)
{
    if (!done)
    {
        test_assert(should_exit==1);
        test_assert(expected_code==code);
        longjmp(jump_env, 1);
    }
    else
    {
        _exit(code);
    }
}

bool test_dyn_array_check_index() {
    int jmp_rval;
    done = 0;
    rslt = 1;

    util_dyn_array vector = util_dyn_array_new(sizeof(int), 16);
    for(int i = 0; i < 16; i++) {
        util_dyn_array_push(&vector, (void*)&i);
    }

    for(int i = 0; i < 16; i++) {
        //if nothing happens, its successful
        should_exit = 0;
        if (!(jmp_rval=setjmp(jump_env)))
        {
            util_dyn_array_check_index(&vector, i);
        }
        test_assert(jmp_rval==0);

    }

    // should call exit(-1)
    {
        should_exit = 1;
        expected_code = 2;
        if (!(jmp_rval=setjmp(jump_env)))
        {
            util_dyn_array_check_index(&vector, 16);
        }

        test_assert(jmp_rval==1);
    }
    done = 1
 
    return rslt;

}    

Before calling a function that could call exit, call setjmp to set a jump point. The stubbed exit function then checks whether exit should have been called and with which exit code, then calls longjmp to jump back out to the test.

If exit was called then the return value of setjmp is 1, indicating it came from a call to longjmp. If not longjmp is not called and the return value of setjmp will be 0 after the function returns.

Run the code in a fork and check the exit value of the fork.

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <unistd.h>
#include <sys/wait.h>

void util_dyn_array_check_index() {
    exit(-1);
}

int main(void) {
    pid_t pid = fork();
    assert(pid >= 0);
    if( pid == 0 ) {
        util_dyn_array_check_index();
        exit(0);
    }
    else {
        int child_status;
        wait(&child_status);
        printf("util_dyn_array_check_index exited with %d\n", WEXITSTATUS(child_status));
    }
}

Note that the exit status will be 255 because despite accepting an integer POSIX exit statuses are unsigned. Consider using exit(1) instead or define an ARGUMENT_ERROR_EXIT_STATUS macro.

Note that I used an assert to check if my fork failed. This might be a better way to implement your checks.

inline void util_dyn_array_check_index(util_dyn_array * self, size_t index) {
    assert(index < self->len);
}

This will provide more information on error, and it can be switched off in production for performance.

Assertion failed: (index < self->len), function util_dyn_array_check_index, file test.c, line 9.

assert calls abort. In your tests you'd close stderr to avoid the assert message from cluttering up the output, and check that WTERMSIG(status) == SIGABRT.

void util_dyn_array_check_index() {
    assert(43 < 42);
}

int main(void) {
    pid_t pid = fork();
    assert(pid >= 0);
    if( pid == 0 ) {
        // Suppress the assert output
        fclose(stderr);
        util_dyn_array_check_index();
        exit(0);
    }
    else {
        int status;
        wait(&status);
        if( WTERMSIG(status) == SIGABRT ) {
            puts("Pass");
        }
        else {
            puts("Fail");
        }
    }
}