What is the canonical way to iterate a table with a __pairs metamethod from C?

To my knowledge, there's no real "elegant" way to go about this as of writing.

The best I could come up with is to create two identically prototyped closures and interact with them directly. This answer focuses mainly on __pairs but I'm sure it could be trivially adapted for __ipairs.

First is the next() iterator which works on bare tables (with no __pairs metamethod). It takes a single upvalue - the target table - and forwards the call to lua_next().

static int next_iterator(lua_State *L) {
    /* -1, +(2|0) */
    /*
        requires upvalues:
           1: the table on which to call lua_next()
    */
    return lua_next(L, lua_upvalueindex(1))
        ? 2
        : 0;
}

Then we have the __pairs() iterator, which takes the table as upvalue 1 and the result of the call to __pairs() as upvalue 2 and calls the iterator with the table and the current invocation's first argument (the key).

static int pair_iterator(lua_State *L) {
    /* -1, +2 */
    /*
        requires upvalues:
           1: the table on which to call the iterator
           2: the iterator function (usually result
              of a call to __pairs() metamethod)
    */

    lua_pushvalue(L, lua_upvalueindex(1));
    lua_pushvalue(L, -2);
    lua_copy(L, lua_upvalueindex(2), -3);
    lua_call(L, 2, 2);
    return 2;
}

Finally, the iterator function. This is meant to be called directly, not as an argument to lua_call() et al.

It pops the iterator function (conceptually, not actually) and the key and pushes the iterator and the key / value onto the stack. If the iteration has ended, then neither are pushed onto the stack - just the iterator.

I make a point to say "pops . . . then pushes the iterator" as a point of documentation - treat this function just as you would lua_next(). This also means you need to lua_pushnil(L) for the first iteration, just like you would with lua_next().

static int iterator_next(lua_State *L) {
    /* -2, +(1|3), r */
    lua_pushvalue(L, -1);
    lua_copy(L, -3, -2);
    lua_call(L, 1, 2);

    if (lua_isnil(L, -2)) {
        lua_pop(L, 2);
        return 0;
    }

    return 1;
}

Finally, we can leverage some C voodoo to create a mostly-clean setup for an iteration. This sets up the upvalues and the typical lua_next()-like iteration with the initial "key" being nil.

/*
    This assumes -1 has our table value

    NOTE: This will also remove the table itself,
          since we specify non-zero upvalue counts
          in the call to `lua_pushcclosure()`.

          This effectively replaces the table with
          a valid iterator so that lua_pop() cleanly
          cleans up the remaining artifacts after
          iteration.

          If you don't want to lose the table,
          make sure to lua_pushvalue(L, -1) before
          this switch statement.
*/
switch (luaL_getmetafield(L, -1, "__pairs")) {
    default:
        /* unsupported type; fall back to default */
        lua_pop(L, 1);
        /* fallthrough */
    case LUA_TNIL:
        /* nothing was pushed; no need to pop first. */
        lua_pushcclosure(L, &next_iterator, 1);
        break;
    case LUA_TFUNCTION:
        /* call the __pairs() metamethod and get a function back */
        lua_pushvalue(L, -2);
        lua_call(L, 1, 1);
        /* now pass the pairs function iterator closure */
        lua_pushcclosure(L, &pair_iterator, 2);
        break;
}

/* Push `nil` as our first "key" */
lua_pushnil(L);

/* Iterate! */
while (iterator_next(L)) {
    /*
        -3 has our iterator function
        -2 has the next key
        -1 has the next value
    */

    /* ... do something ... */
    (void)0;

    /*
        Pop the value so that
        the next call to iterator_next()
        uses the current iteration's key.
    */
    lua_pop(L, 1);
}

/* Finally, pop off the iterator. */
lua_pop(L, 1);