Recursive functions work exactly like non-recursive functions.
One common mistake is to try to think about all the recursive calls at once, as if they had a shared state, but one key ingredient to understanding a recursive function is actually to ignore the recursion and just "think locally".
Perhaps working through an example would clarify things.
Let's look at smallD(321), replacing n in the body of the function with its value.
smallD(321)
if (321 < 10) return 321;
int x = smallD(321 / 10);
if (x < 321 % 10) return x;
else return 321 % 10;
The first condition is clearly false, and in order to determine x, we need smallD(321/10), which is smallD(32).
smallD(32)
if (32 < 10) return 32;
int x = smallD(32 / 10);
if (x < 32 % 10) return x;
else return 32 % 10;
The first condition is false again, so we keep going with smallD(32/10).
smallD(3)
if (3 < 10) return 3;
int x = smallD(3 / 10);
if (x < 3 % 10) return x;
else return 3 % 10;
Now the first condition is true, so the result here is clearly 3.
Now we can go back and use the value of x in each call that was left waiting.
smallD(32)
...
if (3 < 32 % 10) return 3;
else return 32 % 10;
And 3 < 32 % 10 is false, so we return 32 % 10 - 2 to the caller.
smallD(321)
...
if (2 < 321 % 10) return 2;
else return 321 % 10;
And 2 < 321 % 10 is false, so we return 321 % 10, which is 1.