This is because overload resolution depends on the compile-time type of the argument expressions, not the runtime type of the argument. See JLS 15.12.2.2:
The argument expression true ? b : c has the type int, so the Int method is called. This is specified in the JLS 15.25.2, where it is said that if none of the special cases match (none of them do here), then the type of the conditional expression is the type after applying numeric promotion to the second and third operands:
An expression appears in a numeric choice context if the expression is one of the following:
- The second or third operand of a numeric conditional expression
- [...]
Numeric promotion determines the promoted type of all the expressions in a numeric context. [...] The rules are as follows:
[...] (none of these rules match in the case of a short and a byte expression)
Otherwise, none of the expressions are of type double, float, or long. In this case, the kind of context determines how the promoted type is chosen.
In a numeric choice context, the following rules apply:
[...] (none of these rules match in the case of a short and a byte expression)
Otherwise, the promoted type is int, and all the expressions that are not of type int undergo widening primitive conversion to int.
This is why true ? b : c is of type int. The same goes for false ? b : c and false ? 'A' : (byte)0. It doesn't matter what the condition is. The type of the expression is determined by the types of the second and third operands.
As for test(true ? 0 : 'A'), this can be explained by a rule that I omitted above, just before the last "Otherwise..."
- Otherwise, if any expression is of type
char, and every other expression is either of type char or a constant expression of type int with a value that is representable in the type char, then the promoted type is char, and the int expressions undergo narrowing primitive conversion to char.
In this case, the expression "0" is a "a constant expression of type int with a value that is representable in the type char".
One way to think about the type of a conditional expression, is that it's the smallest type that the expressions of both of the branches can "fit". For a byte b and a char c, int is the smallest type that can fit both of them. For 0, and 'A', char is enough to fit both, as it is known that 0 is in the range of char.