How compiler decide which function template to call?

I built the code with command:

$ g++ -o m1 ./maxdefault4.cpp -std=c++14 -DB1

$ g++ -o m2 ./maxdefault4.cpp -std=c++14 -DB2

It means that both two function template can match, but why no ambiguity error caused.

That's not correct: when you pass -DB1, or -DB2, you are compiling only one template, so there is no question of any ambiguity, and it's not surprising that it compiles.

And if buid with command:

$ g++ -o m ./maxdefault4.cpp -std=c++14

the compiler show me the error: ...

That happens when you compile both the templates, and is much more interesting since some calls compile, but others don't.

To understand what's happening, let's see what happens when you have the following templates:

template <typename T1, typename T2> 
auto max(T1 a, T2 b);                // #1

template <typename RT, typename T1, typename T2> 
RT max(T1 a, T2 b);                  // #2

and you make the call:

max<double>(4, 7.2);

Here, the compiler will substitute the provided parameter double, and deduce the remaining template parameters from the arguments.

This results in the generation of:

template<>
double max<double, double>(double a, double b);  // #1' from #1
// T1 = double (explicitly specified for 1st template parameter)
// T2 = double (deduced from 2nd function argument)

template<>
double max<double, int, double>(int a, double b); // #2' from #2
// RT = double (explicitly specified for 1st template parameter)
// T1 = int (deduced from 1st function argument)
// T2 = double (deduced from 2nd function argument)

Now overload resolution is used to figure out that #2' is a better match for the arguments 4, and 7.2, so #2' is called.

Similarly, for this call:

max<int>(4, 7.2);

the compiler will substitute the provided parameter int, and deduce the remaining template parameters from the arguments.

This results in the generation of:

template<>
double max<int, double>(int a, double b);  // #1' from #1
// T1 = int (explicitly specified for 1st template parameter)
// T2 = double (deduced from 2nd function argument)

template<>
int max<int, int, double>(int a, double b); // #2' from #2
// RT = int (explicitly specified for 1st template parameter)
// T1 = int (deduced from 1st function argument)
// T2 = double (deduced from 2nd function argument)

Now overload resolution is used, but there is an ambiguity between #1', and #2', since neither is a better match than the other (note that the return type plays no part in overload resolution). This causes the error that you observed.

The compiler works in series of stages with the output from one stage being the input to the next.

preprocessor -> compiler -> linker

• Preprocessor - expands all macros and directives (pretty much anything with a # in front of it) and generates code that will go to the compiler. If you wish to just see the output from the preprocess step you can add the -E flag to see what the compiler actually sees.
• compiler - takes code and generates .o binary files. Place holders for any variables in other files are marked in the .o file. These are the needed symbols required to run. Any functions or variable that the file exposes for others to use are also marked in the .o file. These are the symbols this file created. To see the output from the compiler add the -c flag. It should be noted that these files will be binary and thus are not human readable. You can use commands like strings or nm (if you are in the Unix/Linux world) to see the symbols.
• linking - the linker takes the .o files and maps all the needed symbols with all the required symbols in all the .o files it finds. If it finds 2 duplicate symbols, it will throw an error indicating that there are ambiguous definitions. If it can't find a needed symbol it throws and error indicating there are missing symbols. This is like redefining an existing variable or trying to use an undefined variable except it is across files instead of being in one file.

As for your specific question:

The -D compiler flag adds a preprocessor macro which is read by the preprocessor. The preprocessor effectively is a code generator and those directive tell it what should be passed along to the C++ compiler.

With the -DB1 argument added the C++ compiler will see:

/*
Inlined header contents omitted for brevity
*/

template <typename RT, typename T1, typename T2> 
RT max(T1 a, T2 b) {
    std::cout << "RT version called" << std::endl;
    return b < a ? a : b;
}

template <typename T>
void print(T t) {
    std::cout << typeid(t).name() << std::endl;
}

int main() {
    auto b = ::max<long double>(4, 7.2);
    print(b);

    auto c = ::max<int>(4, 7.2);
    print(c);

    auto d = ::max<double>(4, 7.2);
    print(d);

    return 0;
}

The first template is removed and thus the compiler sees no ambiguity. That's what #ifndef means (if not defined). However the addition of the preprocessor macro -DB1 defined B1 and so the preprocessor removed the template from the output.

With the -DB2, the second template is removed and only the first template remains. Without any -D flag both templates are included and thus there is a conflict.

The return type does not participate in overload resolution, so you have two functions that look identical to the compiler.