I want to map two integers A and B (A, B <= 10000) to an integer C with a function f such that f(A, B) = f(B, A) and there are no two other integers E and F such that f(E, F) = f(A, B).
I originally tried using f(A, B) = 10001*A+B, but that didn't satisfy the requirement where f(A, B) = f(B, A).
Sure. I assume that m,n are nonnegative integers. For n <= m put f(m,n) = 1/2 m(m+1) + n. For n > m put f(m,n) = f(n,m).
This assigns a unique integer to every (m,n), except that f(m,n) = f(n,m) for all m,n as desired (i.e. the operation is commutative). There is no need to assume that m,n <= 10000.
I wouldn't call this a hash function though. Such a function distributes elements evenly over a number of buckets, it doesn't typically assign a unique integer to each element.
Edit. The idea is to walk through all the nonnegative integral plane points lying on or below the line y = x. We start at the origin and walk north. When we reach the line y = x, we jump to (m+1,0). Then it's all a matter of counting the steps we have taken.
