Kullback Liebler divergence is famous asymmetric KL(X,Y) != KL(Y,X).
However let X* be arg_max KL(X,Y). Then what do we know about KL(Y,X*)? Is it as large as possible?
Suppose I have a binary variable Y and a much more complicated, multidimensional (but discrete) distribution X.
If I find an X that maximizes KL(X,Y) then does that X also maximize KL(Y,X) (for the same Y).
Suppose the outcome Y is getting a loan. Only 10% of people in the dataset get a loan. P(Y) = .1
However, among white males the probability of getting a loan increases to 20% P(Y|white,male) = .2
Furthermore, lets say white males make up 30% of the dataset P(WM) = .30
From this we can also deduce that WM get 60% of all loans P(WM | Y) = .6
We get
KL(WM,Y) = .2 * ln(.2/.1) + .8 *ln(.8/.9)
In the other direction we have
KL(Y,WM) = .6 * ln(.6/.3) + .4 *ln(.4 / .7)
Now obviously these 2 values do not equal eachother. However, can we prove that no other X will increase KL(Y,WM) higher than this?