I am implementing a Policy Network RL method for solving the game of Pong, based on this tutorial. However, I am having trouble understanding the custom_loss() function.
The custom_loss() function takes two inputs, y_true and y_pred. However, in a RL setting, there is no y_true (i.e. labels), as you are simply acting in an environment and receiving rewards.
So how is the loss function even being computed in this example? My only hypothesis is that y_true is the taken action from the distribution:
(action = np.random.choice(range(num_actions),p=predict)),
and y_pred is the probability distribution (i.e. the output of the policy network):
predict = model_predict.predict([state])[0]
If this is the case, the probability distribution output by the network isn't being passed in the training call loss = model_train.train_on_batch([states, discounted_rewards], actions_train), only the selected, one-hot encoded actions are.
Here are the snippets of relevant code, however it is probably easier to follow the link above.
...
def custom_loss(y_true, y_pred):
# actual: 0 predict: 0 -> log(0 * (0 - 0) + (1 - 0) * (0 + 0)) = -inf
# actual: 1 predict: 1 -> log(1 * (1 - 1) + (1 - 1) * (1 + 1)) = -inf
# actual: 1 predict: 0 -> log(1 * (1 - 0) + (1 - 1) * (1 + 0)) = 0
# actual: 0 predict: 1 -> log(0 * (0 - 1) + (1 - 0) * (0 + 1)) = 0
log_lik = K.log(y_true * (y_true - y_pred) + (1 - y_true) * (y_true + y_pred))
return K.mean(log_lik * adv, keepdims=True)
model_train = Model(inputs=[inp, adv], outputs=out)
model_train.compile(loss=custom_loss, optimizer=Adam(lr))
model_predict = Model(inputs=[inp], outputs=out)
...
loss = model_train.train_on_batch([states, discounted_rewards], actions_train)
I expected the loss function to take into account the probability distribution output by the network, the chosen action, and the advantage (discounted rewards), but am confused in regard to the custom loss function implemented.