I'm considering the paper Policy Optimization with Model-based Explorations, section 2 on the derivation of V(s) and Q(s, a)
The goal of the agent is to find the policy π∗ from a restricted family of parametrized policy functions that maximizes its performance,
The paper states in equation 2 where the performance objective is defined as follows:
where J(π) is the performance objective defined as
Next, the authors create this arbitrary function (eq 3), defined as:
For convenience, let ρ π (s) denote the (unnormalized) discounted cumulated state distribution induced by policy π,
And for some reason they use this to get the following:
then the performance objective can rewrite as
Furthermore, they find the following based on the previous derivation.
My questions are the following:
- Could anyone explain to me, how to construct this rho function
- How they construct the new J function?
- Based on the new J function, how do they construct the Q from that?
- Also, I've not seen this notation for V before, what do the authors do here?





