Derivation of V(s) and Q(s, a) from the performance objective in MDP's

Viewed 44

I'm considering the paper Policy Optimization with Model-based Explorations, section 2 on the derivation of V(s) and Q(s, a)

The goal of the agent is to find the policy π∗ from a restricted family of parametrized policy functions that maximizes its performance,

enter image description here

The paper states in equation 2 where the performance objective is defined as follows:

where J(π) is the performance objective defined as

enter image description here

Next, the authors create this arbitrary function (eq 3), defined as:

For convenience, let ρ π (s) denote the (unnormalized) discounted cumulated state distribution induced by policy π,

enter image description here

And for some reason they use this to get the following:

then the performance objective can rewrite as

enter image description here

Furthermore, they find the following based on the previous derivation.

enter image description here

enter image description here

My questions are the following:

  1. Could anyone explain to me, how to construct this rho function
  2. How they construct the new J function?
  3. Based on the new J function, how do they construct the Q from that?
  4. Also, I've not seen this notation for V before, what do the authors do here?
0 Answers
Related