Why is Policy Gradient method for Reinforcement Learning good for LARGE action spaces

user_14639080 0
Viewed 206

As stated in the title, I have read multiple sources that Policy Gradient methods are RL are suitable for large actions spaces, but I dont understand why is this so.

I am trying to see if RL can tackle a problem of mine that has a huge combinatorial no. of possible actions. Hypothetically it is about sending n no. of items from location i to j. Any combination of (i,j,n) is a possible action, and (i,j,n) all have magnitude in the 1000s, this makes more than a billion possible actions.

Since the output layer nodes of the neural net in Policy Gradient methods represents the no. of actions. With >1000,000,000 possible actions, how can Policy Gradient be a good method to solve such problems?

policy gradient
1 Answers

For large or continuous action spaces, you need to use function approximation methods to approximate the optimal policy. This is known as policy approximation. There are a number of possible approaches that include least-squares optimization or gradient-based optimization. Nearly all of these techniques utilize random sampling to produce and compare possible actions that maximize return over the infinite-time horizon.

From Sutton and Barto's RL book 1:

Policy-based methods offer practical ways of dealing with large action spaces, even continuous spaces with an infinite number of actions. Instead of computing learned probabilities for each of the many actions, we instead learn statistics of the probability distribution. For example, the action set might be the real numbers, with actions chosen from a normal (Gaussian) distribution.

Check out:

  • Section 13.7 in Sutton and Barto's book for more theoretical explanation
  • This GitHub repo for code examples that have viable approaches for this problem
user_80725460
Related