How to train PPO using actions from matches already played?

user_12417798 0
Viewed 164

The idea is to initially calibrate the neural network with some prior knowledge before releasing the algorithm to evolve on its own. To make the question simpler, imagine that an agent can take 10 actions (discrete space). Instead of training the PPO algorithm to figure out by itself which actions are best for each state, I would like to perform a training by considering that some actions were performed in some states. I'm using Stable Baselines with Gym.

I thought about creating an action wrapper like this:

class RandomActionWrapper(gym.ActionWrapper):
            def __init__(self, env):
            super(RandomActionWrapper, self).__init__(env)
            def action(self, action):
                a = self.env.action_space.sample()
                return a

Ps: this wrapper is just a proof of concept, choosing random actions all the time, but the model just doesn't learn that way (I simulated many iterations in ridiculously simple to learn custom environments, something like: "action 2 always results in reward=1 while other actions result in reward=0). Apparently the updates on the network are being made considering the actions that the model chose (the model always predicts actions by itself) while the rewards are being calculated based on the actions defined in my wrapper. This mismatch makes learning impossible.

openai-gym stable-baselines reinforcement-learning deep-learning
2 Answers

I think you are looking for some kind of action mask implementation. In several games/enviroments, some actions are invalid in a particular state (it is not your case, but it could be the first approach). You can check this paper and the github

user_159724320

As PPO is an on-policy method, there is a mismatch between my generated data and the algorithm’s cost function. There's no reason to insist on PPO here. I'll look into off-policy algorithms

user_124177980
Related