deal with illegal move on Reinforcment Learning

user_14529479 0
Viewed 84

I'm trying to create an agent to play 'nim' game. Nim game picture

(numbers are the 'id' associated to each links)

I created a gym env.

My observation space is: Box(0,2, shape=(,144)) of type integer.

There are 144 possible links

0 = no link

1 = link connected by us

2 = link connected by opponent

My action space is: Discrete(144) We chose the number of the link to connect.

At the beginning all 144 link are possible to connect And as the game goes on, links are connected and we can't play on a connected link.

My rewards are:

  • +100 if we connect a link
  • -1000 if opponent close a box
  • +1000 if we close a box
  • -10000 if opponent win the game
  • +10000 if we win the game
  • -100000000 if we make an inegal move

here is the code I use to train my agent :

env = DotsAndBoxesEnv()
state = env.reset()
model = A2C('MlpPolicy', env, verbose=1, tensorboard_log=logPath)
model.learn(total_timesteps=1000000)

the problem is that the agent use to play only illegal moves when I train it. to be more precise, he play the same number over and over.

How could I improve his training ?

openai-gym stable-baselines reinforcement-learning deep-learning python
1 Answers

I had a similar problem in my custom multiplayer environment. What seemed to be the reason was that the opponent my agent was training against was an RNG, basically, which just chooses a random legal move. My guess is that random moves from the opponent give too little info for training and the nnet can't just figure out how to move.

What solved the problem for me was my decision to make the opponent a MiniMax algorithm, so it has at least some logic behind its behavior. That seemed to solve the issue with nnet choosing illegal moves after just 10 minutes of training.

user_158857900
Related