How to set your own value function in Reinforecement learning?

nuemlouno 0
Viewed 29

I am new to using reinforcement learning, I only read the first few chapters in R.Sutton (so I have a small theoretical background).

I try to solve a combinatorial optimization problem which can be broken down to:

I am looking for the optimal configuration of points (qubits) on a grid (quantum computer).

I already have a cost function to qualify a configuration. I also have a reward function.

Right now I am using simulated annealing, where I randomly move a qubit or swap two qubits.

However, this ansatz is not working well for more than 30 qubits.

That's why I thought to use a policy, which tells me which qubit to move/swap instead of doing it randomly.

Reading the gym documentation, I couldn't find what option I should use. I don't need Q-Learning or deep reinforcement learning as far as I understood since I only need to learn a policy?

I would also be fine using Pytorch or whatever. With this little amount of information, what do you recommend to chose? More importantly, how can I set my own value function?

reinforcement-learning python
1 Answers

There are two categories of RL algorithms.

One category like Q-learning, Deep Q-learning and other ones learn a value function that for a state and an action predicts the estimated reward that you will get. Then, once you know for each state and each action what the reward is, your policy is simply to select for each state the action that provides the biggest reward. Thus, in the case of these algorithms, even if you learn a value function, the policy depends on this value function.

Then, you have other deep rl algorithms where you learn a policy directly, like Reinforce, Actor Critic algorithms or other ones. You still learn a value function, but at the same time you also learn a policy with the help of the value function. The value function will help the system learn the policy during training, but during testing you do not use the value function anymore, but only the policy.

Thus, in the first case, you actually learn a value function and act greedy on this value function, and in the second case you learn a value function and a policy and then you use the policy to navigate in the environment.

In the end, both these algorithms should work for your problem, and if you say that you are new to RL, maybe you could try the Deep Q-learning from the gym documentation.

Otniel-Bogdan Mercea0
Related