What is the difference between Neural Network Frameworks and RL Algorithm Libraries?

user_14622788 0
Viewed 215

I know this is a silly question, but I cannot find a good way to put it.

I've worked with TensorFlow and TFAgents, and am now moving to Ray RLlib. Looking at all the RL frameworks/libraries, I got confused about the difference between the two below:

  • frameworks such as Keras, TensorFlow, PyTorch
  • RL implementation libraries such as TFAgents, RLlib, OpenAi Baseline, Tensorforce, KerasRL, etc

For example, there are Keras codes in TensorFlow and Ray RLlib supports both TensorFlow and PyTorch. How are they all related?

My understanding so far is that Keras allows to make neural networks and TensorFlow is more of a math library for RL (I don't have enough understanding about PyTorch). And libraries like TFAgents and RLlib use frameworks like Keras and TensorFlow to implement existing RL algorithms so that programmers can utilize them with ease.

Can someone please explain how they are interconnected/different? Thank you very much.

reinforcement-learning ray pytorch keras tensorflow
1 Answers

Yes you are kind of right. Frameworks like Keras, TF (which also uses keras btw) and Pytorch are general Deep Learning frameworks. For most artificial neural network use-cases these frameworks work just fine and your typical pipeline is going to look something like:

  1. Preprocess your dataset
  2. Select an appropriate model for this problem setting
  3. model.fit(dataset)
  4. Analyze results

Reinforcement Learning though is substantially different from most other Data Science ML applications. To start with, in RL you actually generate your own dataset by having your model (the Agent) interact with an environment; this complicates the situation substantially particularly from a computational standpoint. This is because in the traditional ML scenario most of the computational heavy-lifting is done by that model.fit() call. And the good thing about the aforementioned frameworks is that from that call your code actually enters very efficient C/C++ code (usually also implementing CUDA libraries to use the GPU).

In RL the big problem is the environment that the agent interacts with. I separate this problem in two parts:

a) The environment cannot be implemented in these frameworks because it will always change based on what you are doing. As such you have to code the environment and - chances are - it's not gonna be very efficient.

b) The environment is a key component in the code and it constantly intreacts multiple times with your Agent, and there are multiple ways in which that interaction can be mediated.

These two factors lead to the necessity to standardize the environment and the interaction between it and the agent. This standardization allows for highly reusable code and also code that is more interpretable by others in how it exactly operates. Furthermore it is possible this way to, for example, easily run parallel environments (TF-agents allows this for example) even though your environment object is not really written to manage this.

RL frameworks are thus providing this standardization and features that come with it. Their relation to Deep Learning frameworks is that RL libraries often come with a lot of pre-implemented and flexible agent architectures that have been among the most relevant in the literature. These agents are usually nothing more than a some fancy ANN architecture wrapped in some class that standardizes their operation within the given RL framework. Therefore as a backend for these ANN models, RL frameworks use DL frameworks to run the computations efficiently.

user_128129850
Related