keras-rl2 reinforcement learning agent reshaping data

user_11664078 0
Viewed 558

I am trying to build a reinforcement learning agent to learn off a custom environment, built to openai's gym specifications.

I have np arrays of size (20, 7) which I want to pass to the network, and output one of 7 actions.

I am having trouble building the actual network, as I want to include LSTM layers. My code is as follows:

def build_model():

    model = Sequential()
    model.add(LSTM(60, return_sequences = True, input_shape=(20, 7), activation = 'relu'))
    model.add(Dense(21, activation = "relu"))
    model.add(Flatten())
    model.add(Dense(7, activation="linear"))
    model.compile(loss="mse", optimizer=Adam(lr=0.0002), metrics=['accuracy'])
    return model

However, when I build the agent, there is suddenly an extra dimension added on which the network does not expect:

def build_agent(model, actions):
    policy = BoltzmannQPolicy()
    memory = SequentialMemory(limit=50000, window_length = 1)
    dqn = DQNAgent(model=model, memory=memory, policy=policy, 
                  nb_actions=actions, nb_steps_warmup=10, target_model_update=1e-2)
    return dqn

dqn = build_agent(model, actions)
dqn.compile(Adam(lr=1e-3), metrics=['mae'])
dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)

ValueError: Error when checking input: expected lstm_input to have 3 dimensions, but got array with shape (1, 1, 20, 7)

Im not exactly sure why the agent is reshaping the data to add an extra dimension, (or two?) but if anyone had an idea on how to stop this from happening so I can train my network I would be very grateful. My solution runs when I code it myself however I want to make use of the keras rl-2 library.

Thanks in advance!

reinforcement-learning keras tensorflow python
3 Answers

For anyone looking for an answer, I fixed this by adding layer:

model.add(Reshape((20, 7), input_shape=(1, 20, 7)))

as the first layer

user_116640780

As I understand it your agent is able to compute your environment but is unable to compute other keras rl-2 environments because they add another dimension to the feature vector (the input). I believe this is because the environments you are trying to run returns a feature vector that includes channels. Channels simply mean how many values do you need to describe one pixel. For instance, RGB needs 3 channels while your environment returns a simplification of only one channel.

Since you are only interested in one channel you could either squeeze away that channel since you do not need it with:

state = state.squeeze(axis=1)

Before passing it into the net. Or you could define your model to include channels by setting your input to be (1,20,7) which could be useful if you in the future want to apply convolutional layers which require a defined count of channels.

user_152407050

try this at first layer

window_length = 1
model.add(Flatten(input_shape=(window_lenght ,) + env.observation_space.shape))
user_128869850
Related