TF Metric ChosenActionHistogram crashes with error

user_13492017 0
Viewed 98

When using TF's tf_agents.metrics.tf_metrics.ChosenActionHistogram with TF's dynamic step driver and my own environment, I encounter the following error:

ValueError: Shapes must be equal rank, but are 1 and 0 for '{{node ResourceScatterUpdate}} = ResourceScatterUpdate[Tindices=DT_INT32, dtype=DT_INT32](ResourceScatterUpdate/resource, FloorMod, value)' with input shapes: [], [], [1]

I've attached observers to the step driver like so:

self.average_episode_length_metric = tf_metrics.AverageEpisodeLengthMetric()
self.average_return_metric = tf_metrics.AverageReturnMetric()

self.selected_action_histogram_metric = tf_metrics.ChosenActionHistogram()
self.observers = [self.average_episode_length_metric,
                  self.average_return_metric,
                  self.selected_action_histogram_metric
                  ]
self.eval_step_driver = dynamic_step_driver.DynamicStepDriver(
            self.eval_env,
            self.agent.policy,
            num_steps=self.num_eval_steps,
            observers=self.observers
            )

and then run the step driver like such:

self.eval_step_driver.run()

Some more of the error trace is as follows:

File "./bot/DQN.py", line 109, in record_policy_metrics
    self.eval_step_driver.run()
    tf_agents-0.4.0-py3.8.egg/tf_agents/metrics/tf_metrics.py:50 extend  *
        self.add(v)

I understand the premise of the issue, that tensor shapes are not matching, but I can't figure out why that might be happening. Removing ChosenActionHistorgram from the observers resolves the error and the other metrics work correctly. What could be going on here? Could the trajectory tensors be missing some value?

reinforcement-learning tensorflow python
1 Answers

For anyone who comes across this issue I solved it for my case. I had mistakenly defined the action spec as (1,), a 1 dimensional vector, instead of (), a scalar value. This seemed to work for every other metric except for the tf_metrics.ChosenActionHistogram().

Ensuring my action spec complied with () instead of (1,) resolved the issue.

user_134920170
Related