I am currently having a problem with getting the metrics from the pre-built estimator models.This my code to extract the metrics from the model:
v_metrics = {
"accuracy":
tf.contrib.learn.MetricSpec(
metric_fn=tf.contrib.metrics.streaming_accuracy,
prediction_key = tf.contrib.learn.PredictionKey.SCORES),
"precision":
tf.contrib.learn.MetricSpec(
metric_fn=tf.contrib.metrics.streaming_precision,
prediction_key = tf.contrib.learn.PredictionKey.SCORES),
"recall":
tf.contrib.learn.MetricSpec(
metric_fn=tf.contrib.metrics.streaming_recall,
prediction_key = tf.contrib.learn.PredictionKey.SCORES)
}
v_monitor = tf.contrib.learn.monitors.ValidationMonitor(
input_fn=lambda: input_fn(df_test),
eval_steps = 1,
every_n_steps=10,
metrics=v_metrics
)
m = build_estimator(model_dir, model_type)
m.fit(input_fn=lambda: input_fn(df_train), steps=train_steps, monitors=[v_monitor])
results = m.evaluate(input_fn=lambda: input_fn(df_test), steps=1)
However, when I run this it gives me:
INFO:tensorflow:Saving dict for global step 28: accuracy = 0.0, global_step = 28, loss = 24.7031, precision = 1.0, recall = 1.0
I looked into the API and the estimator.py and metric_spec.py scripts, but nowhere did I find a way to extract the tensors responsible for the accuracy, precision, and recall. I also looked at other answers, but all them are solutions for custom estimators. I also checked the results on tensorboard and its still 0s and 1s.
Note that when I used the exact same method for tf.contrib.learn.DNNLinearCombinedRegressor with prediction_key = tf.contrib.learn.PredictionKey.CLASSES It works properly.
I am not sure what I am doing wrong with the combined Regressor. I would like to know where to get the proper prediction keys (or what I am doing wrong), so when I run the model, it gives me the proper metric results.