Join metrics of every output in Keras (in multiple output)

Question

I am working on a multiple output model in Keras. I've implemented two custom metrics, auroc and auprc, that are passed to the compile methods of Keras model:

def auc(y_true, y_pred, curve='PR'):
    score, up_opt = tf.compat.v1.metrics.auc(y_true, y_pred, curve=curve, summation_method="careful_interpolation")
    K.get_session().run(tf.local_variables_initializer())
    with tf.control_dependencies([up_opt]):
        score = tf.identity(score)
    return score

def auprc(y_true, y_pred):
    return auc(y_true, y_pred, curve='PR')

def auroc(y_true, y_pred):
    return auc(y_true, y_pred, curve='ROC')

mlp_model.compile(loss=...,
                    optimizer=...,
                    metrics=[auprc, auroc])

Using this method, I obtain an auprc/auroc values for every output but, to optimize my hyperparameters with a Bayesian optimizator, I need a single metrics (e.g: the average or the sum of auprc for every output). I can't figure out how I can join my metrics in a single one.

EDIT: here an example of desired results

Now for every epochs the following metrics are printed:

out1_auprc: 0.0267 - out2_auprc: 0.0277 - out3_auprc: 0.0294

where out1, out2, out3 are my neural network outputs, I desire to obtain something like:

average_auprc: 0.0279 - out1_auprc: 0.0267 - out2_auprc: 0.0277 - out3_auprc: 0.0294

I am using Keras Tuner for Bayesian Optimization.

Any help is appreciated, thank you.

user_10375049 · Answer

I override the problem creating a custom callback

class MergeMetrics(Callback):

    def __init__(self,**kargs):
        super(MergeMetrics,self).__init__(**kargs)

    def on_epoch_begin(self,epoch, logs={}):
        return

    def on_epoch_end(self, epoch, logs={}):
        logs['merge_metrics'] = 0.5*logs["y1_mse"]+0.5*logs["y2_mse"]

I use this callback to merge 2 metrics coming from 2 different outputs. I use a simple problem for example but you can integrate it easily in your problem and integrate it with a validation set

this is the dummy example

X = np.random.uniform(0,1, (1000,10))
y1 = np.random.uniform(0,1, 1000)
y2 = np.random.uniform(0,1, 1000)


inp = Input((10))
x = Dense(32, activation='relu')(inp)
out1 = Dense(1, name='y1')(x)
out2 = Dense(1, name='y2')(x)
m = Model(inp, [out1,out2])
m.compile('adam','mae', metrics='mse')


checkpoint = MergeMetrics()
m.fit(X, [y1,y2], epochs=10, callbacks=[checkpoint])

the printed output

loss: ..... y1_mse: 0.0863 - y2_mse: 0.0875 - merge_metrics: 0.0869

Join metrics of every output in Keras (in multiple output)

1 Answers