I am training a sequence model on a dataset that consists of N sequences of different lengths, and the model predicts a binary label for each of the steps in each of the sequences.
Because the data doesn't fit in memory at once, I'm using the tf.dataset API to parse it in a streaming fashion and create batches of batch_size sequences each. Since the sequences have different lengths, each batch is padded to the length of the longest sequence in that batch, so the tensors that are sent to the model are of dimension (batch_size, max_sequence_length_in_batch, number_of_features).
Some of the time steps should be ignored: a) because they're there only due to padding, or b) because they're marked as "invalid" in one of the features.
I'm using masking layers to mask both types of time steps to be ignored.
Questions:
1) I have checked that the mask is used internally by Keras to ignore the loss and avoid backpropagating from time steps that are "masked out". Will these "masked out" time steps be ignored when computing metrics as well?
2) If so, will they be ignored with a custom implementation of a global, stateful metric similar to Tensorflow's tf.sensitivity_at_specificity, using a strategy similar to https://gist.github.com/vsocrates/7ff65268c2ed533a62f8f9f9d86786af or https://github.com/keras-team/keras/blob/master/tests/keras/metrics_test.py#L115? Note that the custom metric can only receive y_true and y_pred as arguments, so anything involving masks must be done upstream by Keras (i.e. filtering out "masked-out" time steps before calling the custom metric).