What is Keras doing if my sample size is smaller than my batch size?

Viewed 953

fairly new to LSTM, but I already searched for a solution and could not find anything satisfying or even similar enough.

So here is my problem: I am dealing with sleep classifaction and have annotated records for about 6k patients. To train my bidirectional LSTM, I pick one patient and fit the model on that data instead of putting all the data from all the patients into one big matrix because I want to prevent patient samples mixing when Keras is training with mini batches. The sequence length or samples_size per patient are not the same. Then I loop over all patients and do a additional loop for the number of epochs I considered to train the model for (as described in Developer Guides).
So since LSTMs (if not stateful) reset their cell and hidden state after each batch and the default batch_size for tf.keras.Sequential.fit() is 32 I wanted it to match the sample_size of the patient I am showing to the network. If I do so I am getting a warning and the training process errors after some time. The error is:

WARNING:tensorflow:6 out of the last 11 calls to .distributed_function at 0x0000023F9D517708> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/beta/tutorials/eager/tf_function#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

So I looked up what my longest sample_size is and set my batch_size accordingly.


tl;dr: What is Keras doing in all the instances where my variable sample_size is not matching my batch_size=max(len(sample_size))?

  1. Is it just showing the available samples to the network?
    • If so: Why is there the warning mentioned above where setting the batch_size=sample_size leads to the failed training?
  2. Or is it showing the available samples to the network and filling up the rest with zeros to match the given batch_size?
    • If so: Why is there the necessity of masking when using e.g. stateful mode?

edit: So, I tried some additional workarounds and built my own Data Generator, which proves data of one patient as one batch. I then set steps_per_epoch=len(train_patients) to include all patients into one epoch. No warnings about retracing, which I do not understand either.
It seems to solve my problem of showing one patient per batch without mixing patient data and have a variable sample_size, but I really do not understand the differences between all these posibilities and their different warnings.

0 Answers
Related