B"H
Is there any way to implement a data generator (either by creating a python generator or by subclassing Sequence) for training data if I don't know in advance how many records I will have in an epoch?
Often times, even if you can't load the entire training set into memory, you still know how may items you will have - for images you can get a file count, or something similar. But sometimes you don't know in advance and it is too expensive to get an exact count - if you have multiple variable length files each with many records, or if you are being fed the training data by an outside source. You may actually have more data at each epoch, by the time the first epoch is done, there may have been more data collected.
So you don't know what number to pass to steps_per_epoch or have a real way to implement __len__
Currently I am actually working on a cnn but this would be more comon by an lstm