How to use Keras data generator (or a different method ) with multiple .npy files of different length?

user_10244372 0
Viewed 215

I am working on a dataset of 30,000 .npy files. Each file contains a matrix of the following form: (75, N), where 75 is the number of columns, which represents the number of features. N: number of rows, the values of these features over time and is different from the file to another.

I tried to use TimeseriesGenerator as mentioned in the keras documentation here:

data_gen = TimeseriesGenerator(data, targets,
                               length=10, sampling_rate=2,
                               batch_size=2)
  1. How to use this method with multiple npy files?
  2. Should each file must be matched by a class in the target array or each column?
keras deep-learning python
1 Answers

I solved this problem by creating a custom sequance. The custom sequance concatenates different sequences. This way when a certain time series comes to an end a smaller batch will be given and than it will continue on to the next sequance.

from keras.utils import Sequence

class ConcatGenerator(Sequence):
  # series is a list of timeseires generators
  def __init__(self, series, *args, **kwargs):
    super(*args, **kwargs)
    self.series = series

  def __len__(self):
    return sum([len(s) for s in self.series])

  def __getitem__(self, idx):
    cur = 0
    while idx >= len(self.series[cur]):
        idx -= len(self.series[cur])
        cur += 1
    return self.series[cur][idx]

Please note that this is toy code and you could make the get item more efficient.

So in your case you would create a time series generator from each of the npy files and than use them to create one ConcatGenerator.

user_16920450
Related