Creating a generator from list of sequences for RNN

Question

I need to create a generator for my data to pass into my RNN training function. I have a list of patient samples, where each sample is a time series of length n_i (which varies, annoyingly) in three dimensions, and I want to create batches of data where each sample in a batch only belongs to a single patient but each batch may contain multiple patient samples. Doing it this way should maximise the number of samples I can train using with no consequences as my RNN is not stateful. At first I had the following function

def dataIterator(rawDataList, config):
    batchSize, nSteps = config.batchSize, config.nSteps
    for rawData in rawDataList:
        dataLen, dataWidth = rawData.shape
        batchLen = dataLen // batchSize
        data = np.zeros([batchSize, batchLen, dataWidth], dtype=np.float32)
        for i in xrange(batchSize):
            data[i] = rawData[batchLen*i:batchLen*(i+1), :]

        epochSize = (batchLen - 1) // nSteps

        if epochSize == 0:
            raise ValueError('epoch_size == 0')

        for i in xrange(epochSize):
            x = data[:, i*nSteps:(i+1)*nSteps, :]
            y = data[:, i*nSteps+1:(i+1)*nSteps+1, :]
            yield (x, y)

However this trims each of the patient samples in order to fit the batch size. So I want something that creates all possible batches, including the undersized one at the end. However my unfamiliarity with generators has left me pretty confused. So far I've worked out it's going to have to use modulo aritmetic, but exactly how I'm not sure, so I've only got to this point:

def dataIterator(data, batchSize=batchSize, nSteps=nSteps, nDimensions=3):
    nTimePoints = sum([len(x) for x in data])
    totalBatchLen = 1+(nTimePoints-1)//batchSize
    newData = np.zeros([batchSize, totalBatchLen, nDimensions])
    for i in xrange(batchSize):
        ...

EDIT Here's a short example to show how I would solve the problem without using generators

import numpy as np
np.random.seed(42)
nPatients = 3
tsLength = 5
nDimensions = 3
rnnTSLength = 3
batchSize = 3
inputData = np.random.random((nPatients, tsLength, nDimensions))
inputData[1, :, :] *= 10
inputData[2, :, :] *= 100
outputData = []
for i in xrange(tsLength-rnnTSLength):
    outputData.append(inputData[0, i:i+rnnTSLength, :])
for i in xrange(tsLength-rnnTSLength):
    outputData.append(inputData[1, i:i+rnnTSLength, :])
for i in xrange(tsLength-rnnTSLength):
    outputData.append(inputData[2, i:i+rnnTSLength, :])
temp1 = np.array(outputData[:3])
temp2 = np.array(outputData[3:])
npOutput = np.array((temp1, temp2))
print npOutput

Which produces:

[[[[  3.74540119e-01   9.50714306e-01   7.31993942e-01]
[  5.98658484e-01   1.56018640e-01   1.55994520e-01]
[  5.80836122e-02   8.66176146e-01   6.01115012e-01]]

[[  5.98658484e-01   1.56018640e-01   1.55994520e-01]
[  5.80836122e-02   8.66176146e-01   6.01115012e-01]
[  7.08072578e-01   2.05844943e-02   9.69909852e-01]]

[[  1.83404510e+00   3.04242243e+00   5.24756432e+00]
[  4.31945019e+00   2.91229140e+00   6.11852895e+00]
[  1.39493861e+00   2.92144649e+00   3.66361843e+00]]]


[[[  4.31945019e+00   2.91229140e+00   6.11852895e+00]
[  1.39493861e+00   2.92144649e+00   3.66361843e+00]
[  4.56069984e+00   7.85175961e+00   1.99673782e+00]]

[[  6.07544852e+01   1.70524124e+01   6.50515930e+00]
[  9.48885537e+01   9.65632033e+01   8.08397348e+01]
[  3.04613769e+01   9.76721140e+00   6.84233027e+01]]

[[  9.48885537e+01   9.65632033e+01   8.08397348e+01]
[  3.04613769e+01   9.76721140e+00   6.84233027e+01]
[  4.40152494e+01   1.22038235e+01   4.95176910e+01]]]]

Which as you can see has two batches of size three, both of which contain two different 'patients' in them, but the time series for each 'patient' do not overlap.

Creating a generator from list of sequences for RNN

1 Answers