Getting random array elements from hdf5

user_8793975 0
Viewed 2083

I'm training a neural net and the training set I have is stored on a 1.5Tb hdf5 file which is segmented into 125 groups (grp0...grp124). Each group contains 2 datasets 'x' and 'y' which were numpy arrays saved to hdf5 format. Each dataset x and y consists of roughly 10,000 training examples give or take a few hundred (the groups are not uniform in size, about 1.2 million training examples total). What I want to do is generate random training minibatchs on the fly from this hdf5 file. Right now, the code I have for generating such a batch looks like this:

    #grps is a dictionary of group sizes (they are not exactly uniform)
    def data_gen(PATH,grps,batch_size=32):
        while(True): 
            pick_grp = random.randint(0,len(grps)-1)
            size = grps['grp_'+str(pick_grp)]
            indices = random.sample(range(size),batch_size)
            with h5py.File(PATH,'r') as f:
                X = f['grp'+str(pick_grp)]['x'][indices]
                Y = f['grp'+str(pick_grp)]['y'][indices]
                yield (X,Y)

As you can see, what I'm trying to do is pick a random grp, and then from that grp pick a random batch_sized number of training examples. With numpy arrays, this slicing would have worked, but apparently h5py does not support this kind of slicing of a dataset, I get the error 

    TypeError: Indexing elements must be in increasing order

when I try to run this code. Alternatively, I could manually construct a batch one training example at a time using a for-loop, but I assume that would completely slow down my code. I run mini-batches using a GeForce 1080Ti so training on one single mini-batch is very fast. I don't want to be bottlenecked by the CPU process of constructing mini-batches. Is there a clever way to do what I want to do? I've also considered simply choosing a random starting index and then grabbing a mini-batch from the grps like:

    i = random.randint(0,size-batch_size-1)
    X=f['grp'+str(pick_grp)]['x'][i:i+batch_size]

and I could shuffle the mini-batch after I grab it. But this means that adjacent training examples will likely always appear in the same mini-batch. I don't know what kind of an effect (if any) this could have on the neural network training, but it seems sub-optimal to the random indices slicing approach I have now. Ideally, I wanted to be able to grab examples from any of the 125 groups in one mini batch as well (the file was segmented into 125 groups because of memory restrictions during the production of this hdf5 file), but I haven't figured out a way to do that.

h5py hdf5 python
1 Answers

My solution is simply using sorted() the list of indices then shuffle the batch again:

def data_gen(PATH,grps,batch_size=32):
        while(True): 
            pick_grp = random.randint(0,len(grps)-1)
            size = grps['grp_'+str(pick_grp)]
            indices = random.sample(range(size),batch_size)
            with h5py.File(PATH,'r') as f:
                X = f['grp'+str(pick_grp)]['x'][sorted(indices)]
                Y = f['grp'+str(pick_grp)]['y'][sorted(indices)]
                another_idx = np.random.permutation(X.shape[0]) #shuffle after reading data but not necessary
                yield (X[another_idx],Y[another_idx])
user_92171780
Related