I have a ~100 GB HDF5 file containing two large three-dimensional arrays of floats (training and validation data):
import h5py
dset = h5py.File("my_dataset.hdf5", "r")
Performance is very good for loading multiple rows into memory:
sub_dset_1 = dset["train"][1000:2000]
But indexing over the other axes is extremely slow:
sub_dset_2 = dset["train"][:, :, 0]
Questions:
- What is the explanation for this behaviour?
- Which data formats can be used to store large data on disk, while being able to load only parts of it to memory (along different axes)?