Given a list of list of strings, such as:
test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
I'd like to store it using h5py such that:
f['test_dataset'][0] = ['a1','a2']
f['test_dataset'][0][0] = 'a1'
etc.
Following the advice in the thread H5py store list of list of strings, I tried the following:
import h5py
test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
with h5py.File('test.h5','w') as f:
string_dt = h5py.special_dtype(vlen=str)
f.create_dataset('test_dataset',data=test_array,dtype=string_dt)
However this results in each of the nested lists being stored as strings, i.e.:
f['test_dataset'][0] = "['a1', 'a2']"
f['test_dataset'][0][0] = '['
If this isn't possible using h5py, or any other hdf5-based library, I'd be happy to hear any suggestions of other possible formats/libraries that I could use to store my data.
My data consists of multidimensional numpy integer arrays and nested lists of strings as in the example above, with around >100M rows and ~8 columns.
Thanks!