h5py: Store list of list of strings

Viewed 2751

Given a list of list of strings, such as:

test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]

I'd like to store it using h5py such that:

f['test_dataset'][0] = ['a1','a2']
f['test_dataset'][0][0] = 'a1'
etc.

Following the advice in the thread H5py store list of list of strings, I tried the following:

import h5py
test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
with h5py.File('test.h5','w') as f:
    string_dt = h5py.special_dtype(vlen=str)
    f.create_dataset('test_dataset',data=test_array,dtype=string_dt)

However this results in each of the nested lists being stored as strings, i.e.:

f['test_dataset'][0] = "['a1', 'a2']"
f['test_dataset'][0][0] = '['

If this isn't possible using h5py, or any other hdf5-based library, I'd be happy to hear any suggestions of other possible formats/libraries that I could use to store my data.

My data consists of multidimensional numpy integer arrays and nested lists of strings as in the example above, with around >100M rows and ~8 columns.

Thanks!

2 Answers
Related