Append hdf5 to another hdf5 file

user_5152497 0
Viewed 437

I have several hdf5 files with the same shape, containing x and y columns. I need to append those, to get one hdf5 file, which contains all the data.

My code so far:

def append_to_h5(new_file, file_list):
    f = h5py.File(new_file, 'a')
    for file in file_list:
        with h5py.File(file, 'r') as d:
            f.create_dataset("./", data=d)
    f.close()

#new_file <- is a file path to the new hdf5 file
#file_list <- contains all the pathes of the hdf5 files, which I want to append

The error

   in make_new_dset tid = h5t.py_create(dtype, logical=1)
  File "h5py/h5t.pyx", line 1634, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1656, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1717, in h5py.h5t.py_create
TypeError: No conversion path for dtype: dtype('<U1')

Any ideas are appreciated Thanks

h5py append hdf5 python
1 Answers

This is covered more extensively other SO answers. I created a short example to get you started. The primary change was adding a loop to find and copy top level datasets (only). It assumes there will be no dataset name conflicts and needs tests to be used for a general purpose case. Also, I changed your file object variable names.

def append_to_h5(new_file, file_list):
    f1 = h5py.File(new_file, 'a')
    for file in file_list:
        with h5py.File(file, 'r') as f2:
            for ds in f2.keys():
                f2.copy(ds, f1) 
    f1.close()

#new_file <- is a file path to the new hdf5 file
#file_list <- contains all the pathes of the hdf5 files, which I want to append
user_104628840
Related