Why does h5py throw an error when adding 3 variable length strings to a dataset?

user_33404 0
Viewed 205

I'm trying to set up and write to an HDF5 dataset using h5py (Python 3) that contains a one dimensional array of compound objects. Each compound object is made up of three variable length string properties.

     with h5py.File("myfile.hdf5", "a") as file:
         dt = np.dtype([
             ("label", h5py.string_dtype(encoding='utf-8')),
             ("name", h5py.string_dtype(encoding='utf-8')),
             ("id", h5py.string_dtype(encoding='utf-8'))])
         dset = file.require_dataset("initial_data", (50000,), dtype=dt)
         dset[0, "label"] = "foo"

When I run the example above, the last line of code causes h5py (or more accurately numpy) to throw an error saying:

"Cannot change data-type for object array."

Do I understand correctly that the type for "foo" is not h5py.string_dtype(encoding='utf-8')?

How come? And how can I fix this?

UPDATE 1: Stepping into the stacktrace, I can see that the error is thrown from an internal numpy function called _view_is_safe(oldtype, newtype). In my case oldtype is dtype('O') but newtype is dtype([('label', 'O')]) which causes the error to be thrown.

UPDATE 2: My question has been answered successfully below but for completeness I'm linking to a GH issue that might be related: https://github.com/h5py/h5py/issues/1921

h5py hdf5 numpy python
1 Answers

You're setting the dtype as a tuple of variable length strings, so you'd set the tuple all at once. By only setting the label element, the other two tuple values aren't being set, so they are not string types.

Example:

import h5py
import numpy as np

with h5py.File("myfile.hdf5", "a") as file:
    dt = np.dtype([
        ("label", h5py.string_dtype(encoding='utf-8')),
        ("name", h5py.string_dtype(encoding='utf-8')),
        ("id", h5py.string_dtype(encoding='utf-8'))])
    dset = file.require_dataset("initial_data", (50000,), dtype=dt)

#Add a row of data with a tuple:
    dset[0] = "foo", "bar", "baz"
 
#Add another row of data with a np recarray (1 row):
    npdt = np.dtype([
        ("label", 'S4'),
        ("name", 'S4'),
        ("id", 'S4') ])
    dset[1] = np.array( ("foo1", "bar1", "baz1"), dtype=npdt )
       
#Add 3 rows of data with a np recarray (3 rows built from a list of arrays):
    s1 = np.array( ("A", "B", "C"), dtype='S4' )
    s2 = np.array( ("a", "b", "c"), dtype='S4' )
    s3 = np.array( ("X", "Y", "Z"), dtype='S4' )
    recarr = np.rec.fromarrays([s1, s2, s3], dtype=npdt)
    dset[2:5] = recarr

Result #1:
output

Result using all 3 methods:
enter image description here

user_4090070
Related