MATLAB 7.3 files in python with hdf5storage inflates files and is slow to create files

user_7655871 0
Viewed 191

I'm attempting to use hdf5storage to write numpy data to a .mat file.

import hdf5storage

# For example
numpy_array = [array([(b'<detect>', 192, 1)], dtype=[('packet_sync', 'S8'), ('n_bytes', '<u4'), ('n_detect', '<u4')]), array([(b'<detect>', 192, 2)], dtype=[('packet_sync', 'S8'), ('n_bytes', '<u4'), ('n_detect', '<u4')])]

# The actual array is 192 bytes. and a binary file I am attempting to create a .mat file for contains thousands of these packets.

data = {"data": numpy_array}

hdf5storage.savemat(file_name="data.mat", mdict=data, format="7.3")

Using this conveniance function, or equivelantly

hdf5storage.write(data, '.', 'data.mat', matlab_compatible=True)

The file expands to >10X the binary file size, which is a python list with numpy dtypes composed of basic c types (<u4, <f4, <S8...). It also takes >1hour to process a 70MB file which seems like something isn't right, but I don't have a ton of experience with HDF5 format so this may be expected.

When testing saving a similar variable from MATLAB with

save("test.mat", 'variable', '-v7.3')

The file size is still much larger than the binary size. So as @hpaulj points out, HDF5 is not a compact format. But the time it takes to save in python is also not acceptable. In MATLAB, the file save in a few seconds, to save the same file using the hdf5storage library, it takes around an hour. Perhaps this library is just not performant?

Looking at the disk write speed while this is running though, I see a stat of 2-3 M/s via iotop while the file only grows ~0.5MB/s.

I would like to avoid writing to separate .mat files.

When using scipy's savemat, I am able to save files up to the matlab v5 limit of 2GB, but we are generating more data than that and would like to be able to use v7.3 matlab format. So the problem is with the hdf5storage library as scipy still works.

Is there some numpy dtype restrictions with matlab v7.3 format?

Why are these files getting inflated? Is there an option in hdf5storage that i'm missing? I've looked through the docs and partly through the code to no avail.

Alternatively, I may try loading an hdf5 file into MATLAB

import h5py
hf = h5py.File("test.h5", "w")
hf.create_dataset("data", data=data)
hf.close()

EDIT: I've discovered that my troubles may be dues to non-homogeneous data shape. I could have packets of variable size. HDF5 doesn't deal well with this apparently, so structuring the data for homogeneity is important.

h5py matlab hdf5 python
1 Answers

I don't have hdf5storage.

In [21]: numpy_array = np.array(
    ...:     [(b"<detect>", 192, 1)],
    ...:     dtype=[("packet_sync", "S8"), ("n_bytes", "<u4"), ("n_detect", "<u4")],
    ...: ) 
In [22]: numpy_array
Out[22]: 
array([(b'<detect>', 192, 1)],
      dtype=[('packet_sync', 'S8'), ('n_bytes', '<u4'), ('n_detect', '<u4')])
In [23]: numpy_array.nbytes
Out[23]: 16
In [24]: data = {"data": numpy_array}

But with the pre 7.3 format:

In [25]: from scipy import io
In [26]: io.savemat("test712.mat", data)
In [27]: io.loadmat("test712.mat")
Out[27]: 
{'__header__': b'MATLAB 5.0 MAT-file Platform: posix, Created on: Tue Feb 22 18:02:29 2022',
 '__version__': '1.0',
 '__globals__': [],
 'data': array([[(array(['<detect>'], dtype='<U8'), array([[192]], dtype=uint32), array([[1]], dtype=uint32))]],
       dtype=[('packet_sync', 'O'), ('n_bytes', 'O'), ('n_detect', 'O')])}
In [28]: ll test712.mat
...
-rw-rw-r-- 1 paul 408 Feb 22 18:02 test712.mat

You did give specifics about the 'inflation', but the 16 byte array was saved to a 408 byte file.

With the native numpy save, the file a bit smaller. Most of that is the header block that specifies shape and dtype:

In [29]: np.save("test712.npy", numpy_array)
In [30]: ll test712.npy
-rw-rw-r-- 1 paul 208 Feb 22 18:05 test712.npy

And with the more basic h5py save:

In [32]: f = h5py.File("test712.h5", "w")
In [33]: f.create_dataset("array", data=numpy_array)
Out[33]: <HDF5 dataset "array": shape (1,), type "|V16">
In [34]: f.close()
In [35]: %ll test712.h5
-rw-rw-r-- 1 paul 2064 Feb 22 18:08 test712.h5

In [37]: f = h5py.File("test712.h5", "r")
In [40]: f["array"][:]
Out[40]: 
array([(b'<detect>', 192, 1)],
      dtype=[('packet_sync', 'S8'), ('n_bytes', '<u4'), ('n_detect', '<u4')])

HDF5 file format is not a compact one, so I think talking about inflation is meaningless. In this case, I suspect most of the size is due to layout and headers, not to the data itself. Saving a 4 element array (64 bytes instead of 16) probably won't change the file size that much - for any of the formats.

user_9019250
Related