My team is implementing a database to store machine learning datasets. Currently, we compress numeric numpy arrays (as reshaped 1-dimensional arrays) using lz+zstd; and, compress numpy arrays containing string values (as objects) using _pickle+lz+zstd.
We are a bit worried about the compatibility of such storage choices and wondering if it would be easily accessible by Java/Scala, C/C++ or other eventual clients (Julia, R) that could share the same information.
Is the raw binary info inside a numpy array universal in this sense, or just arbitrary?
ps. The choice of combining two compressors was made after speed vs space benchmarks.