I have many objects ( 1000 - 30k ) which may consist of float vertex data, integer indices, etc. In average they may be from 600 Kb to 6+ Gb. Also I have bmp textures and other various data. I want to store them into a single database or something. I've tried various NoSQL DBs like BerkeleyDB, RocksDB they cannot store such large objects 4gb + and work so slow with my data. I've looked at various archive formats but none of them fit into my needs.
What I need from the storage:
- Fast write and read performance ( it's desirable to be parallel )
- Multi-core compression/decompression ( ZSTD would be perfect )
- Random access ( read, modify, delete and insert )
- Preferable if it stores everything into a single file
- Convenient and simple usage
- C/C++ API
I've also tried TileDB but it's currently very buggy, everything was almost fine but read performance is terrible like retrieving 1 Mb of vertex data takes 3 seconds which is unacceptable for me, may be it's a bug or something I don't know.
I ran out of ideas, so I need help, maybe there is something that perfectly fits into needs but I didn't notice it.