I have an application where I need to use long keys to store values in a key value store database. If I try to store a value using a long key (greater than 511 bytes) in lmdb, using the python API, I receive the following error:
lmdb.BadValsizeError: mdb_put: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
See the example code:
>>> short_key = 'short_key'
>>> long_key = 'long_key' * 100
>>> print(f'short_key ({len(short_key.encode())} bytes), long_key ({len(long_key.encode())} bytes)')
short_key (9 bytes), long_key (800 bytes)
Initialise lmdb environment and check max key size:
>>> import lmdb
>>> env = lmdb.open('db_folder')
>>> print(f'Max key bytes = {env.max_key_size()}')
Max key bytes = 511
Insert value using short key works fine:
>>> with env.begin(write=True) as txn:
... txn.put(short_key.encode(), '1'.encode())
...
True
However with long key the following error is raised:
>>> with env.begin(write=True) as txn:
... txn.put(long_key.encode(), '2'.encode())
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
lmdb.BadValsizeError: mdb_put: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
The documentation here says :
By default record keys are limited to 511 bytes in length, however this can be adjusted by rebuilding the library. The compile-time key length can be queried via Environment.max_key_size().
I am not sure how to follow these instructions? Any advice on how to get around this issue much appreciated.