I am working on microservices architecture and using kafka to communicate b/n microservices. I am using python faust to implement producers and consumers.
Is it mandatory to install faust[rocksdb] extension? When I install it, fails with a
fatal error: rocksdb/slice.h: No such file or directory.
What could be the impact I could see in production if I am not using rocksdb? If the package is mandatory to install how can I solve the issue? I had tried many answers, but nothing seems to work.
RocksDB is not a required dependency of Faust. By default a Faust app saves changelogs to memory, which is in their documentation.
Some systems may have issues installing faust[rocksdb] but I've had success installing it inside a Docker container. I suggest reading this SO answer for more details since they're having the same issue with slice.h.
Be warned that the wheel on PyPi for python-rocksdb is outdated, you'll need to manually install a newer version from https://github.com/twmht/python-rocksdb. There are more features in its actively maintained fork (https://github.com/NightTsarina/python-rocksdb), but I only got that working inside an Ubuntu 22.04 container.
What could be the impact I could see in production if I am not using rocksdb?
If your Faust application needs to frequently restart and your changelogs have a lot of activity, then you'll have to wait a while to wait for your app to update your local state. This is also mentioned in the Faust documentation. This wait time could range from a few seconds to minutes depending on how frequently your tables change.
Update (08/11/2022): I have updated faust-streaming's fork of python-rocksdb, try this:
pip3 install faust-streaming-rocksdb
The C++ dependencies are bundled with this now and I'm able to install it onto Ubuntu 18.04 LTS.
Disclaimer: I'm a maintainer of the fork.