I am looking for a long-term storage for my data processing system that allows to add records, consume them using multiple consumer groups with ability to replay the stream from the first existing record, and that allows to delete selected messages when needed.
So, basically, just like Kafka, but with ability to remove random records. I know, it's possible to keep the list of unwanted records in some other storage and filter incoming messages using it, but in my case the size of removed messages my reach a few terabytes in one year and this is just not rational to keep this dead data on disk. I'm also aware of the Kafka's tombstone records feature, but the marked for removal records are kept for some time, so, again, I have to store the list of unwanted messages somewhere, which makes the consumption process slower and more complex.
The decision of keeping or removing a specific record is not time-based but depends on the content of the message, so it's impossible to deal with it truncating the Kafka topic to a specific date.
I used Google to find a ready-to-use product for my criteria, but I didn't find anything.
The obvious solution is to build a DB-based service that will handle this, but it feels like I'm reinventing the wheel here, and this whole "side quest" could become more complex than the data processing components themselves.
I'd like to know if there is already a project that may fit my needs.