Kafka: Publish message only if it doesn't already exist

user_2737956 0
Viewed 1295

I am publishing messages to Kafka which are consumed by a number of consumers. These are idempotent so it doesn't matter if one message is consumed multiple times.

However for performance reasons I don't want my (single) producer to publish a message that already exists in the queue. Let's say messages are just ID-strings so it's very easy to check if two messages are equal.

I suppose that Kafka alone isn't made to deal with this kind of performance improvement. Are there any tools or concepts to help dealing with this issue?

apache-kafka scala java performance
1 Answers

Kafka is not really well-suited to anything that boils down to looking for a needle in a haystack, because all you can do is linear search.

Getting rarely-more-than-once delivery in Kafka is probably best accomplished by using an external datastore with much better query support (Cassandra or Elasticsearch are two of a great many options here). Then you have the producer check if the message it wants to write is in that datastore, and you have a consumer of the topic whose sole purpose is to write the messages to that datastore. Absent size-based retention and a keying scheme where different messages have the same keys, this should be fail-safe (meaning I can't think of how it's not): you won't erroneously decide not to write a message which you should write.

user_56412440
Related