Is it safe to use Redis pool inside kafka streams?

user_7285198 0
Viewed 715

So, the question is about safety to use external state storage inside stream functions like filter, map and etc.

Is it ok to do something like this:

JedisPool pool = ...;
KStream stream = ...;
stream.map((k, v) -> {
    JedisClient client = pool.getResource();
    ....
    client.close();
});
...
KafkaStreams streams = ...;

Can it cause errors because of using single pool inside multiple streaming tasks? In apache flink i can use Rich*Function<> where i can configure connection pool to any storage only once inside open method. In apache spark i also can configure global connections. Do i need to do the same using kafka streams or not?

apache-kafka-streams apache-kafka redis java
2 Answers

An equivalent to Rich*Function would be to use a transform() instead of map() that allows you to init() and close() a Transformer.

Your approach should work, too, even if you might want to try-catch to ensure close() is executed. However, it's not a recommended pattern.

Depending on your use case, it might be better to load the data from Redis into a Kafka topic (not sure if there is an Redis connector) and load the data into a KTable. Instead of a map() or transform() you would do a stream-table join.

user_49530790

Redis is not recommended for production with Spring Cloud Stream - the binder is not fully functional and message loss is possible.

user_55246040
Related