My use case is that I make a function with a Kafka producer (Kafka Client) that receives data from a REST API endpoint then sends it to a Kafka cluster. The endpoint can have very high load.
My initial solution was as follows: In the class containing the method handling incoming REST API endpoint, I created a shared Kafka producer which is used by the Kafka producer function whenever the endpoint is called. This scenario is also mentioned in the second point of this thread.
Although the Kafka producer is thread safe, I'm not sure this single producer instance can handle a large number of inputs.
Therefore, I wanted to try another approach, suggested by Azure:
Event server also uses multiple sliding queues to control the number of outstanding requests from clients. New requests are queued to one of the multiple queues in an event server instance, which is then processed by multiple parallel Kafka producer threads. Each thread instantiates one producer. The number of sliding queues is controlled by thread pool size.
I'm not well-versed in concurrency programming. My question is how to create threads that each instantiate one Kafka producer. I used Scala.