Kafka file streaming

user_508219 0
Viewed 2355

Consider the following scenario. I have a Kafka broker for shock absorption and a lot of clients, who send me user1.rar files, divided into files user1.r01, user1.r02... (128KB each). The consumption into one topic works quite good as expected. The problem is that I need to unpack the files downstream the topology and stream the unpacked result to the final storage.

The bad part is that I do not have a operational storage to store all rars for one user. Is there any way in kafka to streamline files in one topic, so I can do stream unpacking. I am quite afraid that if

  • I use one consumer, I will overwhelm its RAM, once 1000 of users starts streaming and the rXX files will get mixed up in the topic
  • If I use multiple consumers, I do not think that kafka has "smart routing", so I can read in one consumer only some keys (related to one user/group of users) + how I will do rebalancing + reset, when one of consumers dies...

Is there any pattern, how to deal with this situation?

Thanks!

apache-kafka-streams kafka-consumer-api apache-kafka
1 Answers

I'm rather new to this all but I've read into Kafka now a bit and if I understand the Kafka documentation correctly, you'll need to work with connectors/tasks (https://kafka.apache.org/documentation/#connect_overview - see chapter 8.3), and more specifically, SourceConnector/SourceTasks as you can define a max amount tasks for a connector ("tasks.max"), and together with the polling nature of the SourceConnector/SourceTasks, you can prevent your server being overloaded. After a task is handled it should be deleted, if I understand the documentation correctly.

I am really new to this all, but I hope this helps.

user_17204220
Related