I want to consume several topics one at-a-time in sequences, within the same program. Each topic would be consumed for a few seconds, and then the program would pass to the next one.
In pseudo-python, what I want to do is something like this:
last_partition_offsets_per_topic = {}
while True:
for topic in topics:
(last_partition_offsets, message_batch) = get_last_messages_from_topic(topic, last_partition_offsets_per_topic.get(offset))
last_partition_offsets_per_topic[topic] = last_partition_offsets
process(topic, message_batch)
I would like to know if there are things I need to know about the impact on performance when consuming kafka with such a pattern.