Load testing a kafka consumer

user_6753061 0
Viewed 438

I'm trying to figure out how to load test a kafka consumer.

In my application, the consumer reads message from kafka and does a lot of work most of it is writing stuff in a database. Since it's an important process for my team, I would like to be able to load test the consumer and be able to have some report as to how the consumption did. The end goal of this is that it generates the report in our CI and we would be able to see the evolution of the consumption for same load of message.

Sadly I really don't see how I can achieve such a thing. Would you have any idea as to how I would be able to do this ?

As of now, I'm thinking about duplicating the production topic on a dedicated environement and everytime I want to execute my load tests I would move the offset. This would not help me get a report on the consumption.

Thanks for reading me.

apache-kafka load-testing
1 Answers

Having a separate "load test" topic is a good idea. Depending on the topic retention policy (size/time) you can just delete the consumer offset of the application you want to test and have it start consuming from "earliest".

I don't know about your architecture, but I would highly recommend to continually monitor your application: Write proper metrics and keep an eye on them. CloudWatch would be the obvious choice when running on AWS. But there are lot of other services where you can publish metric data to (Grafana Cloud, New Relic etc.).

If you want to continually run load tests as part of your CI pipeline you should opt for a fixed input. For example use a fixed set of 100k messages you use for testing. Otherwise your results won't be deterministic and will be hard to compare. It's really helpful if your "core" data processing can be be run without depending on Kafka itself: Be input agnostic, messages may come from a file, a database or a Kafka topic.

user_30383920
Related