I am trying to find a solution for below scenario.
A Spark Streaming job reading data from Kafka (at a high velocity ~100 msg per second).
Somehow that job got failed and fixing of that job took more than 2 days.
Now when that spark job restarted, there is a huge backlog in the Kafka queue.
Each msg in the queue is important, I have to process the backlog as well.
Now, what approach should I apply to process and clear the backlog quickly along with dealing with the new data as well?
One obvious approach came in my mind is scaling up the capacity of the cluster (by adding more nodes) and once backlog is cleared scale down the cluster to normal.
Any other thoughts?
Thank you in advance.