Kafka Connect - how to get a failed task to restart with a new configuration

user_14809439 0
Viewed 2438

Whenever we restart a failed task, it will ALWAYS pick up the config it had at the time of the failure, and run with that.. and THEN it picks up the new config.. and runs that as well.

We have connect jobs that we pause, update config, and then resume. This works fine, unless the task has failed.

If we restart a failed task, even if the connector has an updated config, the task will launch with the old config.. run to completion/failure.. then a new task will be launched with the new config.

This can cause various data/etc issues.. if you really don't want that old task to run with that config.

Any ideas how to restart a connector with a failed task.. with a new config.. and NOT have the old config get invoked?

(running Kafka v2.5, btw)

apache-kafka-connect apache-kafka
2 Answers

I don't know if it would make sense for the task to pick up the latest config. For instance, let's assume that your connector fires up 10 distinct tasks and 1 of them fails. It won't make sense to have the remaining 9 tasks of the connector running with the older config while the failed task runs the newest config once it is restarted.

I would say that in cases you want to use a new/different configuration file when a task fails, it might make more sense to restart the connector and not the individual task(s):

POST /connectors/connector-name/restart HTTP/1.1
user_71317570

I was having this problem and managed to "fix" this by a bit of randomness. I increased the number of Tasks in the connector and then reduced it again and it seemed to have picked up a new configuration. Was really random. I do know the restart did not work for me

user_56854820
Related