Kafka will internally kill any nodes that are not correctly polling, but we don't want bad nodes to stay up and we want visibility of which nodes are working, so naturally we want to build out a classical healthcheck.
The obvious way to build a healthcheck is to use KafkaStream#state#isRunning but this is completely deceptive. If the underlying network connection to Kafka is down, or any of a dozen things that don't cause an internal thread to die, KafkaStreams still reports itself as alive even though the logs are throwing up errors all over the place. Is there any correct way to get visibility of what is happening inside Kafka streams? Is expecting a sane healthcheck the wrong approach to take when dealing with Kafka?
To be clear, I am not talking about healthchecks of the Kafka cluster, specifically the Streams processor node.