I am using structured streaming with subscribepattern with a checkpoint location. If I just delete a topic, the stream updates metadata and everything looks fine. But if topic is deleted while data is published to that topic and stream is running microbatch, I start getting this issue. I believe its because, the checkpoint->offset is updated when new batch arrives and before it could process data and update checkpoint->commit the topic is deleted. The stream is trying to fetch data from topic at checkpoint->offset and since topic is already deleted its not able to and timing out. Even if I restart the streaming query, the query looks at checkpoint->offset and starts timing out.
I can workaround this manually by either deleting the checkpoint, creating same topic again, or updating checkpoint->offset and delete offsets for topic that was deleted.
Is there a better way to handle this without any manual workarounds?