I have a complex application flow with multiple stream-stream joins and stream-table joins. To simplify things let's say I am only using stream A and B. I tried migrating the "naive" way by dropping every stream and table and recreating them. However the problem seems to be that new data coming to stream B doesn't get joined with old data that stream A had. It only joins on new messages that have come after the recreation.
I investigated a bit more and discovered a property called 'auto.offset.reset'. As a second attempt I dropped everything again, then opened a new KSQL session and set 'auto.offset.reset' to 'earliest' and re-executed all the create statements. However the problem still seems to occur.
I have read the guides at: https://docs.confluent.io/platform/current/ksqldb/cloud-ksql-migration-guide.html and https://docs.ksqldb.io/en/latest/how-to-guides/update-a-running-persistent-query/ but they don't seem to go very in depth.
The ideal migration would be to recreate everything with new schemas and not have old messages get re-triggered or re-consumed. Also new messages should be able to join with previous messages. Any information about the best practises to migrate running KSQL applications is much appreciated.