I have a central system that publishes new records to a message bus topic.
Multiple agents subscribe to these messages and create new records in their respective systems using REST APIs.
These downstream systems cannot accommodate my central system's record Ids.
So I need to link records across all systems using a central record linkage repository e.g.
| Central System Id | System A Id | System B Id |
|---|---|---|
| 1 | 3231 | 767 |
| 2 | 3232 | 768 |
When each agent creates a new record, there is an opportunity to grab the new downstream system Id in the HTTP response message and use it to populate the above respository.
But the agents have one chance to take note of this Id and either update the central record linkage repository directly or place the Id on a message bus.
If there is a system failure before the agent can persist the Id, there is no way getting the Id back from the downstream system without a human needing to perform record matching.
For these lost records, an agent cannot consult the central record linkage repository to determine whether the record already exists, and therefore creates duplicate records in downstream system.
How can I implement a reliable record linkage strategy?
Alternatively, I could look towards implementing idempotent consumers but the attibutes used for matching existing records could change between source and target systems.