We use Hyperledger fabric backed by CouchDB to store records with a lot of text. We want to build a solution that allows free text search on these records using Elastic Search. The problem then is getting the fabric/CouchDB data into elastic search in real time. What's the best way to do it. I can think of three approaches:-
1) Create a service that listens to the CouchDB changes feed in continuous mode(Couch has great support for this) and pushes the changes to a message queue(Kafka). Another service can then read from Kafka and push to Elastic Search. I like this idea('change data capture') because we could then build off other read views from our Kafka log. Cons of this approach:
- Though we are using hyperledger fabric, we are programming to the underlying data store(CouchDB). What happens if hyperledger moves to MongoDB after a year(or decides to support it along with CouchDB and we want to move to MongoDB ) ? We will have to rewrite/change our Kafka connectors.
- Maintain two states: 1) The offset of the last document read from CouchDB changes feed and 2)The Kafka offset. We can use Zookeeper for this ?
2) Have our client application(the one that writes to fabric) publish an event to a Kafka every time it writes/updates to fabric. Another service could then read those events and write to Elastic. The biggest con of this approach is:
- We enter the realm of dual writes(1 to fabric, 1 to Kafka) and there's no good way of doing both atomically. This leaves us susceptible to race conditions and partial failures(write to fabric succeeds but write to kafka fails etc etc).
3) I believe hyperlerdger fabric peers provide something called 'event services' which publish events every time it writes to the ledger. However, the docs for this one aren't great and I can't find any examples. Could you elaborate on this if you have experience with it ? My biggest problem with this approach is that hasn't been battle tested yet and a lot of developer time will go into fixing things when they go wrong because of the lack of docs/blogs/examples etc.
Thank you so much in advance for the help.