How to handle missing tasks due to non-deterministic failures in a micro service using gRPC?

Viewed 51

Suppose I have two micro services, A and B, where A makes a gRPC call to B with some data, and B performs a long running task and publishes an event with the result on completion that A and other services consume.

enter image description here enter image description here

If B is up when the call is made but, for some non-deterministic reason, crashes while processing the data and has to be respawned, how can A still get its result?
In something like Kafka this isn't a problem as messages are kept until they are deleted/the timeout passes, but without a message queue I can't really see any way for either A to trigger the process again, or for B to keep track of events it hasn't processed.

1 Answers

You can either switch communication pattern between A and B to be asynchronous one via some message broker (or introduce AB service which will handle request from A and turn them into incoming messages for B) or just save the incoming data to some persistent store (i.e. database/disk) before starting the long running processing and use this to track the status of events.

Related