Track back the exception in microservice architecture

A common approach is - Report all exceptions to a centralized exception tracking service that aggregates and tracks exceptions and notifies developers.

The benifit of this pattern is - it is easier to view exceptions and track their resolution.

The drawback of this pattern is - the exception tracking service is additional infrastructure.

In Careem, we use ELK stack - we use Logstash, which is a server-side data processing pipeline that ingests data from all micro-services simultaneously, transforms it and sends it to Elasticsearch. Kibana lets us visualize data with charts and graphs with functionality of wide range of filtering, searching etc.

Moreover, if the communication style of Microservice 1 --> Microservice 2 --> Microservice 3 synchronous, you can always generate and receive some customized error response in Microservice 1 from Microservice 3 through Microservice 2. But to get whole stack-trace of the exception, its better to aggregate the exception logs in some centralized place.

One of the properties of microservice architecture is to separate concerns, which means that in ideal world microservice 1 should not be aware of microservice 3 existence. It works with M2 and only thing it matters - whether response from it valid or not.

In any case, if you want to track calls, there are could be multiple approaches:

When M3 generates exceptions, it sends it back to M2, M2 propagates it to M1 as is (or wraps it without loosing information).

Another variant is to have separate storage for trace information, so M1 will generate unique ID, which is sent to M2, and M2 sends it to M3 to indicate that it is single request. Each service then uses this ID to store information about execution or any other metrics (by calling some X service).

First of all you need to assign a unique request id at or before the first service that starts processing.

How do you generate a unique request id for distributed tracing? Make it combination of instance id/name, and a timestamp.

If microservices are communicating synchnously then you can send it as an http response. But if the microservices are communicating asynchronously over steams like kafka, then you can use a stream to provide the callback mechanism. The stream can be of exceptions with request id becoming the partition key.

Microservice 1 --> Microservice 2 --> Microservice 3 --> Microservice 4 --> Microservice 1

Now suppose there is an exception in microservice 3 how could you track back this exception and tell the microservice 1 that this was the exception and it occurred from microservice 3.

There is an elephant in the room; why should Microservice1 care about what happens beyond Microservice 2 ? As long as Microservice 2 kept it's part of the contract (standard HTTP Error codes are also part of the contract) why should Microservice be even aware of the existence of Microservice 3. On the other hand, if you have a business requirement for Microservice 1 should be aware of Microservice 3's error, then you might need to revisit your system architecture.

Microservices are talking to each other over the network and usually it is using HTTP(s). So, at the boundaries of the microservices, the exceptions will get converted as standard HTTP error codes (for client error 4XX, for server errors 5XX and so on) and optional error message(s). When you invoke an upstream service, if the response is not a success (HTTP2XX), your consumer service just need to look for the agreed up on error codes / messages and translate it into meaningful actions(meaningful for the consumer service).

For debugging/Tracing purpose, if YOU want to know what happened to the request trail, that's another story. As other suggested, you can have centralized logging mechanism like ELK to push or pull logs out of your services and have a request Correlation UUID http header or so to correlate the http requests across different microservices.