How to handle error spikes gracefully with Sentry?

user_511362 0
Viewed 1282

In our project we use Sentry to report errors. This works pretty fine, however on systems with a lot of load, we sometimes see spikes of errors when a third party system is down. For example, if a database is unreachable, we may get a few ten-thousand calls before the problem is solved and all of these result in error reports to Sentry.

I know that there is a general setting for the Sentry client of sample.rate. This allows to only send a certain percentage of all events to Sentry. However, I just want to avoid tons of duplicated error reports in case of a third party system failure. I don't want to limit the amount of errors being sent in general. Is there a way to configure the Sentry client so that in case of spikes it sends only a few samples but otherwise it sends all error reports?

sentry java
2 Answers
user_902970

The Java SDK only does random sampling at this time but Sentry itself has a spike protection as described in this blog post.

The docs also have more info.

Also, if they are 'the same error' they should be grouped together. If not automatically, you could do so using the event fingerprint. So at least you get only 1 notification.

You could also consider that if you are getting tens of thousands of errors, perhaps adding an exponential backoff strategy to try hitting the third party less often could help.

user_19771430
Related