Handle duplicates in Azure Function with EventHub trigger

user_888548 0
Viewed 73

I have an Azure Function which has Event Hub binding trigger and it receives batch of EventData. I also have FixedDelayRetry for enabling retry for unhandled exceptions (done on purpose).

If one event fails during processing it inside a batch then the whole batch is getting retry again which obviously create an issue of duplicates

So how can I identify the events which have been successfully processed?

Is there a good pattern to store event sequence number and the status to check it before processing it? and how to retrieve those information as Azure functions are stateless.

azure-eventhub azure-functions c#
1 Answers

If one event fails during processing it inside a batch then the whole batch is getting retry again which obviously create an issue of duplicates

According to Microsoft Documents, partial success is not possible.

Either all Succeed or all Fail. More about EventDataBatch Class.

Event hubs is designed for millions of events per second, it's not designed for single message processing.

Its work on batches, they work on whole batch processed and if batch fail then retry the specific batch again.

So how can I identify the events which have been successfully processed?

You can use try and catch block, if catch block throws an exception then add this event in Queue and call another function with Queue trigger to process that failed events.

Is there a good pattern to store event sequence number

If you need to process multiple events in an order so you need to introduce a Partition key, which will allow you for ordered processing.

user_189357750
Related