Azure eventhub : Offset vs Sequence number

user_5165110 0
Viewed 3085

I see this question being asked on a lot of forums but none of them are solving my confusion.

This documentation seems to suggest that both offset and sequence number are unique within a partition. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.eventdata?view=azure-dotnet

It is clearly understood that sequence number is an integer which increments sequentially : https://social.msdn.microsoft.com/Forums/azure/en-US/acc25820-a28a-4da4-95ce-4139aac9bc44/sequence-number-vs-offset?forum=azureiothub#:~:text=SequenceNumber%20is%20the%20logical%20sequence,the%20Event%20Hub%20partition%20stream.&text=The%20sequence%20number%20can%20be,authority%20and%20not%20by%20clients.

But what of the offset ? Is it unique only within a partition , or across all partitions within a consumer group ? If it is the former condition, why have two different variables ?

azure-eventhub azure
2 Answers

An offset is a relative position within the partition's event stream. In the current Event Hubs implementation, it represents the number of bytes from the beginning of the partition to the first byte in a given event.

Within the context of a partition, the offset is unique. The same offset value may appear in other partitions - it should not be treated as globally unique across the Event Hub.

If it is the former condition, why have two different variables?

The offset is guaranteed only to uniquely identify an event within the partition. It is not safe to reason about the value or how it changes from event-to-event.

Sequence Number, on the other hand, follows a predictable pattern where numbering is contiguous and unique within the scope of a partition. Because of this, it is safe to use in calculations like "if I want to rewind by 5 events, I'll take the current sequence number and subtract 5."

user_1595480

Offset refers to consumer groups but not to partitions. Offset is a small storage container created for every consumer group, every consumer group has its own read offset, you can have several consumer groups and every group will read the event hub data at its own peace. In other words, the offset container holds a small blob with data regards the read checkpoint that will advance every time you will execute the context.CheckpointAsync(). If you will delete the container created by the consumer group, then the group will begin to read the data from the beginning,

List<EventProcessorHost> eventProcessorHosts = new List<EventProcessorHost>();
var eventProcessorHost = new EventProcessorHost(
    EventHubName,
    PartitionReceiver.DefaultConsumerGroupName,
    EventHubConnectionString,
    StorageConnectionString,
    StorageContainerName);
eventProcessorHosts.Add(eventProcessorHost);
eventProcessorHosts[0].RegisterEventProcessorAsync<SimpleEventProcessor>();
...
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
    foreach (var eventData in messages)
    {
        var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
                                                                                                             
        Console.WriteLine($"messages count: {messages.Count()} Message received. Partition: '{context.PartitionId}', Data: '{data}',  thread:{Thread.CurrentThread.ManagedThreadId}");
    }
    // Writes the current offset and sequenceNumber to the checkpoint store via the
    // checkpoint manager.
    return context.CheckpointAsync();
}

Check the storage container that was passed to EventProcessorHost constructor.

user_5023320
Related