How to determine cause of Azure EventGrid message delivery failure?

Question

How do I determine the cause of an EventGrid message delivery failure?

Tried so far:

Set a Diagnostic Setting on the EventGrid System Topic
Open Log Analytics and query the AEGDeliveryFailureLogs table with

//==================================================//
// Assign variables
//==================================================//
let varStart = ago(5h);
let varEnd = now();
//==================================================//
// Query table
//==================================================//
AegDeliveryFailureLogs
| where TimeGenerated between (varStart .. varEnd)
| project split(Message, ',')

Results in 1700+ records in my case
Sample result:

[
"outcome=Aborted",
" latencyInMs=180004",
" id=",
" outputEventSystemId=",
" state=Filtered",
" deliveryTime=5/26/2021 3:22:01 PM",
" deliveryCount=3",
" probationCount=1",
" deliverySchema=EventGridEvent",
" eventSubscriptionDeliverySchema=EventGridEvent",
" outputEventFields=InputEvent| EventSubscriptionId| DeliveryTime| DeliveryCount| State| Id| ProbationCount| LastDeliveryOutcome| DeliverySchema| LastDeliveryAttemptTime| SystemId| UseMappedResourceArmIdForBilling",
" outputEventFieldCount=12",
" requestExpiration=5/26/2021 3:23:01 PM",
" delivered=False id=",
" inputEventSystemId= publishTime=5/26/2021 3:14:42 PM",
" eventTime=5/26/2021 3:14:42 PM",
" eventType=Microsoft.Storage.BlobCreated",
" deliveryTime=1/1/0001 12:00:00 AM",
" filteringState=FilteringPending",
" inputSchema=EventGridEvent",
" publisher=MICROSOFT-STORAGE-STORAGEACCOUNTS..EVENTGRID.AZURE.NET",
" size=1162",
" subject=/blobServices/default/containers//blobs/2021-05-25//.csv",
" inputEventFields=Id| PublishTime| SerializedBody| EventType| Topic| Subject| FilteringHashCode| SystemId| Publisher| FilteringTopic| TopicCategory| DataVersion| MetadataVersion| InputSchema| EventTime| FilteringPolicy",
" inputEventFieldCount=16",
" type=AzureFunction",
" subType=NotApplicable",
" supportsBatching=False",
" aadIntegration=False",
" useSystemIdentity=False",
"armId=/subscriptions//resourceGroups//providers/Microsoft.Web/sites//functions/",
" deliveryResponse=",
" httpStatusCode=InternalServerError",
" errorType=UnexpectedError",
" errorMessage=An unexpected error has occurred. Please report the x-ms-request-id header value to our forums for assistance or raise a support ticket.",
" errorMessage=The operation timed out after: 00:03:00"
]

There is a clear outcome=Aborted along with an httpStatusCode/errorType/errorMessage that indicate an issue, but this is where the investigation stops.

What is causing these EventGrid message issues and how do I determine cause?

user_10426490 · Answer

It appears the only way to determine why an Event Grid message failed to be delivered is to turn on Dead Lettering.

Here is a sample Dead Letter message.

It clearly shows the deadLetterReason and some other valuable debugging information:

[
    {
        "id": "redacted",
        "eventTime": "2021-06-07T22:52:42.7156553Z",
        "eventType": "Microsoft.Storage.BlobCreated",
        "dataVersion": "",
        "metadataVersion": "1",
        "topic": "/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Storage/storageAccounts/redacted",
        "subject": "/blobServices/default/containers/redacted/blobs/20210525/redacted.csv",
        "deadLetterReason": "TimeToLiveExceeded",
        "deliveryAttempts": 12,
        "lastDeliveryOutcome": "TimedOut",
        "publishTime": "2021-06-07T22:52:43.1497044Z",
        "lastDeliveryAttemptTime": "2021-06-08T21:50:11.3677712Z",
        "data": {
            "api": "PutBlockList",
            "clientRequestId": "redacted",
            "requestId": "redacted",
            "eTag": "0x8D92A06F5883BA6",
            "contentType": "application/octet-stream",
            "contentLength": 4767672448,
            "blobType": "BlockBlob",
            "blobUrl": "https://redacted.blob.core.windows.net/redacted/20210525/redacted.csv",
            "url": "https://redacted.blob.core.windows.net/redacted/20210525/redacted.csv",
            "sequencer": "000000000000000000000000000093100000000000001286",
            "storageDiagnostics": {
                "batchId": "redacted"
            }
        }
    }
]

How to determine cause of Azure EventGrid message delivery failure?

1 Answers