504 Gateway Timeout (UnknownError) when trying to track changes in a user's calendar

For our internal application we synchronize our user's calendars (in our Office 365 tenant) with a local "cache" in our database. We're using the new Delta Queries in Microsoft Graph to do track these changes.

Most of the calendars synchronize correctly, but for some reason there is one single calendar where we consistently hit a 504 Gateway Timeout error when attempting to request the events using the nextLink received from the first request.

First request:

GET https://graph.microsoft.com/v1.0/users/<userId>/calendars/<calendarId>/calendarView/delta?startDateTime=2017-06-10t00%3A00%3A00Z&endDateTime=2018-06-10t00%3A00%3A00Z

First response:

{
    "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(event)",
    "@odata.nextLink": "https://graph.microsoft.com/v1.0/users/<userId>/calendars/<calendarId>/calendarView/delta?$skiptoken=R0usmcdvmMuZCBYV0hguCGmIJqcU0n_6jVFWUlNKbXkBKYVlxLSMsISZI5sLLLJyLJF8hZIj0PURpAeP_XxydW_qbMUoFMTXjOpLa8Ta6rxMRA7Wv6IHYfjyLPcDzCbM_hKvTgq8BZaBeJv-a61mebF6X2wT4HqCAGL5lL4nLZabHk1nD9GbWJ0a4Qq0M41_GPYxEi5YNe9u1673SQ1Djw.F85xXB6GjtO7myCQCOgFvzp1G7mQB0BvuHQJyn0CICQ",
    "value": [
        <list of events>
    ]
}

Second request:

GET https://graph.microsoft.com/v1.0/users/<userId>/calendars/<calendarId>/calendarView/delta?$skiptoken=R0usmcdvmMuZCBYV0hguCGmIJqcU0n_6jVFWUlNKbXkBKYVlxLSMsISZI5sLLLJyLJF8hZIj0PURpAeP_XxydW_qbMUoFMTXjOpLa8Ta6rxMRA7Wv6IHYfjyLPcDzCbM_hKvTgq8BZaBeJv-a61mebF6X2wT4HqCAGL5lL4nLZabHk1nD9GbWJ0a4Qq0M41_GPYxEi5YNe9u1673SQ1Djw.F85xXB6GjtO7myCQCOgFvzp1G7mQB0BvuHQJyn0CICQ

Second response:

504 Gateway Timeout

{
    "error": {
        "code": "UnknownError",
        "message": "",
        "innerError": {
            "request-id": "0784cffb-cba7-424b-be1d-74b2bfef5da1",
            "date": "2017-07-10T09:11:33"
        }
    }
}

I've tried executing the script a few times over the last week, but the request consistently fails when using requesting the second page. Other calendars synchronize with no issues, so I don't really know how to debug these kind of issues. Is there anything we can do to resolve this issue?