Service worker offline support with pushstate and client side routing

I'm using a service worker to introduce offline functionality for my single page web app. It's pretty straightforward - use the network when available, or try and fetch from the cache if not:

service-worker.js:

self.addEventListener("fetch", event => {
    if(event.request.method !== "GET") {
        return;
    }
    event.respondWith(
        fetch(event.request)
            .then(networkResponse => {
                var responseClone = networkResponse.clone();
                if (networkResponse.status == 200) {
                    caches.open("mycache").then(cache => cache.put(event.request, responseClone));
                }
            return networkResponse;
        })
        .catch(_ => {
            return caches.match(event.request);
        })
    )
})

So it intercepts all GET requests and caches them for future use, including the initial page load.

Switching to "offline" in DevTools and refreshing at the root of the application works as expected.

However, my app uses HTML5 pushstate and a client side router. The user could navigate to a new route, then go offline, then hit refresh, and will get a "no internet" message, because the service worker was never told about this new URL.

I can't think of a way around it. As with most SPAs, my server is configured to serve the index.html for a number of catch-all URLs. I need some sort of similar behaviour for the service worker.