Long time waiting Request to Service Worker

I have noticed that the time waiting for service workers to respond with items from the cache is not as fast as you would expect it to be. I have seen the same wait times with both sw-precache and a custom written service worker.

What are the possible causes for this wait time time and how could I reduce it?

My fetch event on the custom service worker looks like:

self.addEventListener('fetch', function(event) {
    event.respondWith(
        caches.match(event.request).then(function(response) {
            if (response) {
                return response;
            }
            return fetch(event.request);
        })
    );
});