Vercel/Next.js (seeminlgy) randomly returns 404s with {"notFound": True}

Question

Intro

Apologies for not being able to provide a reproducible example. Our team is not able to reproduce the bug reliably. We have been investigating the bug for almost a week now, but can't seem to make any headway. We just rolled out our next.js based headless Shopify store (i.e. use next.js for the frontend and Shopify for everything starting from the checkout).

This bug is the strangest thing I have seen with next.js so far and any pointers towards solving the problem are more than appreciated.

Note:

You can navigate to www.everdrop.ch/it and open console to see some some broken links. However, since this is production, we obviously try to fix them as soon as possible.

Problem:

Almost every time we deploy a new version we would get to see some seemingly random 404s in the console, for when next is trying to prefetch Links.

The 404's are always of the form https://domain/_next/data///slug.json where sometimes PATH is e.g. category-pages and sometimes it's empty.

Observation 1

When clicking one of the broken links in console (the .json, I would get a 404:

Navigating to the broken pages on client side will also give a 404

However, when curl -I -L I would get a 200

Observation 2

When checking the Output data in Vercel
everything works like a charm

Note that the URL is different though. It is the same deployment but at a different URL.

Observation 3

The affected Links are seemingly random. However, some seem to be more likely to be affected than others.

Observation 4

Navigating to the page and then refreshing or directly accesing the page does produce the properly rendered page. Suprisingly enough this also results (for most pages that is) in a disapearance of the initital error.

Observation 5

Rerunning the deployment on vercel oftentimes fixes the problem and many of the broken links will then randomly work. Sometimes this leads to other broken links though.

Background & Stack

We use Storyblok and Shopify as data providers to query during build time. Shopify for product data and Storyblok for page and content data. All affected pages so far have been pages where we pull data from Storyblok during build time (which are all pages other than search and product pages).

We use next i18next for multi-language localisation. We use ENV variables to control where data is coming from to build our different stores.

user_2013652 · Answer

Many people have reached out to me on LinkedIn asking how we ultimately solved the problem at hand.

I think, generally speaking, the problem occurs when at build-time, a page build fails (i.e. when you are running into API limits). This is especially problematic in combination with

fallback: true (https://nextjs.org/docs/api-reference/data-fetching/get-static-paths#fallback-true)

As I think, pages that were built but failed will not get updated later on.

Our Solution

For us, we were able to solve it with:

preventing errors at build-time (we implemented a cache, but your errors might be different)
setting revalidate param, so that even if pages fail, they will get rebuilt
fallback: blocking (https://nextjs.org/docs/api-reference/data-fetching/get-static-paths#fallback-blocking)
notFound: true (https://nextjs.org/docs/api-reference/data-fetching/get-static-props#notfound)