Firebase Hosting returning 500 internal error for Googlebot user-agent when using Google Chrome's "network conditions" tab?

user_10128619 0
Viewed 427

I've got the following set up on my Firebase web app (it's a Single Page App built with React):

  • I'm doing SSR for robots user agents, so they get full rendered HTML and no Javascript
  • Users get the empty HTML and get the Javascript to run the app.

firebase.json

"rewrites": [{
  "source": "/**",
  "function": "ssrApp"
}]

Basically every request should go into my ssrApp function, that will detect robot crawlers user-agents and decide wheter it will respond with the SSR version for the robots, or the JS version for the regular users.

It is working as intended. Google is indexing my pages, and I always log some info about the user agents from my ssrApp function. For example, when I'm sharing an URL on Whatsapp, I can see Whatsapp crawler on my logs from Firebase Console (see below):

enter image description here

But the weird thing is that I'm not being able to mimick Googlebot using Chrome's Network Conditions tab:

enter image description here

enter image description here

When I try to access my site by using Googlebot's user agent I get a 500 - Internal error

And my ssrApp functions isn't even triggered, since NOTHING is logged out from it.

Is this a Firebase Hosting built-in protection to avoid fake Googlebots? What could be happening?

NOTE: I'm trying to mimick Googlebot's user agent because I want to inspect the SSR version of my app in production. I know that there are other ways to do that (including some Google Search Console tools), but I thought that this would work.

googlebot firebase-hosting firebase user-agent
2 Answers

Could you check that your pages are still in the Google index? I have the exact same experience and 80% of my pages are now gone... When I look up a page in Google Search Console https://search.google.com/search-console it indicates there was an issue during the last crawl. When I "Test it Live" it spins and reports the error 500 as well and asks to "try again later"...

user_96046990

I had a related issue, so I hope this could help anyone who encounters what I did:

I got the 500 - Internal error response for paths that were routed to a Cloud Run container when serving a GoogeBot user agent. As it happens, if the Firebase Hosting CDN happened to have a cached response for the path, it would successfully serve the cached response, but if it didn't have a cached response, then the request would not reach the Cloud Run container, it would fail at the Firebase Hosting CDN with 500 - Internal error.

It turns out that the Firebase Hosting CDN secretly probes and respects any robots.txt that is served by the Cloud Run container itself (not the Hosting site). My Cloud Run container served a robots.txt which disallowed all access to bots.

Seemingly, when the Hosting CDN attempts to serve a request from a bot from a path that is routed to a Cloud Run container, it will first probe any /robots.txt that is accessible at the root of that container itself, and then refuse to send the request to the container if disallowed by the rules therein.

Removing/adjusting the robots.txt file on the Cloud Run container itself immediately resolved the issue for me.

user_15135570
Related