Web App key restriction is not working for Firebase

I am using firebase. I want to restrict the API key generated by firebase so that only my domain can use that key. I tried setting up the restriction using the Credentials page of GCP by having HTTP referrers set as *.domain.com/*. Even after saving the changes, I am able to access firebase using localhost which should be not allowed according to restriction.