Use authenticated urls for gcs only without setting "console.cloud.google.com" referer

I hope someone can help me with figuring out how to use authenticated URLs with gcs.

What I am trying to archive:

1. I want to send someone an authenticated gcs link (not pre-signed): e.g. https://storage.cloud.google.com/XXX/happy-simon.png
2. That person needs to log into their Google account
3. If they have the proper rights, they can download the file

Problem:

Whem I try to access an authenticated URL for GCS they do not work in the browser, but only when the Header "referer: https://console.cloud.google.com/" is set.

How to reproduce:

1. Upload a file to gcs
2. Give yourself at least read privileges on that file (you should be owner anyways as you uploaded it)
3. Try to download it via the "Authenticated Url"

3.1) When I click on the link in the google cloud console the download works without problems

3.2) When I copy the url and open it in a new tab I get a 403 error

Testing with curl on the console I found out that the difference between the two calls is wether the -H 'referer: https://console.cloud.google.com/' is set or not. If I set it, also the call from the new tab works.