How to generate signed URLs for Google Cloud Storage objects in GKE (Go)

For generating a signed URL, you need to have a private key.

When you are on GCP services (here on Compute Instances, the node of your K8S cluster, but it's the same thing with Cloud RUn, Cloud Functions et other GCP services) and you use the default credential (and there is no GOOGLE_APPLICATION_CREDENTIALS env var defined), the library use the metadata server.

The metadata server allows you to generate an access token

curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

or an identity token (with the audience in parameter)

curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://www.google.com

Thus, without having any secret or private key on your side, the libraries are able to generate a token (access or identity) for reaching external API.

However, the metadata server doesn't provide the secret (the private key) and you can't use it to generated signed URLs.

You need a service account key file here

You have several way to provide it to the pod in secure manner.

• The standard K8S way: use a secret volume
• The GCP managed solution: Secret manager. Thanks to the metadata server, you can instantiate the secret manager client and get your secret (your service account key file) and then use it as before. An alternative, I wrote a script that preload the secret into env variables when the container start, and thus, you just have to use the env var without interacting with the secret manager service

I don't recommend you to put your service account key file directly in the container, it's not really secure

Another solution

Ultimately, you can generate on the fly a key and defined it as the service account key (it's name user-defined service account key).

• You could generate it when you deploy your service on the cluster. Like this, you don't have to store a secret, it's generated on the fly each time.
• You could generate a key at container startup, set it in the service account, and keep it in memory.

Then, use it when you need it in your code and it should work because it's link to your service account.

However, you also need to think how to clean the old and useless keys.

I ran into the same problem and found a solution here: https://github.com/googleapis/google-cloud-go/issues/1130

TL;DR

Instead of passing in the private key to storage.SignedURL, you can give it a custom signing function. You can use Google's IAM SDK to sign a blob using your Service Account's credentials. That way you don't really need to know the private key at all.

A potential downside is that SignBlob() will perform an HTTP request to do the signing, whereas if you pass in a private key the signature will be computed locally.

Prerequisite: Give your Service Account the Service Account Token Creator role.

Your code would then look like this:

//import (
//    "cloud.google.com/go/storage"
//    credentialspb "google.golang.org/genproto/googleapis/iam/credentials/v1"
//    credentials "cloud.google.com/go/iam/credentials/apiv1"
//)

ctx := context.Background()
saEmail := "your-service-account-email@something-something.iam.gserviceaccount.com"

c, err := credentials.NewIamCredentialsClient(ctx)
if err != nil {
    panic(err)
}

url, err := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
    ContentType:    contentType,
    GoogleAccessID: saEmail,
    SignBytes: func(b []byte) ([]byte, error) {
        req := &credentialspb.SignBlobRequest{
            Payload: b,
            Name:    saEmail,
        }
        resp, err := c.SignBlob(ctx, req)
        if err != nil {
            panic(err)
        }
        return resp.SignedBlob, err
    }
})

As of version 1.18.0 you can do it like this:

    storageClient, _ := storage.NewClient(ctx)
    s, _ := storageClient.Bucket(bucketName).SignedURL(objectName, &storage.SignedURLOptions{
        Method:  http.MethodGet,
        Expires: expires,
    })

Quoted from here. Thanks @Fogia for pointing me to the PR that added it.