Google cloud build unable to fetch private go modules

I am having some troubles with private modules in a gcloud build. I generated a new ssh key-pair, stored the private key in my gcp projects secretManager and the public one in the github repos Deploy key . I followed this tutorial without problems: https://cloud.google.com/build/docs/access-private-github-repos

In a build step I am able to clone my repo (which I guess shows that I have a valid connection). But when I try to go get a private repo (private-repo) I get the following:

Step #2: go: github.com/felix/private-repo@<version-nr>: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/e61704280b96667bd155fe2e6c284d9506853b470d6786a1dff7011776527149: exit status 128:
Step #2:    ERROR: Repository not found.
Step #2:    fatal: Could not read from remote repository.
Step #2:    
Step #2:    Please make sure you have the correct access rights
Step #2:    and the repository exists.
Finished Step #2

My cloudbuild yaml file:

steps:
  - name: 'gcr.io/cloud-builders/git'
    secretEnv: ['SSH_KEY']
    entrypoint: 'bash'
    args:
    - -c
    - |
        echo "$$SSH_KEY" >> /root/.ssh/id_rsa
        chmod 400 /root/.ssh/id_rsa
        git config --global url."git@github.com:myOrg".insteadOf https://github.com/myOrg
    volumes:
    - name: 'ssh'
      path: /root/.ssh
    - name: 'git'
      path: /root/.gitconfig

  - name: 'golang'
    args: ['make', 'review']
    volumes:
    - name: 'ssh'
      path: /root/.ssh
    - name: 'git'
      path: /root/.gitconfig

availableSecrets:
  secretManager:
  - versionName: projects/my-project/secrets/ssh-private-key/versions/latest
    env: 'SSH_KEY'

I have also set these go env variables:

GOPRIVATE=github.com/myOrg/*
GO111MODULE=on

So it seems that the SSH part is working fine, but somehow I cant fetch my private dependencies.