Gcloud - cannot provision many VM using one Service Account

user_953258 0
Viewed 2180

I am using Gcloud to run Prow (Continous Integration server). One of my job creates a virtual machine, perform some tests and then delete that instance. I use a service account to create VM, run tests. 

#!/bin/bash

set -o errexit

cleanup() {
    gcloud compute instances delete kyma-integration-test-${RANDOM_ID}
}


gcloud config set project ...
gcloud auth activate-service-account --key-file ...

gcloud compute instances create <vm_name> \
    --metadata enable-oslogin=TRUE \
    --image debian-9-stretch-v20181009 \
    --image-project debian-cloud --machine-type n1-standard-4 --boot-disk-size 20 \

trap cleanup exit

gcloud compute scp --strict-host-key-checking=no --quiet <script.sh> <vm_name>:~/<script.sh>

gcloud compute ssh --quiet <vm_name> -- ./<script.sh>

After some time, I got following error:

ERROR: (gcloud.compute.scp) INVALID_ARGUMENT: Login profile size exceeds 32 KiB. Delete profile values to make additional space.

Indeed, for that service account, describe command returns a lot of data, for example ~70 entries in sshPublicKeys section.

gcloud auth activate-service-account --key-file ... gcloud compute os-login describe-profile

Most of this public keys refer to already removed VM instances. How to perform cleanup of this list? Or is it possible to not store that public keys at all?

google-iam gcloud google-cloud-platform cloud
4 Answers

The permanent solution is to use --ssh-key-expire-after 30s. You still need to cleanup the existing keys with the solutions above or a little more command kungfu like this (without grep).

for i in $(gcloud compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)"); do 
  echo $i; 
  gcloud compute os-login ssh-keys remove --key $i || true; 
done

NOTE: you have to be using the offending account. gcloud config account activate ACCOUNT and/or gcloud auth activate-service-account --key-file=FILE or gcloud auth login

Need a new ssh key in a script:

# KEYNAME should be something like $HOME/.ssh/google_compute_engine
ssh-keygen -t rsa -N "" -f "${KEYNAME}" -C "${USERNAME}" || true
chmod 400 ${KEYNAME}*

cat > ssh-keys <<EOF
${USERNAME}:$(cat ${KEYNAME}.pub)
EOF

Testing this solution:

while :; do
  USERNAME=testing@test-project.iam.gserviceaccount.com
  KEYNAME=~/.ssh/google_compute_engine
  rm -f ~/.ssh/google_compute_engine*
  ssh-keygen -t rsa -N "" -f "${KEYNAME}" -C "${USERNAME}" || true
  chmod 400 ${KEYNAME}*
  cat > ssh-keys <<EOF
  ${USERNAME}:$(cat ${KEYNAME}.pub)
EOF
  gcloud --project=test-project compute ssh --ssh-key-expire-after 30s one-bastion-to-rule-them-all -- date
  gcloud --project=test-project compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)" \
    |wc -l
done
user_74072490

A very crude way to do the above that worked for me was:

for i in $(gcloud compute os-login ssh-keys list); do echo $i; gcloud compute os-login ssh-keys remove --key $i; done

I stopped this (with Control-C) after deleting a few tens of keys and then it worked again.

Actually, in the project metadata in the GUI, I do not see a lot of key. Only :

  • gke...cidr : network-name...
  • sshKeys : gke-e9...
  • SSH Keys => peter_v : ssh-rsa my public key
user_26641340

These key are stored in your Project Metadata yo can remove them by deleting trough the Google Console UI

user_97434180

Seeing as you were mentioning OS Login in your question: there is a way to delete specific SSH keys from a user's profile using this command. Alternatively, instead of performing SCP, I'd advise you, much like John Hanley has, to put the file you're copying into the instance in Storage and retrieve it via a startup script (you could also use a custom Compute image).

user_9943080
Related