Gitlab CI - SSH Permission denied (publickey,password)

user_829816 0
Viewed 24577

I've been trying to setup CD for my project. My Gitlab CI runner and my project will be on same server. I've followed https://docs.gitlab.com/ee/ci/examples/deployment/composer-npm-deploy.html but I keep getting SSH Permission denied (publickey,password). error. All my variables, private key and other variables set correctly in project settings.

I've created my ssh key with ssh-keygen -t rsa -C "my.email@example.com" -b 4096 command with no passphrase and set my PRODUCTION_PRIVATE_KEY variable with content of ~/.ssh/id_rsa file.

This is my gitlab-ci.yml:

stages:
  - deploy

deploy_production:
  stage: deploy
  image: tetraweb/php
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$PRODUCTION_PRIVATE_KEY")
    - mkdir -p ~/.ssh
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - apt-get install rsync
  script:
    - ssh $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER
    - hostname
  only:
    - master

And this is output from Gitlab CI runner:

Running with gitlab-ci-multi-runner 9.2.0 (adfc387)
  on ci-test (1eada8d0)
Using Docker executor with image tetraweb/php ...
Using docker image sha256:17692e06e6d33d8a421441bbe9adfda5b65c94831c6e64d7e69197e0b51833f8 for predefined container...
Pulling docker image tetraweb/php ...
Using docker image tetraweb/php ID=sha256:474f639dc349f36716fb98b193e6bae771f048cecc9320a270123ac2966b98c6 for build container...
Running on runner-1eada8d0-project-3287351-concurrent-0 via lamp-512mb-ams2-01...
Fetching changes...
HEAD is now at dfdb499 Update .gitlab-ci.yml
Checking out dfdb4992 as master...
Skipping Git submodules setup
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 12
$ ssh-add <(echo "$PRODUCTION_PRIVATE_KEY")
Identity added: /dev/fd/63 (rsa w/o comment)
$ mkdir -p ~/.ssh
$ echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ apt-get install rsync
Reading package lists...
Building dependency tree...
Reading state information...
rsync is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ ssh $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '{MY_SERVER_IP}' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
ERROR: Job failed: exit code 1

Thanks in advance.

gitlab-ci ssh
6 Answers

This is the script that worked to me:

before_script:
                - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
                - mkdir -p ~/.ssh
                - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
                - chmod 700 ~/.ssh/id_rsa
                - eval "$(ssh-agent -s)"
                - ssh-add ~/.ssh/id_rsa
                - ssh-keyscan -t rsa 64.227.1.160 > ~/.ssh/known_hosts
                - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
                - chmod 644 ~/.ssh/known_hosts

And I had to unprotect the variable as well.

Gitlab pipelines ssh key

user_25257010

The following can be used alternatively

some_stage:
        - eval $(ssh-agent -s)
        - cd ~
        - touch id.rsa
        - echo "$SSH_PRIVATE_KEY" > id.rsa
        - chmod 700 id.rsa
        - ssh -o StrictHostKeyChecking=no -i id.rsa $SSH_USER@$SERVER
user_129246530

Something important too... The permissions of the ~/.ssh/authorized_keys file should be 600.

user_39979900

It can also be due to restrictions on users you can ssh into.

In my case, on the server, I got the following tail -f /var/log/auth.log:

..
Sep  6 19:25:59 server-name sshd[7943]: User johndoe from WW.XX.YY.ZZ not allowed because none of user's groups are listed in AllowGroups
..

The solution consists in updating the AllowGroups directive on the server's file /etc/ssh/sshd_config:

AllowGroups janesmith johndoe
user_54467490

In our case, we were clueless until we add the flag -v to the SSH command (we knew the public key setup was OK because we were able to connect to this instance from our laptop using the private key).

We saw this :

debug1: Offering public key: ... RSA SHA256:... agent
95debug1: send_pubkey_test: no mutual signature algorithm

And understood the situation thanks to the two links below : our key was generated with RSA format which is considered legacy on up-to-date openssh versions.

So you have two solutions :

  • generate a new key using ed25519 format and setup the public key on your instance
  • use this extra flag below in your ssh command

It should be a temporary workaround :

ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o StrictHostKeyChecking=no your_user@your_instance_url "your command"

I hope it can help you if you are reading this.

Regards!

user_29201310
Related