SSH capture exit code of remote ssh commands

Question

I have written a script, that is part of a gitlab ci pipeline, to deploy code via ssh to a remote server. The script is located locally and forwared to the remote server. That works okay so far, but unfortunately there is a problem when errors occur. Right now the migrate command exits, because there are no db credentials saved. The command exits with a non zero code, but the pipeline still succeeds, which is misleading, and forces me to login to the remote server and check the logs there.

How can I forward the exit codes from the remote to the local machine, so that in this case the gitlab-ci job also fails?

I've come across these Q/A's, but since I'm fairly new to bash I don't understand the solution. Removing set -e didn't help.

The gitlab-ci config part looks like this:

script:
# Change directory to the packaged app (not the repo).
- cd /var/www/html

# Copy over the code.
- |
  rsync \
    -azc \
    --exclude-from="$RSYNC_EXCLUDES" \
    "$(pwd)/" "$SSH_CONNECTION:$BASE_DIR/releases/$CI_COMMIT_SHA/"

# Run the post deployment script.
- ssh -T "$SSH_CONNECTION" "bash -s" < ./.deploy/post_deploy.sh "$BASE_DIR" "$CI_COMMIT_SHA"

The post_deploy.sh script looks like this:

#!/bin/bash

# Terminate execution if any command fails
set -e

########################
# Program arguments:
########################

## The base directory
BASE_DIR=$1

# The commit sha that is being deployed.
COMMIT_SHA=$2

################
# Variables:
################

# The path to the releases directory
RELEASE_DIR=$BASE_DIR/releases/$COMMIT_SHA

# The path to the shared directory.
SHARED_DIR=$BASE_DIR/shared

###############
# Deployment:
###############

echo "Symlink $SHARED_DIR/.env to $RELEASE_DIR/.env."
rm -rf "$RELEASE_DIR/.env" && ln -sf "$SHARED_DIR/.env" "$RELEASE_DIR/.env."

echo "Symlink $SHARED_DIR/storage to $RELEASE_DIR."
rm -rf "$RELEASE_DIR/storage" && ln -sf "$SHARED_DIR/storage" "$RELEASE_DIR"

echo "Fixing permissions."
find "$RELEASE_DIR" -type f -exec chmod 644 {} \;
find "$RELEASE_DIR" -type d -exec chmod 755 {} \;

echo "Running custom scripts."
php "$RELEASE_DIR" artisan storage:link
php "$RELEASE_DIR" artisan migrate --no-interaction --force

php "$RELEASE_DIR" artisan cache:clear
php "$RELEASE_DIR" artisan config:clear

php "$RELEASE_DIR" artisan config:cache
php "$RELEASE_DIR" artisan view:cache

echo "Releasing the new version (symlink current/)."
ln -nsf "$RELEASE_DIR/" "$BASE_DIR/current"

echo "Remove all releases older than a day."
find "$BASE_DIR/releases" -maxdepth 1 -mindepth 1 -mtime +1 -exec rm -r {} \;

user_5236387 · Answer

it should work if you really get non zero code. I tested on my machines

$ ssh user@host 'bash -s' < script.sh 
remote script execute with exit 1
$ echo $?
1

Gitlab must fail a job if at least one script line returns non zero.

As far as I understand you except you said about this line

php "$RELEASE_DIR" artisan migrate --no-interaction --force

I don't know about this tool, but often flags --force should make command without zero code. Please check the flags.

If that doesn't work, try putting

if [[ $? != 0 ]]; then
  exit 1
fi

after line with migrate

user_5779083 · Answer

Okay. It seems that this was a mistake on my end, and everything is working correctly. I wasn't calling one of the commands correctly:

php "$RELEASE_DIR" artisan migrate --no-interaction --force

This fixes it:

php "$RELEASE_DIR/artisan" migrate --no-interaction --force

SSH capture exit code of remote ssh commands

2 Answers