How can I use Expect to automate an AWS job?

I have a very large copy to my local machine from an S3 bucket, which iterates through a file of identifiers and copies matching files. It all works well… except that the copy is so large that it keeps timing out and asking me for my password (every 2 - 3 hours). All told, it took a fortnight to run last time I ran it - but would have been much faster if I'd entered the password immediately that I was requested for it (which was never going to happen because of meals / sleep / other).

There are two scripts. One does the actual work (the core script) and one does the reading of the file of identifiers and calls the core script.

I'd like to automate this task, but I'm having trouble working out how to exactly.

This is the core of the working (but slow, because it asks for the password) script…

#!/bin/bash
function errorexit() {
  unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  exit 1
}
​
FILEID=$(echo $1)
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1 && export AWS_PROFILE=default
if [ $? -ne 0 ]; then
  exit 1
fi
​
ASSUMED_ROLE=$(aws sts assume-role --role-arn myroledeets --role-session-name seshname)
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey)
export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId)
export AWS_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken)
​
if [ ${#AWS_ACCESS_KEY_ID} -lt 16 ] || [ ${#AWS_ACCESS_KEY_ID} -gt 128 ]; then
  echo "Access Key Failure" >&2
  errorexit
fi
​
if [ ${#AWS_SECRET_ACCESS_KEY} -lt 16 ] || [ ${#AWS_SECRET_ACCESS_KEY} -gt 128 ]; then
  echo "Secret Access Key Failure" >&2
  errorexit
fi
​
if [ ${#AWS_SESSION_TOKEN} -lt 128 ]; then
  echo "Session Token Failure" >&2
  errorexit
fi
​
AWSPATH="s3://mypath/$FILEID"
aws s3 ls $AWSPATH | cut -c 32- | while IFS= read -r line; do
  COPYPATH="s3://mypath/$FILEID/$line"
  echo $COPYPATH
  aws s3 cp $COPYPATH ./$FILEID/$line --recursive
done
if [ $? -ne 0 ]; then
  errorexit
fi
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
exit 0

I thought it would be a good idea to use Expect to sort this issue. So I did the following (this doesn't work!):

#!/bin/bash
function errorexit() {
  unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  exit 1
}
​
ACCESSPW=$(echo $1)
FILEID=$(echo $2)
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
expect <<EOD
spawn aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1
set timeout 5
expect -r { 
"Password: $" {
send "$ACCESSPW\r"
}
}
EOD
​
if [ $? -ne 0 ]; then
  exit 1
fi
​
export AWS_PROFILE=default

ASSUMED_ROLE=$(aws sts assume-role --role-arn myroledeets --role-session-name seshname)
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey)
export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId)
export AWS_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken)
​
if [ ${#AWS_ACCESS_KEY_ID} -lt 16 ] || [ ${#AWS_ACCESS_KEY_ID} -gt 128 ]; then
  echo "Access Key Failure" >&2
  errorexit
fi
​
if [ ${#AWS_SECRET_ACCESS_KEY} -lt 16 ] || [ ${#AWS_SECRET_ACCESS_KEY} -gt 128 ]; then
  echo "Secret Access Key Failure" >&2
  errorexit
fi
​
if [ ${#AWS_SESSION_TOKEN} -lt 128 ]; then
  echo "Session Token Failure" >&2
  errorexit
fi
​
AWSPATH="s3://mypath/$FILEID"
aws s3 ls $AWSPATH | cut -c 32- | while IFS= read -r line; do
  COPYPATH="s3://mypath/$FILEID/$line"
  echo $COPYPATH
  aws s3 cp $COPYPATH ./$FILEID/$line --recursive
done
if [ $? -ne 0 ]; then
  errorexit
fi
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
exit 0

So what happens? Well, when I try to run this Expect automated version I get…

./awscopy.sh <password> <file>
spawn aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1
2022-02-14 11:09:29,372 [authenticator authenticator.py:authenticate] [43954-MainProcess] [6536770074-MainThread] - ERROR: Cannot extract saml assertion. Re-authentication needed?
Password:
An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired
Access Key Failure

As you may have guessed, this is the first time I've tried to use Expect. What have I done wrong?