How to transfer deployment package from S3 to EC2 instance to run python script?

AWS beginner here

I have a repo in GitLab which has a python script and a requirements.txt file, and the python script has to be deployed in the EC2 ubuntu instance (and the script has to be triggered only once a day) via Gitlab CI. I am creating a deployment package of the repo using CI and through this, I am deploying the zipped package in the S3 bucket. My .gitlab-ci.yml file:

image: ubuntu:18.04

variables:
    AWS_DEFAULT_REGION: eu-central-1
    GIT_SUBMODULE_STRATEGY: recursive
    S3_TEST_BUCKET: $BUCKET_UNPACK

stages:
    - deploy 

TestJob:
    stage: deploy
    script:
    - apt-get -y update
    - apt-get -y install python3-pip python3.7 zip
    - python3.7 -m pip install --upgrade pip
    - python3.7 -V
    - pip3.7 install virtualenv
    - mv iso_forest_ad.py ~ # This is the python script
    - mv requirements.txt ~

    # Setup virtual environment
    - mkdir ~/forEC2
    - cd ~/forEC2
    - virtualenv -p python3 venv
    - source venv/bin/activate
    - pip3.7 install -r ~/requirements.txt -t ~/forEC2/venv/lib/python3.7/site-packages/

    # Package environment and dependencies
    - cd ~/forEC2/venv/lib/python3.7/site-packages/
    - zip -r9 ~/forEC2/archive.zip .
    - cd ~
    - zip -g ~/forEC2/archive.zip iso_forest_ad.py

    - pip install awscli --upgrade
    - export PATH=$PATH:~/.local/bin
    - aws configure set aws_access_key_id $AWS_TEST_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $AWS_TEST_SECRET_ACCESS_KEY
    - aws configure set default.region $AWS_DEFAULT_REGION
    - aws s3 cp ~/forEC2/archive.zip $BUCKET_UNPACK/anomaly-detection-deployment.zip

Contents of requirements.txt

-i https://pypi.org/simple
joblib==0.16.0; python_version >= '3.6'
numpy==1.19.0
pandas==1.0.5
psycopg2-binary==2.8.5
python-dateutil==2.8.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
pytz==2020.1
scikit-learn==0.23.1
scipy==1.5.1; python_version >= '3.6'
six==1.15.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
sqlalchemy==1.3.18
threadpoolctl==2.1.0; python_version >= '3.5'

Now, I would like to transfer the script and install the dependencies in the ubuntu EC2 instance and run the script.

I know one way would be to connect to the EC2 instance and do

aws s3 sync s3://s3-bucket-name/folder /home/ubuntu

as suggested in the post: Moving files from s3 to EC2 instance. But doing this, I was not able to install the dependencies from the requirements.txt file.

I would like to know if there is an alternate way (perhaps maybe by using shell script or some other way?) for achieving this. Since I am using ubuntu locally too, using putty is not an option for me.

The link you've posted already shows one way of doing this. Namely, by using UserData.

Therefore, you would have to develop a bash script which would not only download the zip file as shown in the link, but also unpack it, and install the requirements.txt file along side with any other dependencies or configuration setup you require.

So the UserData for your instance would be something like this (pseudo-code, this is only a rough example):

#!/bin/bash 

apt update
apt install -y zip awscli python3-pip # awscli is not normally on ubuntu

aws s3 sync s3://optimal-aws-nz-play-config/package.zip .

unzip package.zip

cd package

pip install -r ./requirenements.txt

If this is something you do often, you could create lunch template with the instance settings and the UserData to automatically execute these steps for each instance launched from the template.

There are also other possibilities, involving CodeDeploy, CodePipeline, but plain old UserData would be a good start.

Alternative would be to use run-command. The execution of the command would be triggered from gitlab following upload of the new s3 package.

An example of how to invoke the run-command is in the docs:

aws ssm send-command \
    --document-name "AWS-RunPowerShellScript" \
    --parameters commands=["echo helloWorld"] \
    --targets Key=tag:Env,Values=Dev,Test

Instead of echo helloWorld you would have to write your own bash commands to be executed.

How to transfer deployment package from S3 to EC2 instance to run python script?

1 Answers