How to deploy multiple python scripts together as multiple aws lambda functions?

Question

AWS beginner here

I have a repository on gitlab which has a branch named automatic_invoice_generator. This branch has the following content in it:

Script1.py
Script2.py
Script3.py
.gitlab-ci.yml

Now, I have to deploy these three codes as three different aws lambda functions. Right now, what I have done is create 3 different branches from automatic_invoice_generator branch, script1_branch, script2_branch, script3_branch, and for each branch (I changed the .gitlab-ci.yml file a bit to suit for the particular script).

My .gitlab-ci.yml file for Script1.py looks as follows:

image: ubuntu:latest

variables:
    GIT_SUBMODULE_STRATEGY: recursive
    LAMBDA_NAME: Script1
    AWS_DEFAULT_REGION: eu-central-1
    S3_BUCKET: invoice-bucket

stages:
  - deploy

production:
  stage: deploy
  script:
    - apt-get -y update
    - apt-get -y install python3-pip python3.7 zip
    - python3.7 -m pip install --upgrade pip
    - python3.7 -V
    - pip3.7 install virtualenv
    - mv Script1.py ~
    - mv csv_data~
    - mv requirements.txt ~

    # Move submodules
    - mv edilite/edilite ~
    - mv edilite/pydifact/pydifact ~

    # Setup virtual environment
    - mkdir ~/forlambda
    - cd ~/forlambda
    - virtualenv -p python3 venv
    - source venv/bin/activate

    - pip3.7 install -r ~/requirements.txt -t ~/forlambda/venv/lib/python3.7/site-packages/

    # Package environment and dependencies
    - cd ~/forlambda/venv/lib/python3.7/site-packages/
    - zip -r9 ~/forlambda/archive.zip .
    - cd ~
    - zip -g ~/forlambda/archive.zip Script1.py
    - zip -r ~/forlambda/archive.zip csv_data/*
    - zip -r ~/forlambda/archive.zip edilite/*
    - zip -r ~/forlambda/archive.zip pydifact/*

    # Upload package to S3
    # Install AWS CLI
    - pip install awscli --upgrade # --user
    - export PATH=$PATH:~/.local/bin  # Add to PATH

    # Configure AWS connection
    - aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    - aws configure set default.region $AWS_DEFAULT_REGION
    - aws sts get-caller-identity --output text --query 'Account'  # current account
    - aws s3 cp ~/forlambda/archive.zip s3://$S3_BUCKET/$LAMBDA_NAME-deployment.zip

I am using the same .gitlab-ci.yml file for all the branches (script1_branch, script2_branch, script3_branch), only changing the LAMBDA_NAME and name of the .py scripts. When I run the .gitlab-ci.yml files for all the 3 branches, the code runs and 3 different lambda functions are created and the code runs perfectly fine.

What I would like to know if there is a way I can modify my .gitlab-ci.yml file through which I can, instead of creating 3 different branches for 3 different scripts (script1_branch, script2_branch, script3_branch), create just one branch from the automatic_invoice_generator (say all_scripts_branch) and deploy all the 3 scripts simultaneously as three different lambda functions?

I am a bit new to both aws and gitlab, so any help is appreciated.

user_10266059 · Answer

Consider the following stub .gitlab-ci.yml which illustrates leveraging GitLab CI YAML anchors feature (https://docs.gitlab.com/ee/ci/yaml/#anchors) to reduce code duplication:

image: alpine

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  AWS_DEFAULT_REGION: eu-central-1
  S3_BUCKET: invoice-bucket

stages:
  - deploy

.job_template: &job_definition  # Hidden key that defines an anchor named 'job_definition'
  stage: deploy
  script:
    - echo zip -g ~/forlambda/archive.zip ${LAMBDA_NAME}.py
    - echo aws s3 cp ~/forlambda/archive.zip s3://$S3_BUCKET/${LAMBDA_NAME}-deployment.zip

production1:
  variables:
    LAMBDA_NAME: Script1
  <<: *job_definition           # Merge the contents of the 'job_definition' alias

production2:
  variables:
    LAMBDA_NAME: Script2
  <<: *job_definition           # Merge the contents of the 'job_definition' alias

References: - https://docs.gitlab.com/ee/ci/yaml/#anchors - https://gitlab.com/gitlab-org/gitlab-foss/issues/24535

How to deploy multiple python scripts together as multiple aws lambda functions?

1 Answers