I'm trying to install Apache Airflow on an EC2 instance with the user-data script so it will automatically provision my environment. Airflow typically needs virtualenv to run python3; how do I accomplish this?
I have tried setting up all of the infrastructure needed prior to activating the virtualenv and all of that seems to install just fine. Where I'm running into trouble is installing airflow within the virtualenv. I'm not sure this is the correct approach, but it seems to be required when doing it interactively.
#!/bin/bash
# Path settings
export HOME=/home/ec2-user
export AIRFLOW_HOME=$HOME/venv/airflow
export PATH=$PATH:$AIRFLOW_HOME:$HOME/venv/bin
# Updates and install prereqs
sudo apt update -y
sudo apt install python-pip virtualenv python3-psycopg2 libpq-dev postgresql postgresql-client -y
pip install --upgrade pip
pip install virtualenvwrapper psycopg2 boto3
# Allow these ports through the ufw firewall
ufw allow 8080
ufw allow 5432
# Set up virtualenv and airflow
cd $HOME
virtualenv -p `which python3` venv
mkdir $AIRFLOW_HOME
mkdir $AIRFLOW_HOME/dags/
# Copy dags, run virtualenv, install airflow
pip install awscli --upgrade
cd $AIRFLOW_HOME/dags/
aws s3 cp s3://mydagbucket/dagbag $AIRFLOW_HOME/dags/ --recursive
unzip *.zip
rm *.zip
source $HOME/venv/bin/activate
# Airflow specific setup
pip install apache-airflow
airflow initdb
sed -i 's/sqlite:\/\/\/\/.*\/airflow.db.*/postgres:\/\/airflow-postgres.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432\/airflow?user=postgres\&password=<PASSWORD>/g' $AIRFLOW_HOME/airflow.cfg
psql postgres://airflow-postgres.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432/airflow?user=postgres\&password=pass1word -c "CREATE DATABASE airflow"
rm -f $AIRFLOW_HOME/airflow.db
airflow initdb
airflow webserver -p 8080 &
airflow scheduler &
chown ec2-user:ec2-user -R $HOME
sudo apt autoremove -y