chandnipateltw / basic-aws-infrastructure Goto Github PK

Currently Airflow deployment is not automated.

The script ./scripts/bootstrap_rds.sh $TRAINING_COHORT airflow is not running correctly because tries to instantiate with default subnet group, but it's not MultiAZ. Therefore, terraform training_airflow is not in included in the instructions for starting the environment.

This are the temporal instructions to deploy it manually:

1. Create a temporary new subnet group for RDS with the name: ${COHORT}-bootstrap-db-subnet-group. When creating make sure to add it at least two subnets in different availability zones

2. Run the following script to create bootstrap db:

./scripts/bootstrap_rds.sh $TRAINING_COHORT airflow

Note: The password is saved in Parameter store for future use.

3. Then you can run the terraform code:

./scripts/run_terraform.sh $TRAINING_COHORT training_airflow apply -input=false -auto-approve;

4. Once EC2 started we still have to install airflow manually, ssh into the machine and:

yum groupinstall “Development tools”

yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python-devel wget cyrus-sasl-devel.x86_64

yum -y install python-pip

pip install apache-airflow[postgres,s3]

5. Edit airflow.cfg and put this sql string:

postgresql://airflow:YOURPASSWORD@YOURRDSSERVER:5432/airflow?sslmode=require

6. Start airflow:

airflow initdb
airflow scheduler -D
airflow webserver -D