soggycactus / airflow-repo-template Goto Github PK

Hi @soggycactus ,
thanks for providing this useful template!

One question: I'm just starting off and noticed Airflow 2.0.1 is already out - should I simply adjust AIRFLOW_VERSION in the Makefile to use it or are there other prerequisites to take care of?

Thanks!

Hello!

Lately, I've been trying to kickstart with my personal Airflow instance using your template, but had to upgrade it to 2.2.5. Successfully, I've succeeded doing so by modifying Dockerfile, that I'm sharing below. Maybe that will be useful when you'd decide to upgrade template

FROM apache/airflow:2.2.5

#Install Make pre-requisites
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends build-essential
USER airflow

#Install additional dependencies
COPY Makefile Makefile
COPY airflow.requirements.txt airflow.requirements.txt

RUN make internal-install-airflow
RUN make internal-install-deps

Personally, I've found a little bit difficult to understand code behind make internal-install-airflow component, so I'm putting additional Airflow modules into airflow.requirements.txt file, but I'm pretty sure that this will work too after providing appropriate airflow and python version in Makefile.

Hello!

Thank you for this template.

--

I try to run airflow not from the root user:

For this I change the dockerfile:

From

# Install Airflow and any additonal dependencies
WORKDIR /root

To

# Install Airflow and any additonal dependencies
WORKDIR /app

And then create user airflow and give him all the rights to the /app directory

After that I change all occurencies of /root in docker-compose file, so the volumes point to the /app dir now

--

I expect that when I boot up airflow I'd get my DAGs up and running, but instead it seems like DAGs have not been imported at all

Could you please point me in a way why this might happen? I did not find a way to specify DAGs loading folder in the code :(

Hello, I am currently attempting to run make start-airflow and am getting hit with the following error:

executor failed running [/bin/sh -c make internal-install-airflow]: exit code: 2
ERROR: Service 'initdb' failed to build
make: *** [start-db] Error 1

I've traced this back to the makefile attempting to pip install pyycopg2-binary, which then throws the following error:

#12 12.55     Error: pg_config executable not found.
#12 12.55     
#12 12.55     pg_config is required to build psycopg2 from source.  Please add the directory
#12 12.55     containing pg_config to the $PATH or specify the full executable path with the
#12 12.55     option:
#12 12.55     
#12 12.55         python setup.py build_ext --pg-config /path/to/pg_config build ...
#12 12.55     
#12 12.55     or with the pg_config option in 'setup.cfg'.
#12 12.55     
#12 12.55     If you prefer to avoid building psycopg2 from source, please install the PyPI
#12 12.55     'psycopg2-binary' package instead.
#12 12.55     
#12 12.55     For further information please check the 'doc/src/install.rst' file (also at
#12 12.55     <https://www.psycopg.org/docs/install.html>).
#12 12.55     
#12 12.55     ----------------------------------------

It's funny, since I thought the whole point to pip installing psycopg2-binary was to avoid having to deal with all of these executable files. When I attempt to install psycopg2-binary on its own in a fresh conda environment, the library installs just fine.

I ended up finding this post on the psycopg2 issues page, where someone said that adding libpq-dev to the list of dependencies in the dockerfile solves this issue. I tried it and now I am able to run make start-airflow without any errors. I'm not sure if this is a completely safe fix for this issue, I just wanted to point this issue out in case anyone else runs into it.

Thanks!