very slow build of docker-airflow-spark:1.10.7_3.1.2 about airflow-spark

[riotgibbon]

so, I tried the airflow pip install in a clean container, and it worked. I've moved it to a separate RUN command before the big pip install one, and it seemed to be happy

the Java8 then failed, so I've swapped it to this:

RUN apt update -y && apt-get install -y software-properties-common && \
    apt-add-repository 'deb http://security.debian.org/debian-security stretch/updates main' && apt update -y && \
    apt-get install -y openjdk-8-jdk-headless && \
    pip install --no-cache-dir -r requirements.txt && \
    export JAVA_HOME && \
    apt-get clean

it's built without falling over, waiting for the docker-compose to finish pulling the containers now

[rafael-roano]

Hi @riotgibbon,

Same here! I'm having this issue as well. I was able to build images last week, without any problems. But a few days ago, this problem presented suddenly. The dockerfile is exactly the same, but for some reason the dependency resolver is now taking forever, and I haven't being able to complete an image since.

Thanks!

[riotgibbon]

10 hours later ...

[+] Building 47133.2s (6/16)                                                                                                                  
 => => extracting sha256:71c316554a558d4aa95c0be17c01e4a0366d6026a98eeba53dca5baa6091e815                                                0.2s 
 => => sha256:8973a3f2f69f7267196f90924c19cb72348d9b41f9c0a1bb9bc11180eaa3ac0c 14.21MB / 14.21MB                                        39.7s 
 => => sha256:1f96a121555fcc5208421bb785d99b8fb9a32bd0615e1e7854295600e7319873 241B / 241B                                              37.2s
 => => extracting sha256:721081de66bfc648ce19234c6333d6e031344f6ac90904476c9ec2dba2917e3a                                                2.5s
 => => sha256:19593799e97757b3b713045650eb23af9fc6cfa0303ddc51f0fc31cea074c3fc 2.21MB / 2.21MB                                          38.3s
 => => extracting sha256:239fb482263d7865a4a15a52e4a4be892393343c0c4318a930f770d6da32cba0                                                7.9s
 => => extracting sha256:ec2643d613a16bf8b8eb10a67654b8b4afa12ebc14be6deca2c6e8c9935a1197                                                0.3s
 => => extracting sha256:8973a3f2f69f7267196f90924c19cb72348d9b41f9c0a1bb9bc11180eaa3ac0c                                                0.6s
 => => extracting sha256:1f96a121555fcc5208421bb785d99b8fb9a32bd0615e1e7854295600e7319873                                                0.0s
 => => extracting sha256:19593799e97757b3b713045650eb23af9fc6cfa0303ddc51f0fc31cea074c3fc                                                0.2s
 => [ 2/12] COPY requirements.txt /requirements.txt                                                                                      1.3s 
 => [ 3/12] RUN set -ex     && buildDeps='         freetds-dev         libkrb5-dev         libsasl2-dev         libssl-dev           47043.5s 
 => => #   Downloading cffi-1.8.3.tar.gz (403 kB)                                                                                             
 => => #   Downloading cffi-1.8.2.tar.gz (403 kB)                                                                                             
 => => # Collecting cryptography>=0.9.3                                                                                                       
 => => #   Downloading cryptography-3.2-cp35-abi3-manylinux2010_x86_64.whl (2.6 MB)                                                           
 => => #   Downloading cryptography-3.1.1-cp35-abi3-manylinux2010_x86_64.whl (2.6 MB)                                                         
 => => #   Downloading cryptography-3.1-cp35-abi3-manylinux2010_x86_64.whl (2.6 MB)     

it was definitely OK on Monday, this started happening yesterday I think ...

is there a reliable version of this in DockerHub anywhere that anybody would recommend, I'm in a bit of a spot at the moment!

[riotgibbon]

I've tried adding the --use-deprecated=legacy-resolver tag in the Dockerfile pip commands:

&& pip install --use-deprecated=legacy-resolver -r requirements.txt \

doesn't seem to be helping

[riotgibbon]

I've tracked it down to this line:
&& pip install --use-feature=2020-resolver apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION}

[rafael-roano]

@riotgibbon Thanks for sharing! Another way (probably not the best approach, but more like a temp. solution) is to remove the "--use-feature=2020-resolver" flag from the airflow pip install. Not sure why this flag was working fine before.

[cordon-thiago]

Hi @riotgibbon and @rafael-roano
Thanks for contributing to this repo.
I tried to reproduce this behavior by rebuilding the images but it didn't take too long as you experienced.
Could you share which operating system are you using and which docker daemon version?

tks

[riotgibbon]

hi @cordon-thiago - it all just went away, after causing complete mayhem for a day ...

must have been a broken dependency somewhere down the line ...