tomaszdudek7 / airflow_project Goto Github PK

I tried today running: docker-compose up and I got the following:

Pulling webserver (puckel-airflow-with-docker-inside:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

It seems that the image got removed. Any ideas how can we work around it?

Just a quick question. If it's in a container, why does it need to have a virtual environment set up for it?

Hello, I am looking to set notebooks on docker volumes so when i modify it, I do not have to restart the images with the builder when making test. I have looked the build_images.py code but i don't find somewhere to configure this. Any possible solution?

My use case is this: I have a directory (/dir/my-data) in my computer, which contains input_csv_files and to which a task should save the output_csv_files.

I have added it to docker-compose, under the worker service:
volumes:
- /dir/my-data:/dir/my-data

Now, I want to bind it to the task container when it runs.

I am on master, running in RHEL 7, with docker-py 3.7.2, Docker version 18.03.1-ce, build 9ee9f40.

When I run python build_images.py, I get:

2019-05-08 13:50:27,272 - build_images - INFO - No particular task name specified. Will build every image in /docker/.
2019-05-08 13:50:27,272 - build_images - INFO - Browsing ['task3', 'task1', 'task2']
2019-05-08 13:50:27,272 - build_images - INFO - Handling task3
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/papermill_runner to ./docker/task3/papermill_runner
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/result_saver to ./docker/task3/result_saver
2019-05-08 13:50:27,273 - build_images - INFO - Building image. (run script with -l to see docker logs)
2019-05-08 13:56:56,834 - build_images - INFO - Cleaning up.
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/papermill_runner
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/result_saver
Traceback (most recent call last):
File "build_images.py", line 105, in
ImagesBuilder(parser).build_images()
File "build_images.py", line 33, in build_images
self.build_task(directory)
File "build_images.py", line 40, in build_task
image, build_logs = self.cli.images.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)
File "/opt/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 287, in build
raise BuildError(chunk['error'], result_stream)
docker.errors.BuildError: The command '/bin/sh -c pip install virtualenv' returned a non-zero code: 1

First, this is a pretty slick way of running airflow! I really like it and am going to be using it!

As I digest it all, I noticed that every time the DAG runs it creates new Docker containers, looks like 4 or 5 containers. Am I correct? and if so is this by design? Seems like things could get out of hand quick with the number of orphaned containers. Might we include a docker container prune like command at the end of the DAG run?

I don't know like i said I just got this and am starting to tear it apart to better understand things so I can put it to work.

Thanks

I am running python version 3.6.8 in an isolated conda environment.

When I run python build_images.py I get

File "build_images.py", line 33
self.log.info(f"Browsing {directories}")

SyntaxError: invalid syntax