tomaszdudek7 / airflow_project Goto Github PK
View Code? Open in Web Editor NEWscaffold of Apache Airflow executing Docker containers
License: MIT License
scaffold of Apache Airflow executing Docker containers
License: MIT License
I tried today running: docker-compose up
and I got the following:
Pulling webserver (puckel-airflow-with-docker-inside:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.
It seems that the image got removed. Any ideas how can we work around it?
Just a quick question. If it's in a container, why does it need to have a virtual environment set up for it?
Hello, I am looking to set notebooks on docker volumes so when i modify it, I do not have to restart the images with the builder when making test. I have looked the build_images.py code but i don't find somewhere to configure this. Any possible solution?
My use case is this: I have a directory (/dir/my-data) in my computer, which contains input_csv_files and to which a task should save the output_csv_files.
I have added it to docker-compose, under the worker service:
volumes:
- /dir/my-data:/dir/my-data
Now, I want to bind it to the task container when it runs.
I am on master, running in RHEL 7, with docker-py 3.7.2, Docker version 18.03.1-ce, build 9ee9f40.
When I run python build_images.py, I get:
2019-05-08 13:50:27,272 - build_images - INFO - No particular task name specified. Will build every image in /docker/.
2019-05-08 13:50:27,272 - build_images - INFO - Browsing ['task3', 'task1', 'task2']
2019-05-08 13:50:27,272 - build_images - INFO - Handling task3
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/papermill_runner to ./docker/task3/papermill_runner
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/result_saver to ./docker/task3/result_saver
2019-05-08 13:50:27,273 - build_images - INFO - Building image. (run script with -l to see docker logs)
2019-05-08 13:56:56,834 - build_images - INFO - Cleaning up.
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/papermill_runner
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/result_saver
Traceback (most recent call last):
File "build_images.py", line 105, in
ImagesBuilder(parser).build_images()
File "build_images.py", line 33, in build_images
self.build_task(directory)
File "build_images.py", line 40, in build_task
image, build_logs = self.cli.images.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)
File "/opt/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 287, in build
raise BuildError(chunk['error'], result_stream)
docker.errors.BuildError: The command '/bin/sh -c pip install virtualenv' returned a non-zero code: 1
First, this is a pretty slick way of running airflow! I really like it and am going to be using it!
As I digest it all, I noticed that every time the DAG runs it creates new Docker containers, looks like 4 or 5 containers. Am I correct? and if so is this by design? Seems like things could get out of hand quick with the number of orphaned containers. Might we include a docker container prune
like command at the end of the DAG run?
I don't know like i said I just got this and am starting to tear it apart to better understand things so I can put it to work.
Thanks
I am running python version 3.6.8 in an isolated conda environment.
When I run python build_images.py
I get
File "build_images.py", line 33
self.log.info(f"Browsing {directories}")
SyntaxError: invalid syntax
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.