infopen / ansible-role-airflow Goto Github PK

View Code? Open in Web Editor NEW

41.0 41.0 16.0 122 KB

Ansible role to deploy and configure Airflow

License: MIT License

Python 100.00%

airflow ansible ansible-role infopen-template-0-28-0

ansible-role-airflow's People

Contributors

Stargazers

Watchers

Forkers

gr8routdoors yilab prashant-jain kissingwolf emurphy ycyang cermati rawiron decisionlab smowden intento coolpalani oggers geoxing victorboissiere

ansible-role-airflow's Issues

Installation on systemd is missing 'daemon-reload'

Running the playbook on a systemd (Centos 7) host results in the following error:

TASK [infOpen.airflow : SERVICE | Manage airflow services states] ********************************************************************
failed: [192.168.122.242] (item={u'state': u'started', u'enabled': True, u'name': u'airflow-webserver'}) => {"changed": false, "item": {"enabled": true, "name": "airflow-webserver", "state": "started"}, "msg": "Could not find the requested service airflow-webserver: host"}
failed: [192.168.122.242] (item={u'state': u'started', u'enabled': True, u'name': u'airflow-scheduler'}) => {"changed": false, "item": {"enabled": true, "name": "airflow-scheduler", "state": "started"}, "msg": "Could not find the requested service airflow-scheduler: host"}

Manually running systemctl daemon-reload on the host, and re-running the playbook/role results in successful completion.

Stuck in task "Manage configuration templating"

Hi,

while trying to use this role (latest master with airflow 1.10.11) I get stuck in the following task:

TASK [infOpen.airflow : CONFIG | Manage configuration templating] **********************
task path: /Users/pjs/.ansible/roles/infOpen.airflow/tasks/manage_configuration.yml:24

[...]

fatal: [airflow-server]: FAILED! => {
    "changed": false,
    "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'iteritems'"
}

Looks like a variable is not defined? Any idea which one?

SSL support?

I added some variables and other configuration to get SSL config via letsencrypt and https://github.com/geerlingguy/ansible-role-certbot going.

Facing some minor issues still but I can do a PR one they are resolved if this is of general interest to you.

Install outside of 'home' directory results in permission denied

If non-default install locations are used, where Airflow is installed outside of the airflow_user_home_path, then permission denied errors can result, as the paths are not created / checked correctly.

e.g. using the following install locations:

airflow_user_home_path: "/home/airflow"
airflow_base: "/opt/airflow"
airflow_virtualenv: "{{ airflow_base }}/venv"
airflow_defaults_config:
  core:
    airflow_home: "{{ airflow_base }}/app"

Results in the following error:

TASK [infOpen.airflow : INSTALL | Manage airflow installation] ***********************************************************************
failed: [192.168.122.242] (item={u'version': u'2.1.5', u'name': u'GitPython'}) => {"changed": false, "cmd": "/bin/virtualenv --no-site-packages -ppython2 /opt/airflow/venv", "item": {"name": "GitPython", "version": "2.1.5"}, "msg": "stdout: Running virtualenv with interpreter /bin/python2\n\n:stderr: Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/virtualenv.py\", line 2327, in <module>\n    main()\n  File \"/usr/lib/python2.7/site-packages/virtualenv.py\", line 712, in main\n    symlink=options.symlink)\n  File \"/usr/lib/python2.7/site-packages/virtualenv.py\", line 924, in create_environment\n    site_packages=site_packages, clear=clear, symlink=symlink))\n  File \"/usr/lib/python2.7/site-packages/virtualenv.py\", line 1109, in install_python\n    mkdir(lib_dir)\n  File \"/usr/lib/python2.7/site-packages/virtualenv.py\", line 323, in mkdir\n    os.makedirs(path)\n  File \"/usr/lib64/python2.7/os.py\", line 150, in makedirs\n    makedirs(head, mode)\n  File \"/usr/lib64/python2.7/os.py\", line 150, in makedirs\n    makedirs(head, mode)\n  File \"/usr/lib64/python2.7/os.py\", line 150, in makedirs\n    makedirs(head, mode)\n  File \"/usr/lib64/python2.7/os.py\", line 157, in makedirs\n    mkdir(name, mode)\nOSError: [Errno 13] Permission denied: '/opt/airflow'\n"}

Apply new version of role template

Apply Infopen Ansible role template 0.41.1 to this role

Problem with ansible-devel

With py27-ansible-devel test case, failed to template config file and broke the build.
Not occurs with 2.0 and 2.1, so I temporaly disabled the "-devel" build.

Take time to check why happens. I already test these case, without changes:

without "role_path"
with relative paths

< TASK [. : Manage configuration templating] >
 --------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

fatal: [f3472f9f03916825f6bd0d5cd5c8bc69a26890ff7e6f5f24aa24f3baeda72302]: FAILED! => {"changed": true, "failed": true, "msg": "Source /tmp/ansible-tmp-1466515761.43-60345451370661/source not found"}
    to retry, use: --limit @./testing_deployment.retry

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create separate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Adding 'airflow-kerberos' service needs new handler

Adding a service for airflow kerberos to airflow_services_systemd and airflow_services_states fails because a handler does not exist to start/restart the service.

Fix role builds

Last builds failed on Travis.

Check with updated pip and/or virtualenv packages to be able to process waiting PR

Unable to install python-virtualenv and python-pip

Hi,

I tried using your role and struck up on this step
TASK [deploy_airflow : INSTALL | Manage system dependencies] ******************************************************************************************
ok: [dmpServer] => (item={'name': 'python3-dev'})
ok: [dmpServer] => (item={'name': 'libpq-dev'})
ok: [dmpServer] => (item={'name': 'libssl-dev'})
ok: [dmpServer] => (item={'name': 'libffi-dev'})
ok: [dmpServer] => (item={'name': 'libxml2'})
ok: [dmpServer] => (item={'name': 'libxml2-dev'})
ok: [dmpServer] => (item={'name': 'libxslt1-dev'})
ok: [dmpServer] => (item={'name': 'build-essential'})
failed: [dmpServer] (item={'name': 'python-virtualenv'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "python-virtualenv"}, "msg": "No package matching 'python-virtualenv' is available"}
failed: [dmpServer] (item={'name': 'python-pip'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "python-pip"}, "msg": "No package matching 'python-pip' is available"}

How to proceed further.

Regards
Rajaniesh

infopen / ansible-role-airflow Goto Github PK

ansible-role-airflow's People

Contributors

Stargazers

Watchers

Forkers

ansible-role-airflow's Issues

Installation on systemd is missing 'daemon-reload'

Stuck in task "Manage configuration templating"

SSL support?

Install outside of 'home' directory results in permission denied

Apply new version of role template

Problem with ansible-devel

Initial Update

Adding 'airflow-kerberos' service needs new handler

Fix role builds

Unable to install python-virtualenv and python-pip

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent