Coder Social home page Coder Social logo

dci-ansible's Introduction

Ansible modules to interact with Distributed-CI (DCI)

A set of Ansible modules and callbacks to ease the interaction with the Distributed-CI platform.

Distributed-CI (DCI) is a platform that allows a company to extend its CI coverage beyond its own walls; allowing them to let third-party partners contribute back their CI results into the platform.

DCI comes as a webservice that exposes its resources via a REST API. The dci-ansible project aims to map each of those REST resources to an Ansible module.

Table of Contents

Get Started

Installation

There are two ways to install dci-ansible, either via rpm packages or directly via Github source code.

Unless you are looking to contribute to this project, we recommend you use the rpm packages.

Packages

Officially supported platforms:

  • CentOS (latest version)
  • RHEL (latest version)
  • Fedora (latest version)

If you're looking to install those modules on a different Operating System, please install it from source

First, you need to install the official Distributed-CI package repository

#> yum install https://packages.distributed-ci.io/dci-release.el7.noarch.rpm

Then, install the package

#> yum install dci-ansible

Sources

Define a folder where you would like to checkout the code and clone the repository.

#> cd /usr/share/dci && git clone https://github.com/redhat-cip/dci-ansible.git

How to use it

Authentication

The modules provided by this project covers all the endpoints Distributed-CI offers.

This means that this project allows one to interact with Distributed-CI for various use-cases:

  • To act as an agent: Scheduling jobs, uploading logs and tests results.
  • To act as a feeder: Creating Topics, Components and uploading Files.
  • To complete adminitrative tasks: Creating Teams, Users, RemoteCIs.
  • And more...

For any of the modules made available to work with the Distributed-CI API, one needs to authenticate itself first.

Each module relies on 3 environment variables to authenticate the author of the query:

  • DCI_CLIENT_ID: The ID of the resource that wish to authenticate (RemoteCI, User, Feeder)
  • DCI_API_SECRET: The API Secret of the resource that wish to authenticate
  • DCI_CS_URL: The API address (default to https://api.distributed-ci.io)

The recommended way is to retrieve the dcirc.sh file directly from the https://www.distributed-ci.io or create it yourself in the same folder as your playbook:

#> cat > dcirc.sh <<EOF
export DCI_CLIENT_ID=<resource_id>
export DCI_API_SECRET=<resource_api_secret>
export DCI_CS_URL=https://api.distributed-ci.io
EOF

And then run the playbook the following way:

#> source dcirc.sh && ansible-playbook playbook.yml

By now my dci-test/ folder looks like this:

#> ls -l dci-test/
total 837
-rw-rw-r--. 1 jdoe jdoe 614 Oct 19 14:51 dcirc.sh
-rw-rw-r--. 1 jdoe jdoe 223 Oct 19 14:51 playbook.yml

File organization

Since the modules of dci-ansible are not part of Ansible, one needs to tell Ansible where to look for the extra modules and callbacks this project is providing. This is done via the Ansible configuratin file.

The Distributed-CI team recommends that you place an ansible.cfg file in the same folder as your playbook with the following content:

[defaults]
library            = /usr/share/dci/modules/
module_utils       = /usr/share/dci/module_utils/
callback_whitelist = dci
callback_plugins   = /usr/share/dci/callback/

Note: If you installed the modules from source, please update the pathes accordingly.

By now my dci-test/ folder looks like this:

#> ls -l dci-test/
total 1014
-rw-rw-r--. 1 jdoe jdoe 177 Oct 19 14:51 ansible.cfg
-rw-rw-r--. 1 jdoe jdoe 614 Oct 19 14:51 dcirc.sh
-rw-rw-r--. 1 jdoe jdoe 223 Oct 19 14:51 playbook.yml

Samples

The following examples will highlight how to interact with a resource. The remoteci resource will be taken as an example. The same pattern applies to all Distributed-CI resources,

  • Create a RemoteCI
---
- hosts: localhost
  tasks:
    - name: Create a RemoteCI
      dci_remoteci:
        name: MyRemoteCI
  • List all RemoteCI
---
- hosts: localhost
  tasks:
    - name: List all RemoteCIs
      dci_remoteci:
  • Update a RemoteCI
---
- hosts: localhost
  tasks:
    - name: Update a RemoteCIs
      dci_remoteci:
        id: <remoteciid>
        name: NewName
  • Delete a RemoteCI
---
- hosts: localhost
  tasks:
    - name: Delete a RemoteCIs
      dci_remoteci:
        id: <remoteciid>
        state: absent

Real life scenarios examples are available in the samples/ directory.

Contributing

We'd love to get contributions from you!

If you'd like to report a bug or suggest new ideas you can do it here.

If you'd like to contribute code back to dci-ansible, our code is hosted on Software Factory and then mirrored on Github. Software Factory is Gerrit based, if you don't feel comfortable with the workflow or have any question, feel free to ping someone on IRC.

Running tests

Before you can run test you need to get familiarized with the dci-dev-env project.

dci-dev-env is a Docker based environment that will deploy a Distributed-CI Control Server API, the UI and more. Once deployed locally, you'll be able to run the test suite against this deployment.

To run the test, ensure the api is running by running docker ps and then simply run ./run_tests.sh in the tests/ folder

License

Apache License, Version 2.0 (see LICENSE file)

Contact

Email: Distributed-CI Team [email protected]

IRC: #distributed-ci on Freenode

dci-ansible's People

Contributors

spredzy avatar ylamgarchal avatar goneri avatar dsavineau avatar fredericlepied avatar rh-gvincent avatar fcharlier avatar p3ck avatar hguemar avatar prophidys avatar nsilla avatar tonyskapunk avatar thekad avatar sf-project-io avatar pierreblanc avatar

Stargazers

Adam Huffman avatar Pavan Chavva avatar David Newswanger avatar Tom Hensel avatar  avatar  avatar

Watchers

Lucian avatar  avatar James Cloos avatar Guillaume Vincent avatar  avatar  avatar Matthieu Huin avatar  avatar  avatar  avatar Tatiana Krishtop avatar  avatar

Forkers

goneri

dci-ansible's Issues

dci_component hangs

I had the case where dci_component has been frozen for hours. I'm not sure what happened thus.

dci_job doesn't work

Hello,
dci_job doesn't work for me

---
- name: dci new job
  hosts: localhost
  tasks:
    - name: schedule a new job
      dci_job:
        topic: 'Fedora-Atomic-26'
      register: job_informations

topic exists

$ dcictl topic-show 2c3626a4-8a51-45b5-9e4b-b3b48c5e0536
id
2c3626a4-8a51-45b5-9e4b-b3b48c5e0536

name 
Fedora-Atomic-26

component_types
['qcow2']

component for topic exists

dcictl component-list --topic-id 2c3626a4-8a51-45b5-9e4b-b3b48c5e0536
id
e651818b-96a4-4d9f-ade5-984c51f2e61e

name 
Fedora-Atomic-26-20170821.0

But my ansible playbook crash:

TASK [schedule a new job] *****************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible__DXauH/ansible_module_dci_job.py\", line 275, in <module>\n    main()\n  File \"/tmp/ansible__DXauH/ansible_module_dci_job.py\", line 252, in main\n    res = dci_job.get_full_data(ctx, ctx.last_job_id)\n  File \"/usr/lib/python2.7/site-packages/dciclient/v1/api/job.py\", line 75, in get_full_data\n    job = base.get(context, RESOURCE, id=id, embed=embed).json()['job']\n  File \"/usr/lib/python2.7/site-packages/requests/models.py\", line 892, in json\n    return complexjson.loads(self.text, **kwargs)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 516, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 374, in decode\n    obj, end = self.raw_decode(s)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 404, in raw_decode\n    return self.scan_once(s, idx=_w(s, idx).end())\nsimplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
 [WARNING]: Failure using method (v2_runner_on_failed) in callback plugin (</usr/share/dci/callback/dci.CallbackModule object at 0x302f7d0>): 'jobstate'

Unauthorized resource access

When you try to access to a resource which is not authorized for your team, you get a stack trace because the http return code 412 is not handled correctly.

This is an exemple with dci_topic:

        - name: Retrieve information about the next topic
          dci_topic:
            id: '{{ job_informations.job.topic.next_topic }}'

If you have access to the current topic but not the next topic :

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_KVNLtV/ansible_module_dci_topic.py", line 187, in <module>
    main()
  File "/tmp/ansible_KVNLtV/ansible_module_dci_topic.py", line 181, in main
    result = parse_http_response(http_response, dci_topic, context, module)
  File "/tmp/ansible_KVNLtV/ansible_modlib.zip/ansible/module_utils/common.py", line 172, in parse_http_response
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decode

dci_file returns a UnicodeDecodeError exception

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)
failed: [localhost] (item={u'path': u'/var/lib/dci-ansible-agent/.quickstart/collected_files/undercloud.tar.gz', u'mime': u'application/x-compressed', u'name': u'undercloud.tar.gz'}) => {"failed": true, "item": {"mime": "application/x-compressed", "name": "undercloud.tar.gz", "path": "/var/lib/dci-ansible-agent/.quickstart/collected_files/undercloud.tar.gz"}, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_d19GWY/ansible_module_dci_file.py\", line 217, in <module>\n    main()\n  File \"/tmp/ansible_d19GWY/ansible_module_dci_file.py\", line 199, in main\n    res = dci_file.create(ctx, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/dciclient/v1/api/file.py\", line 36, in create\n    return context.session.post(uri, headers=headers, data=content)\n  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 522, in post\n    return self.request('POST', url, data=data, json=json, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 461, in request\n    prep = self.prepare_request(req)\n  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 394, in prepare_request\n    hooks=merge_hooks(request.hooks, self.hooks),\n  File \"/usr/lib/python2.7/site-packages/requests/models.py\", line 298, in prepare\n    self.prepare_auth(auth, url)\n  File \"/usr/lib/python2.7/site-packages/requests/models.py\", line 500, in prepare_auth\n    r = auth(self)\n  File \"/usr/lib/python2.7/site-packages/dciclient/v1/api/context.py\", line 98, in __call__\n    payload=r.body)\n  File \"/usr/lib/python2.7/site-packages/dciclient/v1/auth.py\", line 55, in sign\n    payload_hash = hashlib.sha256(payload.encode('utf-8')).hexdigest()\nUnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

Cannot update job status with dci_job

I tried to create a job state for job id xxxxxx

I tried:

- name: clean environment
  hosts: localhost
  tasks:
    - include: hooks/clean.yml
      vars:
        dci_status: 'pre-run'

and

- dci_job:
    id: "xxxxxx"
    status: 'success'

None of the two solutions works

Testing only in Localhost can create problem

Greetings,
I am an engineer who is interested in infrastructure as code testing. Currently, I am looking for testing anti-patterns in iac test scripts. I noticed Local-only Testing occurring in test instances. This can give happy test results in the local environment but in a real production system test can fail due to the difference in the environment. My recommendation is to test IaC code in an isolated non-local environment which needs to be similar to the production environment.

So I have the following queries:

Do you agree that this is an IaC testing anti-pattern?
Do you want to fix this?

Any feedback is appreciated.

Source Files:
https://github.com/redhat-cip/dci-ansible/blob/master/tests/scenario-tests/openstack/feeder.yml

dci_component doesn't handle http error 500

Uploading a component file returning a http error 500 from the API doesn't raise an exception/failure from the ansible module.

The code:

    - name: Upload the component content
      dci_component:
        id: '{{ item.component.id }}'
        path: "{{ temp_dir.path }}/{{ item.component.canonical_project_name }}.tar"
      with_items: "{{ components.results }}"

The result:

TASK [Upload the component content] ********************************************************************************************************************************************************************************************************************************************
task path: /home/centos/dci-ansible/tests/scenario-tests/openstack/feeder.yml:47
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, 'failed': False, u'changed': False, u'component': {u'name': u'RH7-RHOS-OSP12-2018-01-13', u'title': None, u'canonical_project_name': u'RH7-RHOS-OSP12.0-2018-01-13', u'data': {}, u'created_at': u'2018-11-22T19:54:32.128041', u'updated_at': u'2018-11-22T19:54:32.128041', u'id': u'0f2d3031-cb4a-4cdc-8117-5bc93ceb33e0', u'url': u'https://www.example.com', u'state': u'active', u'etag': u'6ff9f1426799f7daa781e3631b39879b', u'topic_id': u'fbd34dac-549e-4141-b380-fd1307ddf098', u'message': None, u'type': u'puddles', u'export_control': True}, 'item': {u'next_topic_id': u'f3f8d341-de64-492e-b349-700a978b2588', u'product_id': u'322c9266-1ca7-4086-b481-ecf15fe48a63', u'created_at': u'2018-11-22T19:53:20.504518', u'updated_at': u'2018-11-22T19:53:20.504518', u'label': None, u'state': u'active', u'etag': u'7fd8882ffc02558c93fdb91a12e8d8b1', u'id': u'fbd34dac-549e-4141-b380-fd1307ddf098', u'component_types': [u'puddles'], u'data': {u'releasename': u'pike', u'registry': {u'login': None, u'password': None}}, u'export_control': False, u'name': u'OSP12'}, u'invocation': {u'module_args': {u'canonical_project_name': u'RH7-RHOS-OSP12.0-2018-01-13', u'dci_client_id': u'feeder/7ad59c5d-95e4-4005-8204-4901a0da6231', u'path': None, u'active': True, u'dci_cs_url': u'http://192.168.100.14', u'data': None, u'id': None, u'dci_login': None, u'name': u'RH7-RHOS-OSP12-2018-01-13', u'url': u'https://www.example.com', u'dci_password': None, u'dest': None, u'state': u'present', u'dci_api_secret': u'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER', u'topic_id': u'fbd34dac-549e-4141-b380-fd1307ddf098', u'embed': None, u'type': u'puddles'}}, '_ansible_ignore_errors': None, '_ansible_item_label': {u'component_types': [u'puddles'], u'name': u'OSP12', u'created_at': u'2018-11-22T19:53:20.504518', u'updated_at': u'2018-11-22T19:53:20.504518', u'label': None, u'state': u'active', u'etag': u'7fd8882ffc02558c93fdb91a12e8d8b1', u'id': u'fbd34dac-549e-4141-b380-fd1307ddf098', u'next_topic_id': u'f3f8d341-de64-492e-b349-700a978b2588', u'data': {u'releasename': u'pike', u'registry': {u'login': None, u'password': None}}, u'export_control': False, u'product_id': u'322c9266-1ca7-4086-b481-ecf15fe48a63'}}) => {"changed": true, "item": {"changed": false, "component": {"canonical_project_name": "RH7-RHOS-OSP12.0-2018-01-13", "created_at": "2018-11-22T19:54:32.128041", "data": {}, "etag": "6ff9f1426799f7daa781e3631b39879b", "export_control": true, "id": "0f2d3031-cb4a-4cdc-8117-5bc93ceb33e0", "message": null, "name": "RH7-RHOS-OSP12-2018-01-13", "state": "active", "title": null, "topic_id": "fbd34dac-549e-4141-b380-fd1307ddf098", "type": "puddles", "updated_at": "2018-11-22T19:54:32.128041", "url": "https://www.example.com"}, "failed": false, "invocation": {"module_args": {"active": true, "canonical_project_name": "RH7-RHOS-OSP12.0-2018-01-13", "data": null, "dci_api_secret": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "dci_client_id": "feeder/7ad59c5d-95e4-4005-8204-4901a0da6231", "dci_cs_url": "http://192.168.100.14", "dci_login": null, "dci_password": null, "dest": null, "embed": null, "id": null, "name": "RH7-RHOS-OSP12-2018-01-13", "path": null, "state": "present", "topic_id": "fbd34dac-549e-4141-b380-fd1307ddf098", "type": "puddles", "url": "https://www.example.com"}}, "item": {"component_types": ["puddles"], "created_at": "2018-11-22T19:53:20.504518", "data": {"registry": {"login": null, "password": null}, "releasename": "pike"}, "etag": "7fd8882ffc02558c93fdb91a12e8d8b1", "export_control": false, "id": "fbd34dac-549e-4141-b380-fd1307ddf098", "label": null, "name": "OSP12", "next_topic_id": "f3f8d341-de64-492e-b349-700a978b2588", "product_id": "322c9266-1ca7-4086-b481-ecf15fe48a63", "state": "active", "updated_at": "2018-11-22T19:53:20.504518"}}}
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, 'failed': False, u'changed': False, u'component': {u'name': u'RH7-RHOS-OSP13-2018-01-13', u'title': None, u'canonical_project_name': u'RH7-RHOS-OSP13.0-2018-01-13', u'data': {}, u'created_at': u'2018-11-22T19:54:32.613795', u'updated_at': u'2018-11-22T19:54:32.613795', u'id': u'bac6b842-92df-41e2-b51d-9092c0664c39', u'url': u'https://www.example.com', u'state': u'active', u'etag': u'63045b7465d69616510d28e137e164ef', u'topic_id': u'f3f8d341-de64-492e-b349-700a978b2588', u'message': None, u'type': u'puddles', u'export_control': True}, 'item': {u'next_topic_id': None, u'product_id': u'322c9266-1ca7-4086-b481-ecf15fe48a63', u'created_at': u'2018-11-22T19:53:17.473287', u'updated_at': u'2018-11-22T19:53:17.473287', u'label': None, u'state': u'active', u'etag': u'13478a8ea90cacf50c0870673f069138', u'id': u'f3f8d341-de64-492e-b349-700a978b2588', u'component_types': [u'puddles'], u'data': {u'releasename': u'queens', u'registry': {u'login': None, u'password': None}}, u'export_control': False, u'name': u'OSP13'}, u'invocation': {u'module_args': {u'canonical_project_name': u'RH7-RHOS-OSP13.0-2018-01-13', u'dci_client_id': u'feeder/7ad59c5d-95e4-4005-8204-4901a0da6231', u'path': None, u'active': True, u'dci_cs_url': u'http://192.168.100.14', u'data': None, u'id': None, u'dci_login': None, u'name': u'RH7-RHOS-OSP13-2018-01-13', u'url': u'https://www.example.com', u'dci_password': None, u'dest': None, u'state': u'present', u'dci_api_secret': u'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER', u'topic_id': u'f3f8d341-de64-492e-b349-700a978b2588', u'embed': None, u'type': u'puddles'}}, '_ansible_ignore_errors': None, '_ansible_item_label': {u'component_types': [u'puddles'], u'name': u'OSP13', u'created_at': u'2018-11-22T19:53:17.473287', u'updated_at': u'2018-11-22T19:53:17.473287', u'label': None, u'state': u'active', u'etag': u'13478a8ea90cacf50c0870673f069138', u'id': u'f3f8d341-de64-492e-b349-700a978b2588', u'next_topic_id': None, u'data': {u'releasename': u'queens', u'registry': {u'login': None, u'password': None}}, u'export_control': False, u'product_id': u'322c9266-1ca7-4086-b481-ecf15fe48a63'}}) => {"changed": true, "item": {"changed": false, "component": {"canonical_project_name": "RH7-RHOS-OSP13.0-2018-01-13", "created_at": "2018-11-22T19:54:32.613795", "data": {}, "etag": "63045b7465d69616510d28e137e164ef", "export_control": true, "id": "bac6b842-92df-41e2-b51d-9092c0664c39", "message": null, "name": "RH7-RHOS-OSP13-2018-01-13", "state": "active", "title": null, "topic_id": "f3f8d341-de64-492e-b349-700a978b2588", "type": "puddles", "updated_at": "2018-11-22T19:54:32.613795", "url": "https://www.example.com"}, "failed": false, "invocation": {"module_args": {"active": true, "canonical_project_name": "RH7-RHOS-OSP13.0-2018-01-13", "data": null, "dci_api_secret": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "dci_client_id": "feeder/7ad59c5d-95e4-4005-8204-4901a0da6231", "dci_cs_url": "http://192.168.100.14", "dci_login": null, "dci_password": null, "dest": null, "embed": null, "id": null, "name": "RH7-RHOS-OSP13-2018-01-13", "path": null, "state": "present", "topic_id": "f3f8d341-de64-492e-b349-700a978b2588", "type": "puddles", "url": "https://www.example.com"}}, "item": {"component_types": ["puddles"], "created_at": "2018-11-22T19:53:17.473287", "data": {"registry": {"login": null, "password": null}, "releasename": "queens"}, "etag": "13478a8ea90cacf50c0870673f069138", "export_control": false, "id": "f3f8d341-de64-492e-b349-700a978b2588", "label": null, "name": "OSP13", "next_topic_id": null, "product_id": "322c9266-1ca7-4086-b481-ecf15fe48a63", "state": "active", "updated_at": "2018-11-22T19:53:17.473287"}}}

But the API returns a http error 500 as showned in the httpd logs:

POST /api/v1/components/0f2d3031-cb4a-4cdc-8117-5bc93ceb33e0/files HTTP/1.1" 500 538 "-" "Ansible/2.7.2 (python-dciclient/0.5.1, python-dciauth/2.0.2)
POST /api/v1/components/bac6b842-92df-41e2-b51d-9092c0664c39/files HTTP/1.1" 500 538 "-" "Ansible/2.7.2 (python-dciclient/0.5.1, python-dciauth/2.0.2)

It seems that the dci_component hasn't been migrated to the new methods implemented in dci_common (with http error 500 handled) [1].

$ grep -r parse_http_response modules/
modules/dci_feeder.py:    result = parse_http_response(http_response, dci_feeder, context, module)
modules/dci_file.py:    result = parse_http_response(http_response, dci_file, context, module)
modules/dci_job.py:    result = parse_http_response(http_response, dci_job, context, module)
modules/dci_product.py:    result = parse_http_response(http_response, dci_product, context, module)
modules/dci_remoteci.py:    result = parse_http_response(http_response, dci_remoteci, context, module)
modules/dci_role.py:    result = parse_http_response(http_response, dci_role, context, module)
modules/dci_team.py:    result = parse_http_response(http_response, dci_team, context, module)
modules/dci_test.py:    result = parse_http_response(http_response, dci_test, context, module)
modules/dci_topic.py:    result = parse_http_response(http_response, dci_topic, context, module)
modules/dci_user.py:    result = parse_http_response(http_response, dci_user, context, module)

[1] https://github.com/redhat-cip/dci-ansible/blob/master/module_utils/dci_common.py

The 'cryptography' distribution was not found and is required by ansible

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 85, in <module>
    mycli = getattr(__import__("ansible.cli.%s" % sub, fromlist=[myclass]), myclass)
  File "/usr/lib/python2.7/site-packages/ansible/cli/__init__.py", line 38, in <module>
    from ansible.inventory.manager import InventoryManager
  File "/usr/lib/python2.7/site-packages/ansible/inventory/manager.py", line 29, in <module>
    from ansible.inventory.data import InventoryData
  File "/usr/lib/python2.7/site-packages/ansible/inventory/data.py", line 29, in <module>
    from ansible.plugins.cache import FactCache
  File "/usr/lib/python2.7/site-packages/ansible/plugins/cache/__init__.py", line 30, in <module>
    from ansible.plugins.loader import cache_loader
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 22, in <module>
    from ansible.parsing.plugin_docs import read_docstring
  File "/usr/lib/python2.7/site-packages/ansible/parsing/plugin_docs.py", line 12, in <module>
    from ansible.parsing.yaml.loader import AnsibleLoader
  File "/usr/lib/python2.7/site-packages/ansible/parsing/yaml/loader.py", line 30, in <module>
    from ansible.parsing.yaml.constructor import AnsibleConstructor
  File "/usr/lib/python2.7/site-packages/ansible/parsing/yaml/constructor.py", line 29, in <module>
    from ansible.parsing.vault import VaultLib
  File "/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py", line 58, in <module>
    from Crypto.Cipher import AES as AES_pycrypto
  File "build/bdist.linux-x86_64/egg/Crypto/Cipher/AES.py", line 50, in <module>
  File "build/bdist.linux-x86_64/egg/Crypto/Cipher/_AES.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/Crypto/Cipher/_AES.py", line 3, in __bootstrap__
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'cryptography' distribution was not found and is required by ansible

ansible-2.4.2.0-1.el7.noarch

I had the problem on RHEL7. Looks like a packaging issue to me. I fixed that with: yum install -y python-cryptography

unhandled Python exception in dci_file.py

Oct 11 15:04:11 rhjump2.fv2.org python2[20775]: detected unhandled Python exception in '/tmp/ansible_HGfqBy/ansible_module_dci_file.py'
Oct 11 15:04:12 rhjump2.fv2.org ansible-playbook[18114]: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Oct 11 15:04:12 rhjump2.fv2.org ansible-playbook[18114]: failed: [localhost] (item=/auto_results/overcloud_deployment.log) => {"changed": false, "item": "/auto_results/overcloud_deployment.log", "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_HGfqBy/ansible_module_dci_file.py\", line 196, 
in <module>\n    main()\n  File \"/tmp/ansible_HGfqBy/ansible_module_dci_file.py\", line 190, in main\n    result = parse_http_response(http_response, dci_file, context, module)\n  File \"/tmp/ansible_HGfqBy/ansible_modlib.zip/ansible/module_utils/common.py\", line 182, in parse_http_response\n  File \"/usr/lib/pytho
n2.7/site-packages/requests/models.py\", line 826, in json\n    return complexjson.loads(self.text, **kwargs)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 516, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 374,
 in decode\n    obj, end = self.raw_decode(s)\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 404, in raw_decode\n    return self.scan_once(s, idx=_w(s, idx).end())\nsimplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n", "module_stdout": "", "msg": "MODULE FAILURE
", "rc": 1}

The callback (at least) is not python3 compatible.

Raises errors trying to serialize bytes from comments.

 [WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin (</home/naz/eNovance/git/DCI/dci-
ansible/callback/dci.CallbackModule object at 0x7f85e48f8a58>): b'success' is not JSON serializable

Which is fixed by the following:

--- a/callback/dci.py
+++ b/callback/dci.py
@@ -178,7 +178,7 @@ class CallbackModule(CallbackBase):
             # If no name has been specified to the play, play.name is equal
             # to the hosts value
             elif play.name and play.name not in play.hosts:
-                comment = play.name.encode('UTF-8')
+                comment = play.name #play.name.encode('UTF-8')
             else:
                 comment = ''

Also throws a bunch of exceptions after running:

Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f0eb5f15268>
Traceback (most recent call last):
  File "/home/naz/.virtualenvs/dci/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable

dci_file: global name 'get_standard_action' is not defined

dci_file fails with the following message:

TASK [Upload logs] ***********************************************************************************************************************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NameError: global name 'get_standard_action' is not defined
failed: [localhost] (item=/auto_results/deployer.log.2018.01.02-14.10) => {"changed": false, "item": "/auto_results/deployer.log.2018.01.02-14.10", "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_628aIS/ansible_module_dci_file.py\", line 195, in <module>\n    main()\n  File \"/tmp/ansible_628aIS/ansible_module_dci_file.py\", line 183, in main\n    action_name = get_standard_action(module.params)\nNameError: global name 'get_standard_action' is not defined\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NameError: global name 'get_standard_action' is not defined
failed: [localhost] (item=/auto_results/deployer.log.2018.01.02-14.31) => {"changed": false, "item": "/auto_results/deployer.log.2018.01.02-14.31", "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_QOgmb5/ansible_module_dci_file.py\", line 195, in <module>\n    main()\n  File \"/tmp/ansible_QOgmb5/ansible_module_dci_file.py\", line 183, in main\n    action_name = get_standard_action(module.params)\nNameError: global name 'get_standard_action' is not defined\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

callback: Remove ansible setup task

Since 335b77d the setup task are uploaded in dci files.

$ dcictl job-list-file 6af3badd-9b13-43a0-8360-f6bfd4f5e78a|grep setup
| f728041e-17f3-41ef-8d03-246ce22cee8d |                 setup                  | 86b7879dca157c5c80559dc25865fa15 | 2017-05-26T14:33:08.400237 | 2017-05-26T14:33:08.400245 | active | 27c36b85-5a49-469e-b3d8-e372db8c2e06 | 4b555c07-f36a-49c3-9628-1e81e50b4293 | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| 689caea1-51fc-437d-a40b-81fce8f5b4a3 |                 setup                  | 017f8fbd39147a94409f6b472bce5e95 | 2017-05-26T14:31:16.960796 | 2017-05-26T14:31:16.960804 | active | 27c36b85-5a49-469e-b3d8-e372db8c2e06 | d5708f0b-c133-4d90-89a6-c83a94cb9ccc | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| d6dca37f-5e98-44b2-8ff8-34bfb760ac64 |                 setup                  | 59ee35cc22b2566cbc0d8b1a73f11207 | 2017-05-26T14:27:34.923780 | 2017-05-26T14:27:34.923788 | active | 87f0ade9-9796-4c79-8bee-f790d2f21eb6 | 00d089ef-8c1c-44cc-b43c-1c29362ec8de | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| 923844c1-67a6-4916-b9ce-2007bf2b3d6a |                 setup                  | 3ca0c29efce5fa22d8d6ded72c2fc98a | 2017-05-26T14:25:14.442245 | 2017-05-26T14:25:14.442254 | active | 27c36b85-5a49-469e-b3d8-e372db8c2e06 | 1af9483b-696f-49a8-852e-89e4778c9f71 | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| 8e53ecd2-8879-4923-8018-aa46908839b8 |                 setup                  | 19f6e7b3f0c029cb3016792fc31e258c | 2017-05-26T14:25:07.125879 | 2017-05-26T14:25:07.125888 | active | 87f0ade9-9796-4c79-8bee-f790d2f21eb6 | f652e27e-630f-458a-896c-b3cdd1f8d50f | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| dbc90676-dca7-40e6-8793-d79d1109d3eb |                 setup                  | c78fee67ab7da44b7c774bf5029b60ea | 2017-05-26T14:24:05.202560 | 2017-05-26T14:24:05.202569 | active | 87f0ade9-9796-4c79-8bee-f790d2f21eb6 | 9a0937f8-44bf-4017-9e09-8cffee06fc8e | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| e4c62274-a186-4318-8fc9-6d7624a71fd0 |                 setup                  | 56d2f5373c045dc0e8dca79bf8a2cb6c | 2017-05-24T14:37:00.539098 | 2017-05-24T14:37:00.539112 | active | be863e82-0444-4bfe-a78a-01996f5b81f7 | c3fd9864-eeb8-41c5-befe-1a2a78154ab2 | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| 2a3438d2-0fbd-4a9e-bc1e-e0605d2741f7 |                 setup                  | 03f255c6fd6e4074f23c112f1bb9a703 | 2017-05-24T13:03:38.849329 | 2017-05-24T13:03:38.849336 | active | be863e82-0444-4bfe-a78a-01996f5b81f7 | 0a73d8dd-d562-415a-9b15-a8277c433a37 | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |
| 1b4053af-1424-47dc-bf3c-038587a59990 |                 setup                  | a90127104d9dd8fd7435d9825fbc1663 | 2017-05-24T13:01:52.129014 | 2017-05-24T13:01:52.129035 | active | be863e82-0444-4bfe-a78a-01996f5b81f7 | c86727ea-206e-4f96-926f-981660dfb6eb | None | text/plain |   20   | ddfe879b-71dc-41a6-967c-d3caabaf4943 |   None  |

callback: job_informations not refresh on new register

Since we are using the job_informations value coming from the fact_cache (via register in the playbook) if we run an job upgrade in the same playbook, the new tasks will still be registered in the previous job and not in the new one.

$ dcictl job-list
+--------------------------------------+--------------------+---------------+---------+-----------+
|                  id                  | jobdefinition/name | remoteci/name |  status | team/name |
+--------------------------------------+--------------------+---------------+---------+-----------+
| dc8b4507-72cb-4383-899f-4faa358027ca |       OSP 11       |      toto     |   new   |   testor  |
| 56c309b4-68db-4c8e-8989-a5f0d6dc6592 |       OSP 10       |      toto     | success |   testor  |
+--------------------------------------+--------------------+---------------+---------+-----------+
$ dcictl jobstate-list --where job_id:56c309b4-68db-4c8e-8989-a5f0d6dc6592
+--------------------------------------+----------------------------+-------------------------+--------------------------------------+----------+--------------------------------------+
|                  id                  |         created_at         |         comment         |                job_id                |  status  |               team_id                |
+--------------------------------------+----------------------------+-------------------------+--------------------------------------+----------+--------------------------------------+
| c38f0009-8cea-4d85-9bb4-a7d04ca3a46b | 2017-06-14T15:48:29.046723 |  Success state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | success  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 7dc5fec4-a29b-4d23-8462-ef47c1f3335d | 2017-06-14T15:48:24.076172 | Post-run state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | post-run | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| ca4f858a-3e5f-4e53-abf7-2b6f3a974e97 | 2017-06-14T15:48:22.189393 |  Running state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | running  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 8b706a5b-f0a7-4ec5-9a1f-ee285d01fbed | 2017-06-14T15:48:20.292509 |  Pre-run state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | pre-run  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| fd8c5be6-2e59-4606-aa3d-d19ff07e4fc7 | 2017-06-14T15:48:18.401393 |    New state commands   | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 |   new    | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 9fcfb624-a5d8-428d-be5b-08f95bb51169 | 2017-06-14T15:48:12.064783 |  Success state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | success  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 228fc7d9-32d1-404c-9e2d-f6984c6639c1 | 2017-06-14T15:48:07.266719 | Post-run state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | post-run | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 565c360d-f706-4f61-a52b-e16959cb3a4f | 2017-06-14T15:48:05.396912 |  Running state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | running  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 45844715-1e06-4e99-92fa-cc4c7568f114 | 2017-06-14T15:48:03.537091 |  Pre-run state commands | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 | pre-run  | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
| 8ccae0e1-d958-4241-bddd-bebe26cc2e94 | 2017-06-14T15:48:01.283013 |    New state commands   | 56c309b4-68db-4c8e-8989-a5f0d6dc6592 |   new    | b498e565-d3e7-4750-a95b-14b1cdc598d7 |
+--------------------------------------+----------------------------+-------------------------+--------------------------------------+----------+--------------------------------------+
$ dcictl jobstate-list --where job_id:dc8b4507-72cb-4383-899f-4faa358027ca
++
||
++
++

The sample https://github.com/redhat-cip/dci-ansible/blob/master/samples/sample_upgrade.yml can be used to reproduce the issue.

cryptic error when recreating a resource that has not been purged

If a resource has been remove but not purge, the SQL contraint will still take it into account and refuse to create a new resource. The server will return a 409 error and DCI Ansible pop up the following error.

TASK [Create OpenStack] **********************************************************************************************
task path: /home/goneri/git_repos/dci-ansible/tests/unit-tests/dci_topic/playbook.yml:9
Using module file /home/goneri/git_repos/dci-ansible/modules/dci_product.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: goneri
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735 `" && echo ansible-tmp-1515026454.23-227325190824735="` echo /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmp5goM0m TO /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735/dci_product.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735/ /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735/dci_product.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/goneri/venv/py2/bin/python2 /home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735/dci_product.py; rm -rf "/home/goneri/.ansible/tmp/ansible-tmp-1515026454.23-227325190824735/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_pxJTJp/ansible_module_dci_product.py", line 164, in <module>
    main()
  File "/tmp/ansible_pxJTJp/ansible_module_dci_product.py", line 158, in main
    result = parse_http_response(http_response, dci_product, context, module)
  File "/tmp/ansible_pxJTJp/ansible_modlib.zip/ansible/module_utils/common.py", line 158, in parse_http_response
IndexError: list index out of range

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_pxJTJp/ansible_module_dci_product.py\", line 164, in <module>\n    main()\n  File \"/tmp/ansible_pxJTJp/ansible_module_dci_product.py\", line 158, in main\n    result = parse_http_response(http_response, dci_product, context, module)\n  File \"/tmp/ansible_pxJTJp/ansible_modlib.zip/ansible/module_utils/common.py\", line 158, in parse_http_response\nIndexError: list index out of range\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.