schubergphilis / towerlib

A python library to interface with ansible tower's (awx) api.

towerlib

A python library to interface with ansible tower's (awx) api.

• Documentation: https://towerlib.readthedocs.org/en/latest

Development Workflow

The workflow supports the following steps

• lint
• test
• build
• document
• upload
• graph

These actions are supported out of the box by the corresponding scripts under _CI/scripts directory with sane defaults based on best practices. Sourcing setup_aliases.ps1 for windows powershell or setup_aliases.sh in bash on Mac or Linux will provide with handy aliases for the shell of all those commands prepended with an underscore.

The bootstrap script creates a .venv directory inside the project directory hosting the virtual environment. It uses pipenv for that. It is called by all other scripts before they do anything. So one could simple start by calling _lint and that would set up everything before it tried to actually lint the project

Once the code is ready to be delivered the _tag script should be called accepting one of three arguments, patch, minor, major following the semantic versioning scheme. So for the initial delivery one would call

$ _tag --minor

which would bump the version of the project to 0.1.0 tag it in git and do a push and also ask for the change and automagically update HISTORY.rst with the version and the change provided.

So the full workflow after git is initialized is:

• repeat as necessary (of course it could be test - code - lint :) ) * code * lint * test
• commit and push
• develop more through the code-lint-test cycle
• tag (with the appropriate argument)
• build
• upload (if you want to host your package in pypi)
• document (of course this could be run at any point)

Important Information

This template is based on pipenv. In order to be compatible with requirements.txt so the actual created package can be used by any part of the existing python ecosystem some hacks were needed. So when building a package out of this do not simple call

$ python setup.py sdist bdist_egg

as this will produce an unusable artifact with files missing. Instead use the provided build and upload scripts that create all the necessary files in the artifact.

Project Features

• Can get, create and delete all entities in a tower deployment.

Todo

This code is MVP and needs a lot of improvements. Some of them are, optimize all queries, implement searching and filtering and implement smart caching of entities.

towerlib's Issues

Job Launch Failures not displaying reason

Hello,

First of all - great library! been using it internally and it has made my life a whole lot easier :)

I have noticed that when a job template is launched (whereby a survey or variable is missing, credential cant be found etc), it returns None as expected (based on the entity code), however the logger when resp.ok is not true is not displaying the actual output.

I have attempted to use the default logging framework, however that did not yield any output. My current project is using loguru to handle logging, with my test cases failing as expected.. however without a displayable cause.

Is it expected that the log message (

towerlib/towerlib/entities/job.py

Line 1365 in e4ae19f

self._logger.error('Error launching job %s, response was :%s', self.name, response.text)

) should be displayed? (i.e., pk credential not found, missing survey vars)

If not.. how can this be caught to surface the reason of failure?

Cheers,
Jeremy

CI test failled (Temporary failure in name resolution)

Error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='placeholder_hostname', port=80): Max retries exceeded with url: /api/v2/users/?page_size=25&username__iexact=workflow_normal (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5dc5fe9250>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

The 'placeholder_hostname' seems to be a string.
Need to be added to /etc/hosts?
Need to be transformed with a sed command before executing?

What was the idea? :)
How can we help.

Error when launching a job: return the error

Hello there 👋

When a job fail the current behavior is to only log and return None.

Would it be possible to return the error? Could be useful 😁

Thanks !

AttributeError: 'Tower' object has no attribute 'get_inventory_by_name'

Hello,
i was trying to follow https://towerlib.readthedocs.io/en/latest/usage.html
i did not try all the examples, but the ones i tried were working fine, except for
# or inventory = tower.get_inventory_by_name('inventory_name')
AttributeError: 'Tower' object has no attribute 'get_inventory_by_name'

credentials password/token setter overwrites entire input dict.

Getting a credential will get you something like below(truncated section of the objects _data property)

'inputs': {
      'hashi_vault_addr': 'anexampleaddress', 
      'hashi_vault_pre_python_279_cahostverify': 'no', 
      'hashi_vault_token': '$encrypted$'}

in the case of a credential of type hashivault token, using the setter on 'token' like this

credential = tower.get_organization_credential_by_name('my_org', 'my_credential_name', 'Hashicorp Vault')
credential.token = 'new_token_value' 

will wipe out the other k/vs in the inputs dictionary and leave you with a broken credential item in awx if those other inputs are required.

Issue is noticed on AWX 9.0.1.0

Issue does not present against tower 3.3.0

TypeError: Object of type User is not JSON serializable

Hello,

We use towerlib with flask project. An endpoint returns a user object but we have an error because the object User is not serializable.

A quick example :

$ from towerlib.entities import User
$ import json
$ user = User(None, None)
$ json.dumps(user)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type User is not JSON serializable

Do you have a solution to expose a User object in json ?

"copy" features are missed.

on awx web UI, users may create job templates or inventories by copying existing ones. it would be great helpful if the towerlib supports theses features.

Allow all prompt on launch fields in launch Job

(See the commit)

Unset default values on prompt on launch fields

We must unset value or set value to false for some parameters of job launch. (See the commit)

Support for HTTP Patch

We need to update some job_template on AWX/Tower.
I added a method to make an HTTP PATCH request

PR is coming

Anthony

Usage documentation does not include filtering exmaples

Hi,

I have raise a PR (#77) to update the USAGE docs as the filtering implementation is really easy to use but is not covered on the USAGE page.

In my use case, iterating all "successful" jobs was significantly quicker using the filter via the API then making thousands of API calls and filtering upon their receipt with Python. I think this is such as common use case it should have a little coverage in the USAGE page as I had to look through the source code to find it.

credential should become an optional argument for job template

The credential attribtue should become an optional argument for create_job_template.

Why?
Currently only credential type machine are supported. There are use cases where job template do not care about credential variables.

IMO there are two possibilities.

1. Keep the credential property as long it's supported by the AWX api but make it optional.
2. Keep the credential as required but make it accept None.

The introduction of credentials property (plural) (see #28) should be kept seperated from this.

pypi link

Hi,

The project link on pypi points to https://github.com/costastf/towerlib.git, which gives a 404. So I had to search on github to find you. I think I've got the correct one.

Have a nice day,
Bert

Feature Request: self-signed certificate authority support?

From what I can tell, I don't see this as a current feature within towerlib. I think it would be awesome if towerlib supported self-signed HTTPS SSL certificates.

For example, my Ansible Tower installation is on my local network. It uses a certificate that I signed with the internal certificate authority that I have created. I would love to be able to reference my certificate authority when calling towerlib; something like:

from towerlib import Tower

# using https with a self-signed certificate
tower = Tower('ansible.internal.ericoc.com', 'eric', 'password', secure=True, ssl_verify=True, ssl_cacert='/etc/ssl/certs/internal.ericoc.com.ca.crt')

The python-requests documentation makes me think that this should theoretically be possible, but I am no Python expert.

Thanks in advance for the consideration - cheers!

add support for cloud credential type

It would be a big improvement to have the ability to attach credential of type cloud to a job template via api call.

Currently its only possible to use machine credentials with the single credential property in job_template. Other credentials types are not supported. The single credential type is subject to delete¹.

The proper way to to attach credentials of other types would be to use the following endpoint after the job template has been created:

/api/v2/job_templates/<id>/credentials/

A simple payload containing the credential id is neccessary:

{'id': 3}

¹ ansible/awx#3413

__init__() got an unexpected keyword argument 'token'

tower = Tower('ansible.domain.com', None, None, secure=True, token="token-data-here")
TypeError: init() got an unexpected keyword argument 'token'

ConnectionError on job template relative credential

Using towerlib version from pip:

$ pip list|grep tower
towerlib           2.3.4    

When fetching a job template and trying to receive the credential for the job template this fails with a ConnectionError:

t = Tower('awx2.admin.lab', 'admin', '*******')
base_job_template = t.get_job_template_by_name('vm-expansion-template')
base_job_template.project                                                                                                                                                                                                
<towerlib.entities.project.Project at 0x7fd2bb167668>

base_job_template.credential                                                                                                                                                                                             

~/.virtualenvs/airflow/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    514                 raise SSLError(e, request=request)
    515 
--> 516             raise ConnectionError(e, request=request)
    517 
    518         except ClosedPoolError as e:

ConnectionError: HTTPConnectionPool(host='awx2.admin.labnone', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd2bb01f7b8>: Failed to establish a new connection: [Errno -2] Name or service not known'))

The URL instead of awx2.admin.labnone should be awx2.admin.lab. I don't know why the none is appended.

if entity_name else f'{self._tower.host}{url}', when url=None, var url returns string None

class EntityManager:
    """Manages entities by making them act like iterables but also implements contains and other useful stuff."""

    # pylint: disable=too-many-arguments
    def __init__(self, tower_instance, entity_object, primary_match_field, entity_name=None, url=None):
        if not any([entity_name, url]):
            raise ValueError('Either entity_name or url needs to be provided, received none.')
        self._tower = tower_instance
        self._object_type = entity_object
        self._primary_match_field = primary_match_field
        self._name = entity_name
        self._next_state = None
        self._url = f'{self._tower.api}/{entity_name}' if entity_name else f'{self._tower.host}{url}'

i think i am hitting this
if entity_name else f'{self._tower.host}{url}'
and getting
jobs has failed with exception: HTTPSConnectionPool(host='localhostnone', port=443): Max retries exceeded with url: / (Caused by NewConnection Error('<urllib3.connection.HTTPSConnection object at 0x7f9aa2b4e520>: Failed to establish a new connection: [Errno -2] Name or service not known'))

what do you think

Feature request: Link the survey spec to the job template when exist

The current JobTemplate object only return a bool survey_enabled.

It would be nice to retrieve the full survey spec.

towerlib.Tower class constructor: mutual TLS connectivity option

I am using AWX/tower through a bastion-host/access-proxy (i.e. teleport) which mandates mutual TLS connections. The python requests library (used by towerlib) already has a way to specify client-side certificates for a mutual-TLS connection.

(https://requests.readthedocs.io/en/latest/user/advanced/#client-side-certificates)

Are you able to expose this option through the towerlib.Tower class API?

Connection pool is full

Hello,

After running this through a few internal pipelines for verification, I've noticed on longer-running jobs I begin to see the following in the job monitor/output:

Connection pool is full, discarding connection: my.ansible.url.here

I understand this is part of the urllib3/requests Session management. I was wondering if you have faced this before?

I suspect it might be related to max_workers, in the _get_paginated_response method (

towerlib/towerlib/towerlib.py

Line 293 in e4ae19f

with concurrent.futures.ThreadPoolExecutor(max_workers=25) as executor:

).

Are there any recommended approaches to optimize the connection pool being used?

Python version compatibility

Hi there,

I got an issue with the pipfile and python version compatibility. I know that this package is mainly written for python 3.7 and I had a look if I could run this code with python 3.6. As far as I can tell it worked, but still I noticed a very special case of package dependency. For python 3.6 there is a pip package for "dataclasses" which is not necessary for python 3.7.

To reflect this requirement, I can write the following line into the pipfile:

[packages]
dataclasses = { version = "==0.6", markers="python_version == 3.7" }

Such usage of pipfile is described here: https://pipenv-fork.readthedocs.io/en/latest/advanced.html

However, by doing so, the code in _CI/library/core_library.py breaks. It seems to me, that it can't handle a package dict.

lookup of jobs

It should be possible to lookup jobs either by id or by name.

Version 3.12.1 should not have been pushed on main

Version 3.12.1 seems to be released (the tag was published March, 29th 2023), but tests does not passes. It's a little bit confusing while documentation states that this version exists, but CI build step failed on tests and it wasn't published on pypi.

Feature request: workflow job template add_schedule method

Hello,

Actually it's not possible to create a schedule for a workflow job template.
Do you have planned to add this to the library ?

Feature Request : extra vars for schedules

Hello,

Actually it's not possible to add extra-vars for a schedule (workflow job template / job template).
Do you have planned to add this to the library ?

Wrong URL on follow up request to the REST API

Environment: RHEL, Python 3.6, towerlib pip installed earlier today.

My code is more or less this:

    tower = Tower('localhost', 'username', 'password', secure=True, ssl_verify=False)

    for organization in tower.organizations:
        for cred in organization.credentials:
            print(cred)

Fails on the looping over organization.credentials.
Noticed the line with HTTPSConnectionPool(host='localhosthttps', port=443), which seems to indicate the url is build up wrong for this follow up request.

I have the same error for other iterable variables, not only organization.credentials.

Traceback (most recent call last):
  File "./awx_security_audit.py", line 109, in <module>
    print_report(tower)
  File "./awx_security_audit.py", line 54, in print_report
    for cred in organization.credentials:
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/towerlib/entities/core.py", line 310, in _get_entity_objects
    for data in self._tower._get_paginated_response(self._url, params=params):  # pylint: disable=protected-access
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/towerlib/towerlib.py", line 270, in _get_paginated_response
    response = self.session.get(url)
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/soft/ansible/home/auditor/venv/lib64/python3.6/site-packages/requests/adapters.py", line 507, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='localhosthttps', port=443): Max retries exceeded with url: //localhost/api/v2/organizations/7/credentials/?page_size=25 (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 notresolvable',)))

How to pass multiple credentials to JobTemplate.launch?

awx 15.0.0
towerlib 3.4.1
python 3.8

I have a job template that needs multiple credentials. I can fetch the credentials with tower.get_credentials_by_name() and find the job template with tower.get_job_template_by_name().

JobTemplate.launch() takes a credential parameter. I have tried passing a list of the found credentials, and also their IDs (the docs don't say which is required), but neither succeeds.

     job_template = tower.get_job_template_by_name(template_name)
     extra_vars = {}
     creds = [12,8]  # (IDs fetched from get_credentials_by_name()
     job = job_template.launch(extra_vars=extra_vars, credential=creds)    

    {"written_at": "2020-11-03T16:20:49.335Z", "written_ts": 1604420449335948000, "msg": "Unknown entity type None", "type": "log", "logger": "jobs", "thread": "Dummy-3", "level": "ERROR", "module": "job", "line_no": 75}

The error appears to be related to something internal to towerlib checking a type variable that I don't set.

Why return a generator when the return only have one instence?

towerlib/towerlib/towerlib.py

Lines 963 to 973 in e4ae19f

	def get_hosts_by_name(self, name):
	"""Retrieves hosts by name.
	Args:
	name: The name of the hosts to retrieve.
	Returns:
	hosts (Generator): A generator with the matching hosts
	"""
	return self.hosts.filter({'name__iexact': name})

Appreciated your job.

Why we return a generator rather than a host instence in this function?
BTW, the name get_hosts_by_name seems has multiple hosts return, but the filter used exactly match...

Running Jobs

Can you run jobs and retrieve their status with this library? Seems this is just for creating, modifying, and deleting objects. How about running job templates?

towerlib.entities import error

Hello, could you pls suggest why I encounter error “no muddle towerlib found” happened on attached snippet.
Per my understanding, entities is a package not module, how can we use like that? I am freshman for python and confused for that. Appreciated.

Exception message triggers UnboundLocalError

The exception message in Credential.new uses a variable 'credential_type' defined in the try block.
Depending on the behaviour of the remote AWX the following UnboundLocalError might be triggered.

[ERROR] UnboundLocalError: local variable 'credential_type' referenced before assignment
Traceback (most recent call last):
  File "/opt/python/lib/python3.10/site-packages/datadog_lambda/wrapper.py", line 187, in __call__
    self.response = self.func(event, context, **kwargs)
  File "/var/task/cm_rotate_machine_process/lambda_function.py", line 59, in lambda_handler
    def lambda_handler(event, _):
  File "/opt/python/lib/python3.10/site-packages/ddtrace/contrib/aws_lambda/patch.py", line 96, in __call__
    self.response = self.func(*args, **kwargs)
  File "/var/task/cm_rotate_machine_process/lambda_function.py", line 88, in lambda_handler
    update_machine_credentials_in_awx(organization, vault, stack, environments)
  File "/var/task/cm_rotate_machine_process/library/actions.py", line 137, in update_machine_credentials_in_awx
    credentials = list(
  File "/opt/python/lib/python3.10/site-packages/towerlib/entities/core.py", line 374, in _get_entity_objects
    yield entity_object(self._tower, data)
  File "/opt/python/lib/python3.10/site-packages/towerlib/entities/credential.py", line 205, in __new__
    credential_type)

Retrieving credentials returns an error

Snippet of code to retrieve credentials:

for credential in tower.credentials:
print(credential.name);

Returns the following error:

Traceback (most recent call last):
File "./tower.py", line 280, in
GatherCredentialInfo(tower)
File "./tower.py", line 22, in GatherCredentialInfo
for credential in tower.credentials:
File "/home/mrloth/VPython/lib/python2.7/site-packages/towerlib/entities/core.py", line 322, in _get_entity_objects
yield entity_object(self._tower, data)
TypeError: this constructor takes no arguments