ansible-collections / ansible.utils

A collection of ansible utilities for the content creator.

Ansible Utilities Collection

The Ansible ansible.utils collection includes a variety of plugins that aid in the management, manipulation and visibility of data for the Ansible playbook developer.

Ansible version compatibility

This collection has been tested against following Ansible versions: >=2.14.0.

For collections that support Ansible 2.9, please ensure you update your network_os to use the fully qualified collection name (for example, cisco.ios.ios). Plugins and modules within a collection may be tested with only specific Ansible versions. A collection may contain metadata that identifies these versions. PEP440 is the schema used to describe the versions of Ansible.

Included content

Filter plugins

Name	Description
ansible.utils.cidr_merge	This filter can be used to merge subnets or individual addresses.
ansible.utils.consolidate	Consolidate facts together on common attributes.
ansible.utils.fact_diff	Find the difference between currently set facts
ansible.utils.from_xml	Convert given XML string to native python dictionary.
ansible.utils.get_path	Retrieve the value in a variable using a path
ansible.utils.hwaddr	HWaddr / MAC address filters
ansible.utils.index_of	Find the indices of items in a list matching some criteria
ansible.utils.ip4_hex	This filter is designed to convert IPv4 address to Hexadecimal notation with optional delimiter.
ansible.utils.ipaddr	This filter is designed to return the input value if a query is True, else False.
ansible.utils.ipcut	This filter is designed to get 1st or last few bits of IP address.
ansible.utils.ipmath	This filter is designed to do simple IP math/arithmetic.
ansible.utils.ipsubnet	This filter can be used to manipulate network subnets in several ways.
ansible.utils.ipv4	To filter only Ipv4 addresses Ipv4 filter is used.
ansible.utils.ipv6	To filter only Ipv6 addresses Ipv6 filter is used.
ansible.utils.ipv6form	This filter is designed to convert ipv6 address in different formats. For example expand, compressetc.
ansible.utils.ipwrap	This filter is designed to Wrap IPv6 addresses in [ ] brackets.
ansible.utils.keep_keys	Keep specific keys from a data recursively.
ansible.utils.macaddr	macaddr / MAC address filters
ansible.utils.network_in_network	This filter returns whether an address or a network passed as argument is in a network.
ansible.utils.network_in_usable	The network_in_usable filter returns whether an address passed as an argument is usable in a network.
ansible.utils.next_nth_usable	This filter returns the next nth usable ip within a network described by value.
ansible.utils.nthhost	This filter returns the nth host within a network described by value.
ansible.utils.param_list_compare	Generate the final param list combining/comparing base and provided parameters.
ansible.utils.previous_nth_usable	This filter returns the previous nth usable ip within a network described by value.
ansible.utils.reduce_on_network	This filter reduces a list of addresses to only the addresses that match a given network.
ansible.utils.remove_keys	Remove specific keys from a data recursively.
ansible.utils.replace_keys	Replaces specific keys with their after value from a data recursively.
ansible.utils.slaac	This filter returns the SLAAC address within a network for a given HW/MAC address.
ansible.utils.to_paths	Flatten a complex object into a dictionary of paths and values
ansible.utils.to_xml	Convert given JSON string to XML
ansible.utils.usable_range	Expand the usable IP addresses
ansible.utils.validate	Validate data with provided criteria

Lookup plugins

Name	Description
ansible.utils.get_path	Retrieve the value in a variable using a path
ansible.utils.index_of	Find the indices of items in a list matching some criteria
ansible.utils.to_paths	Flatten a complex object into a dictionary of paths and values
ansible.utils.validate	Validate data with provided criteria

Modules

Name	Description
ansible.utils.cli_parse	Parse cli output or text using a variety of parsers
ansible.utils.fact_diff	Find the difference between currently set facts
ansible.utils.update_fact	Update currently set facts
ansible.utils.validate	Validate data with provided criteria

Test plugins

Name	Description
ansible.utils.in_any_network	Test if an IP or network falls in any network
ansible.utils.in_network	Test if IP address falls in the network
ansible.utils.in_one_network	Test if IP address belongs in any one of the networks in the list
ansible.utils.ip	Test if something in an IP address or network
ansible.utils.ip_address	Test if something in an IP address
ansible.utils.ipv4	Test if something is an IPv4 address or network
ansible.utils.ipv4_address	Test if something is an IPv4 address
ansible.utils.ipv4_hostmask	Test if an address is a valid hostmask
ansible.utils.ipv4_netmask	Test if an address is a valid netmask
ansible.utils.ipv6	Test if something is an IPv6 address or network
ansible.utils.ipv6_address	Test if something is an IPv6 address
ansible.utils.ipv6_ipv4_mapped	Test if something appears to be a mapped IPv6 to IPv4 mapped address
ansible.utils.ipv6_sixtofour	Test if something appears to be a 6to4 address
ansible.utils.ipv6_teredo	Test if something appears to be an IPv6 teredo address
ansible.utils.loopback	Test if an IP address is a loopback
ansible.utils.mac	Test if something appears to be a valid MAC address
ansible.utils.multicast	Test for a multicast IP address
ansible.utils.private	Test if an IP address is private
ansible.utils.public	Test if an IP address is public
ansible.utils.reserved	Test for a reserved IP address
ansible.utils.resolvable	Test if an IP or name can be resolved via /etc/hosts or DNS
ansible.utils.subnet_of	Test if a network is a subnet of another network
ansible.utils.supernet_of	Test if a network is a supernet of another network
ansible.utils.unspecified	Test for an unspecified IP address
ansible.utils.validate	Validate data with provided criteria

Installing this collection

You can install the ansible.utils collection with the Ansible Galaxy CLI:

ansible-galaxy collection install ansible.utils

You can also include it in a requirements.yml file and install it with ansible-galaxy collection install -r requirements.yml, using the format:

---
collections:
  - name: ansible.utils

Using this collection

The most common use case for this collection is when you want to work with the complex data structures present in an Ansible playbook, inventory, or returned from modules. See each plugin documentation page for detailed examples for how these utilities can be used in tasks.

NOTE: For Ansible 2.9, you may not see deprecation warnings when you run your playbooks with this collection. Use this documentation to track when a module is deprecated.

See Also:

• Using collections in the Ansible documentation for more details.

Contributing to this collection

This collection is intended for plugins that are not platform or discipline specific. Simple plugin examples should be generic in nature. More complex examples can include real world platform modules to demonstrate the utility of the plugin in a playbook.

We welcome community contributions to this collection. If you find problems, please open an issue or create a PR against the ansible.utils collection repository. See Contributing to Ansible-maintained collections for complete details.

See the Ansible Community Guide for details on contributing to Ansible.

Developer notes

• 100% code coverage is the goal, although it's not always possible. Please include unit and integration tests with all PRs. PRs should not cause a decrease in code coverage.
• Filter plugins should be 1 per file, with an included DOCUMENTATION string, or reference a lookup plugin with the same name.
• Action, filter, and lookup plugins should use argspec validation. See AnsibleArgSpecValidator.
• This collection should not depend on other collections for imported code
• Use of the latest version of black is required for formatting (black -l79)
• The README contains a table of plugins. Use the collection_prep utilities to maintain this.

Code of Conduct

This collection follows the Ansible project's Code of Conduct. Please read and familiarize yourself with this document.

Release notes

Release notes are available here For automated release announcements refer here.

Roadmap

For information on releasing, versioning and deprecation see the stratergy document.

In general, major versions can contain breaking changes, while minor versions only contain new features (like new plugin addition) and bugfixes. The releases will be done on an as-needed basis when new features and/or bugfixes are done.

More information

• Ansible Collection overview
• Ansible User guide
• Ansible Developer guide
• Ansible Community code of conduct

Licensing

GNU General Public License v3.0 or later.

See LICENSE to see the full text.

ansible.utils's Issues

[to_xml] Support "indent" parameter for xmltodict engine

SUMMARY

to_xml defaults to indenting XML documents with tabs, as xmltodict .
xmltodict library unparse function supports and indent parameter that can be used to specify a different indent character (for example spaces).

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ansible.utils.to_xml

ADDITIONAL INFORMATION

The plugin should have two additional parameters:

• indent (can be "tab" or "space", default to "tab")
• indent_width (int, only used when indent="space", default to 4)

Parameters have to be parsed before sending them to unparse function, since this function only supports indent parameter to specify the character to use for indentation.

- name: Define JSON data
  ansible.builtin.set_fact:
    data:
      "interface-configurations":
          "@xmlns": "http://cisco.com/ns/yang/Cisco-IOS-XR-ifmgr-cfg"
          "interface-configuration":
- debug:
    msg:  "{{ data|ansible.utils.to_xml(indent="space", indent_width=2) }}"

If you are open to adding this feature, I am available to make a pull request for this.

ipaddr('prefix') returns [prefix] instead of just prefix

SUMMARY

ipaddr('prefix') returns [prefix] instead of just prefix like ansible.netcommon.ipaddr did before the migration.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipaddr('prefix')

ANSIBLE VERSION

ansible [core 2.12.3]
  python version = 3.9.10 (main, Jan 15 2022, 11:40:53) [Clang 13.0.0 (clang-1300.0.29.3)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

# /Users/.../ansible/.venv-ansible/lib/python3.9/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.5.0

# /Users/.../.ansible/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.5.1

CONFIGURATION

DEFAULT_STDOUT_CALLBACK(/Users/.../ansible/ansible.cfg) = yaml

OS / ENVIRONMENT

STEPS TO REPRODUCE

- set_fact:
     ansible_default_ipv4:
       address: "10.10.184.252"
       netmask: "255.255.240.0"

- debug:
    msg: "{{ (ansible_default_ipv4.address + '/' + ansible_default_ipv4.netmask) | ansible.netcommon.ipaddr('prefix') }}"

EXPECTED RESULTS

Result of debug: 20

ACTUAL RESULTS

Result of debug: [20]

from_xml filter returns a string instead of a python dictionary

SUMMARY

The from_xml filter returns a string instead of returning a python dictionary as its documentation reports and as the from_json and from_yaml filters do.

ISSUE TYPE

• Bug Report

COMPONENT NAME

from_xml

ANSIBLE VERSION

ansible [core 2.11.3] 
  config file = /home/user/workspace/ansible-playbooks-2/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/workspace/ansible-playbooks-2/.direnv/python-3.8.10/lib/python3.8/site-packages/ansible
  ansible collection location = /home/user/workspace/ansible-playbooks-2/collections
  executable location = /home/user/workspace/ansible-playbooks-2/.direnv/python-3.8.10/bin/ansible
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

# /home/user/workspace/ansible-playbooks-2/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.4.2  

CONFIGURATION

AGNOSTIC_BECOME_PROMPT(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = False
ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
CALLBACKS_ENABLED(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['timer', 'ansible.posix.profile_tasks']
COLLECTIONS_PATHS(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['/home/user/workspace/ansible-playbooks-2/collections']
DEFAULT_CONNECTION_PLUGIN_PATH(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['/home/user/workspace/ansible-playbooks-2/connection_plugins']
DEFAULT_FORKS(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = 10
DEFAULT_HOST_LIST(env: ANSIBLE_INVENTORY) = ['/home/user/workspace/ansible-playbooks-2/inventories/test']
DEFAULT_TIMEOUT(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = 10
DEFAULT_VAULT_PASSWORD_FILE(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = /home/user/workspace/ansible-playbooks-2/inventories/vault_password.sh
DISPLAY_SKIPPED_HOSTS(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = False
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False
INVENTORY_ANY_UNPARSED_IS_FAILED(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = True
INVENTORY_ENABLED(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['host_list', 'ini', 'constructed']
INVENTORY_IGNORE_EXTS(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['~', '.orig', '.bak', '.cfg', '.retry', '.pyc', '.pyo', 'LICENSE', '.md', '.txt', 'secrets.yml', 'vars.yml', 'ssh_private_key']
PLAYBOOK_DIR(env: ANSIBLE_PLAYBOOK_DIR) = /home/user/workspace/ansible-playbooks-2
RETRY_FILES_ENABLED(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = True
RETRY_FILES_SAVE_PATH(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = /tmp
TASK_TIMEOUT(env: ANSIBLE_TASK_TIMEOUT) = 180
VARIABLE_PRECEDENCE(/home/user/workspace/ansible-playbooks-2/ansible.cfg) = ['all_inventory', 'groups_inventory', 'all_plugins_play', 'groups_plugins_play', 'all_plugins_inventory', 'groups_plugins_inventory']

OS / ENVIRONMENT

Ubuntu 20.04.3 LTS (Focal Fossa)

STEPS TO REPRODUCE

- hosts: localhost
  diff: true
  gather_facts: false

  tasks:
    - debug:
        msg: "{{ (some_xml | ansible.utils.from_xml)['root']['leaf'] }}"
      vars:
        some_xml: |
          <?xml version="1.0" encoding="UTF-8"?>
          <root>
              <leaf>
                <attribute>foobar</attribute>
              </leaf>
          </root>

EXPECTED RESULTS

This should print

"msg": {                 
    "attribute": "foobar"
}                        

ACTUAL RESULTS

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'str object' has no attribute 'root'\n\nThe error appears to be in '/home/john/workspace_tagpay/ansible-playbooks-2/foo.tmp.yml': line 6, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - debug:\n      ^ here\n"}

Unit Tests Fail In Container

SUMMARY

I wanted to work on a filter. To start I wanted write a unit test that replicated my failure. I initially was unable to run the current tests. I wanted to execute them in a container and it was failing to gather the necessary requirements (netaddr).

ISSUE TYPE

• Bug Report

COMPONENT NAME

Unit Tests in ansible.utils for ipsubnet

ANSIBLE VERSION

ansible --version
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/bbbb/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  ansible collection location = /home/bbbb/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

Look at the steps to reproduce. In this case it is the head of the repo.

CONFIGURATION

No change.

$ ansible-config dump --only-changed

OS / ENVIRONMENT

Fedora 35 Workstation, fresh installation, using the Podman container runtime and base container image in ansible-test.

STEPS TO REPRODUCE

git clone https://github.com/ansible-collections/ansible.utils.git
cd ansible.utils
ansible-galaxy collection build .
ansible-galaxy collection install -p . ansible-utils-\*.tar.gz
ansible-test units --docker base tests/unit/plugins/filter/test_ipsubnet.py --requirements --verbose

EXPECTED RESULTS

All current tests pass.

ACTUAL RESULTS

A cut of the failures are below. All tests fail for ipsubnet until netaddr is present.

[gw1] [100%] FAILED ../../../ansible/test/lib/ansible_test/_data/::TestIpSubnet::test_ipvsubnet_filter_subnet_with_1st_index 

====================================================== FAILURES ======================================================
_____________________________________ TestIpSubnet.test_ipvsubnet_address_subnet _____________________________________
[gw0] linux -- Python 3.10.0 /usr/bin/python3.10
self = <ansible_collections.ansible.utils.tests.unit.plugins.filter.test_ipsubnet.TestIpSubnet testMethod=test_ipvsubnet_address_subnet>

    def test_ipvsubnet_address_subnet(self):
        """convert address to subnet"""
        args = ["", address, ""]
        result = _ipsubnet(*args)
>       self.assertEqual(result, "192.168.144.5/32")
E       AssertionError: False != '192.168.144.5/32'

Fix documentation errors for plugin documentation

SUMMARY

ansible.utils shows errors on docs.ansible.com that prevent the plugin documentation from displaying for users.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

ANSIBLE VERSION

2.14

textfsm 1.1.3 is broken

SUMMARY

Due to the nature of this module not using strict versioning in requirements.txt which is very bad practice due to issues like this. In short and as the title says, textfsm 1.1.3 has issues and cannot be installed which breaks anything that is using ansible.utils.

ISSUE TYPE

• Bug Report

issue in the textfsm repo google/textfsm#105

Undo DOCUMENTATION changes made in #172

SUMMARY

In both the devel and milestone branches for ansible-core 2.14, a regression was introduced which causes ansible test sanity to fail if the constant DOCUMENTATION is used anywhere, more than once, in a lookup plugin.

A workaround was put in place to get it from globals lowercased.

When the below ansible-core issue is fixed, these changes need to be removed.

This is what was done in each of the ansible.utils lookup plugins.

schema = [v for k, v in globals() if k.lower() == "documentation"]

The is the ansible-core issue that needs to be resolved prior to removing the code from the ansible.utils lookup plugins:
ansible/ansible#77764

This is the PR where the lookup plugin changes went in:
Related #172

ISSUE TYPE

• Cleanup

COMPONENT NAME

lookup plugins

Update network user guide to explain use of cli_parse and validate plugins

SUMMARY

• Update network user guide to point to ansible.utils.cli_parse module as ansible.netcommon.cli_parse is deprecated from ansible.netcommon 2.0.0 release

• Add new docs to explain ansible.utils.validate plugin usage.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

ANSIBLE VERSION

ansible.utils.ipsubnet documentation is incorrect about the query argument

SUMMARY

The documentation at https://docs.ansible.com/ansible/latest/collections/ansible/utils/docsite/filters_ipaddr.html#subnet-manipulation incorrectly states in multiple places that the optional first parameter can be a subnet size. This is incorrect, because the function instead expects the size of the prefix for the subnet. I took me a while to figure out why I can have subnets of size 4 in 192.168.0.0/16 and similar:

- name: asdf
    debug:
    msg: |
        {{ item | ansible.utils.ipsubnet(2) }}
        {{ item | ansible.utils.ipsubnet(16) }}
        {{ item | ansible.utils.ipsubnet(20) }}
        {{ item | ansible.utils.ipsubnet(80) }}
        {{ item | ansible.utils.ipsubnet(96) }}
    loop:
    - '192.168.0.0/16'
    - 'fc00:0:0:0:1::/80'

Output:

TASK [asdf] **************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.168.0.0/16) => {
    "msg": "0\n1\n16\nFalse\nFalse\n"
}
ok: [localhost] => (item=fc00:0:0:0:1::/80) => {
    "msg": "0\n0\n0\n1\n65536\n"
}

ISSUE TYPE

• Documentation Report

COMPONENT NAME

ansible.utils.ipsubnet

update

SUMMARY

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Moving ipaddr from ansible.netcommon to ansible.utils breaks stuff

SUMMARY

When the ipaddr filter moved from ansible.netcommon to ansible.utils it changed behaviour and breaks downstream usage.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipaddr

For more information see ansible-collections/ansible.netcommon#375

Important information for collection maintainers

SUMMARY

Dear maintainers,

This is important for your collections!

• In accordance with the Community decision, we have created the news-for-maintainers repository for announcements of changes impacting collection maintainers (see the examples) instead of Issue 45 that will be closed soon.

  • To keep yourself well-informed and, therefore, things in your collection working, please subscribe to the repository by using the Watch button in the upper right corner on the repository's home page.
  • If you do not want to get notifications about related discussions, please subscribe only to Issues.
  • Please read the brief guidelines on how the repository should be used.
  • Please avoid unnecessary discussions in issues, use the Discussions feature. Every comment posted will notify a lot of folks!

• Also we would like to remind you about the Bullhorn contributor newsletter which has recently started to be released weekly. To learn what it looks like, see the past releases. Please subscribe and talk to the Community via Bullhorn!

• Join us in #ansible-social (for news reporting & chat), #ansible-community (for discussing collection & maintainer topics), and other channels on Matrix/IRC.

• Help the Community and the Steering Committee to make right decisions by taking part in discussing and voting on the Community Topics that impact the whole project and the collections in particular. Your opinion there will be much appreciated!

Thank you!

Move ipaddr documentation from netcommon to utils

SUMMARY

As we are migrating ipaddr filters from netcommon to utils,
We need to update https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters_ipaddr.html documentation.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

ANSIBLE VERSION

Netutils filter_plugins

SUMMARY

netutils is very handy for the average infrastructure engineer. There are a number of functions that would be ideal to create filter_plugins from.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

If the maintainers are interested, I can pick out some components that don not lap with the current collection filter_plugins.

ADDITIONAL INFORMATION

I would be interested to contribute assuming some blessing on the feature idea.

Docs update

Switch "filter" key to name in docs, check other types to do same, request from toshio

Using cli_parse with until directive causes error "'dict object' has no attribute 'parsed'"

SUMMARY

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

$ ansible --version
ansible [core 2.13.1]

COLLECTION VERSION

$ ansible-galaxy collection list ansible.utils                        

# /Users/akira/envs/a6/lib/python3.9/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.6.1  

CONFIGURATION

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/Users/akira/Documents/ansible/ansible.cfg) = False

OS / ENVIRONMENT

Cisco IOS 15.9(3)M3

STEPS TO REPRODUCE

The task is bellow:

    - name: show ip ospf neighbor
      ansible.utils.cli_parse:
        command: show ip ospf neighbor
        parser:
          name: ansible.netcommon.ntc_templates
      register: result_show_ip_ospf_neighbor
      until:
        - (result_show_ip_ospf_neighbor.parsed | length) >= 2
        - result_show_ip_ospf_neighbor.parsed[1].state == 'FULL/DR'

EXPECTED RESULTS

Run show ip ospf neighbor command until the second ospf neighbor is FULL state.

ACTUAL RESULTS

When retrying with until directive, the value of the register variable is not as intended. And an error 'dict object' has no attribute 'parsed' occurs.

TASK [show ip ospf neighbor] **********************************************************************************************************************************
task path: /Users/akira/Documents/ansible/ios_test.yml:10
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
FAILED - RETRYING: [ios01]: show ip ospf neighbor (3 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "parsed": [
        {
            "address": "10.1.1.1",
            "dead_time": "00:00:32",
            "interface": "GigabitEthernet0/3",
            "neighbor_id": "192.168.1.105",
            "priority": "1",
            "state": "FULL/DR"
        }
    ],
    "retries": 4,
    "stdout": "Neighbor ID     Pri   State           Dead Time   Address         Interface\n192.168.1.105     1   FULL/DR         00:00:32    10.1.1.1        GigabitEthernet0/3",
    "stdout_lines": [
        "Neighbor ID     Pri   State           Dead Time   Address         Interface",
        "192.168.1.105     1   FULL/DR         00:00:32    10.1.1.1        GigabitEthernet0/3"
    ]
}
fatal: [ios01]: FAILED! => {
    "msg": "The conditional check '(result_show_ip_ospf_neighbor.parsed | length) >= 2' failed. The error was: error while evaluating conditional ((result_show_ip_ospf_neighbor.parsed | length) >= 2): 'dict object' has no attribute 'parsed'"
}

When displaying the value of the register variable at the time of retry, it contained an error of parameters are mutually exclusive: command|text. But I used only command.

    - name: show ip ospf neighbor
      ansible.utils.cli_parse:
        command: show ip ospf neighbor
        parser:
          name: ansible.netcommon.ntc_templates
      register: result_show_ip_ospf_neighbor
      until:
        - false # for debug

The result is bellow:

...omitted...
FAILED - RETRYING: [ios01]: show ip ospf neighbor (2 retries left).Result was: {
    "attempts": 2,
    "changed": false,
    "errors": [
        "parameters are mutually exclusive: command|text"
    ],
    "msg": "argspec validation failed for cli_parse module plugin",
    "retries": 4
}
...omitted...

ansible.utils jsonschema version conflicts with ansible-lint jsonschema requirements

SUMMARY

When installing the requirements.txt for ansible.utils, jsonschema gets downgraded to 3.2.0, and pip complains that ansible-lint requires jsonschema=>4.5.1. I'm not sure if upgrading jsonschema to 4.5.1 will have an adverse affect on ansible.utils after installing the ansible.utils requirements.txt?

ISSUE TYPE

• Bug Report

COMPONENT NAME

Unsure

ANSIBLE VERSION

ansible [core 2.12.5]
  config file = /Users/bryan.hundven/Projects/p-git01.on24.com/netops/centos_base/ansible.cfg
  configured module search path = ['/Users/bryan.hundven/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bryan.hundven/Projects/venv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/bryan.hundven/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/bryan.hundven/Projects/venv/bin/ansible
  python version = 3.9.12 (main, May  8 2022, 18:05:47) [Clang 13.1.6 (clang-1316.0.21.2)]
  jinja version = 3.1.1
  libyaml = True

COLLECTION VERSION

# /Users/bryan.hundven/.ansible/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.6.1

CONFIGURATION

N/A

OS / ENVIRONMENT

MacOS
Linux
Windows

STEPS TO REPRODUCE

pip3 install -U ansible-lint
ansible-galaxy collection install -f ansible.utils
pip3 install -U ~/.ansible/collections/ansible_collections/ansible/utils/requirements.txt

EXPECTED RESULTS

No package conflicts

ACTUAL RESULTS

Package conflicts

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ansible-lint 6.2.1 requires jsonschema>=4.5.1, but you have jsonschema 3.2.0 which is incompatible.

Extend remove_keys and maybe others to take ansible.utils.to_paths generated output as arguments for target

SUMMARY

Extend at least remove_keys from Recursive filter plugins remove_keys | replace_keys | keep_keys to support inputs for target which have the form what ansible.utils.to_paths generates.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• ansible.utils.remove_keys
• ansible.utils.to_paths

ADDITIONAL INFORMATION

If data structure is more complex and has a key which is matched by target on multiple levels you might want to be more specific which key you really want to remove.

    ##example.yaml
    a:
      b:
        c:
          e:
          - 0
          - 1
        d:
          e:
          - True
          - False

    ##Playbook
    vars_files:
      - "example.yaml"
    tasks:
      - ansible.builtin.set_fact:
          paths: "{{ a|ansible.utils.to_paths }}"

      # TASK [ansible.builtin.set_fact] ********************************************
      # ok: [localhost] => changed=false
      #   ansible_facts:
      #     paths:
      #       b.c.e[0]: 0
      #       b.c.e[1]: 1
      #       b.d.e[0]: True
      #       b.d.e[1]: False

      - debug:
          msg:  "{{ a|ansible.utils.remove_keys(target=['^b.d.e'], matching_parameter= 'regex') }}"

      # TASK [debug] *****************************************
      # ok: [localhost] => {
      #     "msg": {
      #         "b": {
      #             "c": {
      #                 "e": [
      #                     0,
      #                     1
      #                 ]
      #             }
      #         }
      #     }
      # }

ansible-lint issues for utils

SUMMARY

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

:warning: Testing with python3.8 fails

On 31 Mar 2023, ansible-navigator 3.0.0 was released which dropped support for python 3.8.

This impacts the integration testing matrix that currently includes ansible 2.9 and python 3.8.

An additional workflow was added to work around this issue until such time the decision is made to change the main workflow.

https://github.com/ansible-network/github_actions/blob/main/.github/workflows/integration_simple_39.yml

The above workflow can be used as a workaround but does not run integration tests with ansible 2.9 or python 3.8 which may or may not be required.

Convert ipv6 addresses to compressed, expanded and special forms

SUMMARY

I was surprised that there is no filter for getting expanded ipv6 addresses in ansible.utils collection.

I would like to get something as:

# {{ '::1' | ansible.utils.ipv6('expand') }}
'0000:0000:0000:0000:0000:0000:0000:0001'

And for ipv6 addresses compression:

# {{ '0000:0000:0000:0000:0000:0000:0000:0001' | ansible.utils.ipv6('compress') }}
'::1'

Also I think it would be great to have filter that converts ipv6 address to x509 notation:

# {{ '::1' | ansible.utils.ipv6('x509') }}
'0:0:0:0:0:0:0:1'

When the subjectAltName extension contains an iPAddress, the address MUST be stored in the octet string in "network byte order", as specified in [RFC791]. The least significant bit (LSB) of each octet is the LSB of the corresponding byte in the network address. For IP version 4, as specified in [RFC791], the octet string MUST contain exactly four octets. For IP version 6, as specified in [RFC2460], the octet string MUST contain exactly sixteen octets.

RFC5820

IPv6 addresses must be in the form x:x:x:x:x:x:x:x, exactly 8 groups of one to four hexadecimal digits separated by colons. The compressed zero form using "::" is not accepted.

X.509 Extensions Parameter

ISSUE TYPE

• Feature Idea

COMPONENT NAME

I think it would be like ansible.utils.ip4_hex filter or ansible.utils.ipwrap wrapper.

It can be also implemented as separate module, e.g. ansible.utils.ip6_form:

ansible.utils.ip6_form('compress')
ansible.utils.ip6_form('expand')
ansible.utils.ip6_form('x509')

ADDITIONAL INFORMATION

In general, for processing ipv6 addresses in different forms with other filters. In particular, x509 notation filter for ipv6 will be used for self-signed certificates. Getting expanded ipv6 using regular expressions is a very complicated and unreliable way.

Example of using:

- community.crypto.openssl_csr_pipe:
    privatekey_path: "{{ privkey.filename }}"
    common_name: "{{ ansible_fqdn }}"
    subject_alt_name:
      - "IP:{{ ansible_default_ipv4.address }}"
      - "IP:{{ ansible_default_ipv6.address|ansible.utils.ip6_form('x509') }}"

Jinja2 3.1 Breaks filter

SUMMARY

Version 3.1 of Jinja2 removed deprecated code, including environmentfilter: pallets/jinja#1544

Attempting to execute Ansible operations that use filter with Jinja2 >= 3.1 results in:

[WARNING]: Skipping plugin (/opt/python3/lib/python3.10/site-packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot import name 'environmentfilter' from 'jinja2.filters'  (/opt/python3/lib/python3.10/site-packages/jinja2/filters.py)

This issue appears to have been addressed in core Ansible here: ansible/ansible#74667

ISSUE TYPE

• Bug Report

COMPONENT NAME

filter

ANSIBLE VERSION

ansible [core 2.12.3]
  config file = None
  configured module search path = ['/home/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/python3/lib/python3.10/site-packages/ansible
  ansible collection location = /home/jenkins/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, Mar 24 2022, 16:14:08) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
  jinja version = 3.1.1
  libyaml = False

COLLECTION VERSION

# /opt/python3/lib/python3.10/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 4.6.0

CONFIGURATION

<no output>

OS / ENVIRONMENT

Linux (EL7)
Python 3.10.4 compiled from source
Jinja2 3.1.1
Ansible 5.5.0 (according to pip3 freeze)
Ansible Core 2.12.3 (according to pip3 freeze)

Same results with Ansible 2.10.7 and Ansible Base 2.10.9 using Python 3.9.10.

STEPS TO REPRODUCE

Install Jinja2 version 3.1.x
Execute playbook

EXPECTED RESULTS

No Python import error

ACTUAL RESULTS

[WARNING]: Skipping plugin (/opt/python3/lib/python3.10/site-packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot import name 'environmentfilter' from 'jinja2.filters'  (/opt/python3/lib/python3.10/site-packages/jinja2/filters.py)

to_xml filter always includes the xml tags

When using to_xml it always returns the xml tags which are only needed once in a files.

Here is an example showing the problem.

server.xml.j2

<?xml version="1.0" encoding="UTF-8"?>

<!-- {{ ansible_managed }} -->

<!-- Stop! Contact your administrator if any changes to this file are needed!! -->

<Server port="{{ instance.server.shutdown_port }}">

	{% for listener in (instance.server.listensers | default(tomcat_server_listeners)) %}
	{{ listener | ansible.utils.to_xml }}
	{% endfor %}

	{% for gnr in (instance.server.gnrs | default(tomcat_server_gnrs)) %}
	{{ gnr | ansible.utils.to_xml }}
	{% endfor %}

	<Service name="Catalina">

		{% for executor in (instance.server.executors | default(tomcat_server_executors)) %}
		{{ executor | ansible.utils.to_xml }}
		{% endfor %}

		{% for connector in instance.server.connectors %}
		{{ connector | ansible.utils.to_xml }}
		{% endfor %}

		<Engine name="Catalina" defaultHost="{{ instance.server.default_host | default(tomcat_server_default_host) }}">

			{% for realm in (instance.server.realms | default(tomcat_server_realms)) %}
			{{ realm | ansible.utils.to_xml }}
			{% endfor %}

			{% for host in (instance.server.hosts | default(tomcat_server_hosts)) %}
			{{ host | ansible.utils.to_xml }}
			{% endfor %}

		</Engine>

	</Service>

</Server>

server.xml

<?xml version="1.0" encoding="UTF-8"?>

<!-- This file is managed by Ansible! All changes will be lost!! -->

<!-- Stop! Contact your administrator if any changes to this file are needed!! -->

<Server port="8081">

                <?xml version="1.0" encoding="utf-8"?>
<Listener className="org.apache.catalina.startup.VersionLoggerListener"></Listener>
                <?xml version="1.0" encoding="utf-8"?>
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"></Listener>
                <?xml version="1.0" encoding="utf-8"?>
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"></Listener>
                <?xml version="1.0" encoding="utf-8"?>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"></Listener>
                <?xml version="1.0" encoding="utf-8"?>
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"></Listener>

                <?xml version="1.0" encoding="utf-8"?>
<GlobalNamingResources>
        <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml"></Resource>
</GlobalNamingResources>

        <Service name="Catalina">


                                <?xml version="1.0" encoding="utf-8"?>
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"></Connector>

                <Engine name="Catalina" defaultHost="localhost">

                                                <?xml version="1.0" encoding="utf-8"?>
<Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">
                <CredentialHandler className="org.apache.catalina.realm.SecretKeyCredentialHandler" algorithm="PBKDF2WithHmacSHA512" iterations="100000" keyLength="256" saltLength="16"></CredentialHandler>
        </Realm>
</Realm>

                                                <?xml version="1.0" encoding="utf-8"?>
<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern='%h %l %u %t "%r" %s %b'></Valve>
</Host>

                </Engine>

        </Service>

</Server>

Indentation issues asside to_xml when used as a filter should not return <?xml version="1.0" encoding="UTF-8"?> or should have an option to disable it.

Collection CI related actions needed

SUMMARY

Relates to ansible-collections/news-for-maintainers#24.

It's been detected that this repository contains ignore-2.14.txt file but not ignore-2.15.txt file.

Actions needed:

• Copy tests/sanity/ignore-2.14.txt to tests/sanity/ignore-2.15.txt otherwise the CI will get failing.
• If it's not done yet, add the stable-2.15 branch to your CI workflow files. If you use GitHub actions, see the content of .github/workflows or, if you use Azure Pipelines, community.postgresql/.azure-pipelines/azure-pipelines.yml. If not added, this will violate the Collection Requirements for collections included in the Ansible package.
• Subscribe to and track the news-for-maintainers repository if you're not subscribed yet.
• If it's not relevant for this repo, just close the issue.

To read more about the topic, see the Ignores guide.

To read more about the context, see the announcement.

Thank you!

Consider using true/false for all booleans in docs

Based on the community decision to use true/false for boolean values in documentation and examples, we ask that you evaluate booleans in this collection and consider changing any that do not use true/false (lowercase).

See documentation block format for more info (specifically, option defaults).

If you have already implemented this or decide not to, feel free to close this issue.

---

P.S. This is auto-generated issue, please raise any concerns here

sub_plugins/fact_diff/native.py does not process dict objects properly

SUMMARY

When passing facts to fact_diff, if you add ignore_lines the process treats the dict like a list and adds its keys as strings (removing values).

Specifically these lines: (60-69):

self._before = [ line for line in self._before if not any(regex.match(str(line)) for regex in self._skip_lines) ] self._after = [ line for line in self._after if not any(regex.match(str(line)) for regex in self._skip_lines) ]

ISSUE TYPE

• Bug Report

COMPONENT NAME

sub_plugins/fact_diff/native.py

ANSIBLE VERSION

ansible [core 2.13.1]

STEPS TO REPRODUCE

- name: Show the difference in json format
  ansible.utils.fact_diff:
    before: "{{ before }}"
    after: "{{ after }}"
    plugin:
      name: ansible.utils.native
      vars:
         skip_lines:
            - '^example.*'

EXPECTED RESULTS

Should properly handle dicts, instead it will break if skip_lines is included (causing two different facts to appear the same).
As it will only compare key names and not their values.

validate does not resolve external references

SUMMARY

The default RefResolver in jsonschema correctly loads references that the validate module does not

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.utils.validate

ANSIBLE VERSION

ansible [core 2.14.5]
  config file = /home/user/ansible_home/ansible.cfg
  configured module search path = ['/home/user/ansible_home/plugins/modules']
  ansible python module location = /home/user/venv/lib64/python3.11/site-packages/ansible
  ansible collection location = /home/user/ansible_dev/collections:/home/user/ansible_home/collections:/home/user/playbooks/collections
  executable location = /home/user/venv/bin/ansible
  python version = 3.11.3 (main, Apr  5 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] (/home/user/venv/bin/python3)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/user/ansible_home/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.9.0

# /home/user/venv/lib/python3.11/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.9.0

# /home/user/venv/lib64/python3.11/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.9.0

CONFIGURATION

ANSIBLE_HOME(env: ANSIBLE_HOME) = /home/user/ansible_home
BECOME_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/become']
COLLECTIONS_PATHS(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_dev/collections', '/home/user/ansible_home/collections', '/home/user/playbooks/collections']
CONFIG_FILE() = /home/user/ansible_home/ansible.cfg
DEFAULT_ACTION_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/action']
DEFAULT_CACHE_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/cache']
DEFAULT_CALLBACK_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/callback']
DEFAULT_CONNECTION_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/connection']
DEFAULT_FILTER_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/filter']
DEFAULT_HOST_LIST(/home/user/ansible_home/ansible.cfg) = ['/home/user/code/netbox_inventory']
DEFAULT_INVENTORY_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/inventory']
DEFAULT_LOOKUP_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/lookup']
DEFAULT_MODULE_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/modules']
DEFAULT_MODULE_UTILS_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/module_utils']
DEFAULT_ROLES_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/roles']
DEFAULT_STRATEGY_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/strategy']
DEFAULT_TERMINAL_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/terminal']
DEFAULT_TEST_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/test']
DEFAULT_VARS_PLUGIN_PATH(/home/user/ansible_home/ansible.cfg) = ['/home/user/ansible_home/plugins/vars']
INVENTORY_ENABLED(/home/user/ansible_home/ansible.cfg) = ['host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'constructed']
PERSISTENT_COMMAND_TIMEOUT(/home/user/ansible_home/ansible.cfg) = 120
RETRY_FILES_ENABLED(/home/user/ansible_home/ansible.cfg) = False
RETRY_FILES_SAVE_PATH(/home/user/ansible_home/ansible.cfg) = /home/user/.ansible-retry

OS / ENVIRONMENT

Fedora Remix 37 on WSL

STEPS TO REPRODUCE

https://gist.github.com/bk2zsto/f9df69ed86a5b0558b103bfaab3a63d1

EXPECTED RESULTS

validate loads external references and/or errors when they are not resolved

ACTUAL RESULTS

validate ignores external references

$ python /tmp/debug.py
Validation of 192.0.2.0 failed with combined
Validation of 192.0.2.0 failed with referring

$ ansible-playbook /tmp/debug.yml

PLAY [localhost] *******************************************************************************************************

TASK [setup test_data] *************************************************************************************************
ok: [localhost]

TASK [validate with combined schema] ***********************************************************************************
ok: [localhost] => (item=cidr)
failed: [localhost] (item=not_cidr) => {"ansible_loop_var": "item", "changed": false, "errors": [{"data_path": "network", "expected": "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}", "found": "192.0.2.0", "json_path": "$.network", "message": "'192.0.2.0' does not match '^[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}/[0-9]{1,2}'", "relative_schema": {"$comment": "CIDR, IPv4-only", "pattern": "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}", "type": "string"}, "schema_path": "properties.network.pattern", "validator": "pattern"}], "item": {"name": "not_cidr", "network": "192.0.2.0"}, "msg": "Validation errors were found.\nAt 'properties.network.pattern' '192.0.2.0' does not match '^[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}/[0-9]{1,2}'. "}
...ignoring

TASK [validate with referring schema] **********************************************************************************
ok: [localhost] => (item=cidr)
ok: [localhost] => (item=not_cidr) <==============================

PLAY RECAP *************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1

[rfe] Add recursive filter plugins for keep keys, replace keys, and remove keys

Please tag your releases

SUMMARY

It looks like you are actively working on this collection and release new versions on galaxy. However, you don't tag them in this repository, which is a requirement for collections that are part of the community package.

Please tag your release, otherwise we will consider removing ansible.utils from ansible.

Not ot put too fine a point on it, but it looks like an easy thing to fix where we don't have to discuss much ;-)

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Confusing wording in error message (netaddr)

SUMMARY

(This is a fairly trivial fix; reported in the main issue list, but I was redirected here)

When using a playbook which uses eg ansible.utils.ipv4('public'), I see an error message:

The ipv4 filter requires python's netaddr be installed on the ansible controller

Trying to parse this, I ended up perplexed – ‘in what sense does “python” have a network address; WTF?!’ – and it took a web search to realise that it's referring to a Python netaddr package. Fairly obvious in retrospect, of course (and I have in fact used that package before), but it took me a good 10 minutes to work out what the error message was reporting, rather than giving an immediate 'aha!’.

I suggest changing the wording to

The ipv4 filter requires that the Python netaddr package be installed on the ansible controller

(One for the intern...?)

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible-playbook

ANSIBLE VERSION

ansible [core 2.14.3]
  config file = /Users/Shared/common/checkouts/itm/ansible/ansible.cfg
  configured module search path = ['/Users/norman/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/Shared/common/checkouts/itm/ansible/ansible-venv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/norman/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/Shared/common/checkouts/itm/ansible/ansible-venv/bin/ansible
  python version = 3.9.6 (default, Oct 18 2022, 12:41:40) [Clang 14.0.0 (clang-1400.0.29.202)] (/Users/Shared/common/checkouts/itm/ansible/ansible-venv/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /Users/Shared/common/checkouts/itm/ansible/ansible-venv/lib/python3.9/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.9.0  

CONFIGURATION

CONFIG_FILE() = /Users/Shared/common/checkouts/itm/ansible/ansible.cfg
DEFAULT_HOST_LIST(/Users/Shared/common/checkouts/itm/ansible/ansible.cfg) = ['/Users/Shared/common/checkouts/itm/ansible/inventory/hosts']
DEFAULT_MANAGED_STR(/Users/Shared/common/checkouts/itm/ansible/ansible.cfg) = Ansible managed: modified on %Y-%m-%d %H:%M:%S
DEFAULT_ROLES_PATH(/Users/Shared/common/checkouts/itm/ansible/ansible.cfg) = ['/Users/Shared/common/checkouts/itm/ansible/roles']
DEFAULT_VAULT_PASSWORD_FILE(/Users/Shared/common/checkouts/itm/ansible/ansible.cfg) = /Users/Shared/common/checkouts/itm/ansible/.vault_pass
RETRY_FILES_ENABLED(/Users/Shared/common/checkouts/itm/ansible/ansible.cfg) = False

OS / ENVIRONMENT

macOS 13.2.1

STEPS TO REPRODUCE

Run any playbook which exploits features of the Python netaddr package, while that package isn't installed on the controller node

EXPECTED RESULTS

I expected an easy-to-parse/non-confusing error message

ACTUAL RESULTS

I got a confusing (clear only in retrospect) error message

Add support for the validation of formats to the jsonschema validator

SUMMARY

Add support for using the following format options: https://python-jsonschema.readthedocs.io/en/stable/validate/#validating-formats

"The format keyword allows for basic semantic validation on certain kinds of string values that are commonly used."

@ganeshrn has provided some preliminary working modifications to the plugin for me to test.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ansible.utils.validation -> jsonschema.py

ADDITIONAL INFORMATION

The python library used for the ansible.utils.validation module as it pertains to jsonschema, has additional support for using jsonschema's format keywords, which help validate common formats like email, IP's, URL's, etc without specifying regex patterns

Ansible Contributor Summit. Tuesday, April 12, 2022.

Ansible Contributor Summit

We are happy to announce that the registration for the Ansible Contributor Summit is open!

Why

• This is a great opportunity for interested people to meet, discuss related topics, share their stories and opinions, get the latest important updates and just to hang out together.

• There will be different announcements & presentations by Community, Core, Cloud, Network, and other teams.

• Current contributors will be happy to share their stories and experience with newcomers.

• There will be links to interactive self-passed instruqt scenarios shared during the event that help newcomers learn different aspects of development.

Where/when

Online on Matrix and Youtube. Tuesday, April 12, 2022, 12:00 - 20:00 UTC.

How to join

• Add the event to your calendar. Use the ical URL (for example, in Google Calendar "Add other calendars" > "Import from URL") instead of importing the .ics file so that any updates to the event will be reflected in your calendar.

• Check out the Summit page:

  • Add you name to attendees.
  • Suggest summit topics that would be interesting to you to hear about.
  • Vote on and propose changes to topics suggested by others.
  • If you want to be a presenter, please contact the Ansible Community team via [email protected].

We are looking forward to seeing you!:)

Make _validate_args(plugin, doc, params) method as utility function for utils collection

SUMMARY

In the current utils implementation this method: https://github.com/ansible-collections/ansible.utils/blob/5fe1d93eb0f07051fc119bf24ef407cad559ea1b/plugins/plugin_utils/base/ipaddress_utils.py#[…]6 is restricted to ipaddress utility. we can make this generic so that it can be used across utils collection.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ipaddress_utils.py

ADDITIONAL INFORMATION

Clarify which version of black is required for formatting

SUMMARY

The README.md file (https://github.com/ansible-collections/ansible.utils/blame/main/README.md#L140) states:

Use of the latest version of black is required for formatting (black -l79)

However:

1. As of 2/6/2022, the latest version of black is 22.1.0. The CI uses black an older version 19.3b0.
2. Even though black does have stability guarantees, some changes are expected across versions. Here is the black stability policy: https://black.readthedocs.io/en/latest/the_black_code_style/index.html#stability-policy
3. The local formatting by black -l79 in a local workspace does not match the formatting by black in the zuul CI.

It might be useful to run the zuul CI with the --diff option instead of the --check option. That would make it easier to identify formatting differences between CI and local workspace.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

black (python formatting)

ANSIBLE VERSION

ADDITIONAL INFORMATION

When the ansible-tox-linters job runs in CI, the following output may be generated. However, we can't tell what would change. This makes it challenging to identify specific differences between black running in CI vs local workspace.

would reformat /home/zuul/src/github.com/ansible-collections/ansible.utils/tests/unit/plugins/filter/test_ipsubnet.py
would reformat /home/zuul/src/github.com/ansible-collections/ansible.utils/tests/unit/plugins/sub_plugins/validate/test_config.py
All done! 💥 💔 💥
2 files would be reformatted, 163 files would be left unchanged.
ERROR: InvocationError for command /home/zuul/src/github.com/ansible-collections/ansible.utils/.tox/linters/bin/black -v -l79 --check . (exited with code 1)

I'm trying to figure out if the issue is caused by the version of black or by configuration. The formatting diffs are for function documentation. Example:
https://github.com/ansible-collections/ansible.utils/pull/131/files#diff-21f3ea4214265402c6b1447e96218830431ff1090545d358a4df63d425e095acL692

-    """ Check if string is a HW/MAC address and filter it """
+    """Check if string is a HW/MAC address and filter it"""

remove_keys filter remove empty string from list

SUMMARY

Hi, I'm not sure if it is the expected behavior:
remove_keys applied on [ "" ] always return [ ] in place of [ "" ]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.utils.remove_keys

ANSIBLE VERSION

ansible [core 2.14.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/ubuntu/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Same problem with:

ansible-playbook [core 2.12.3.post0]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.12 (default, Sep 21 2021, 00:10:52) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True

COLLECTION VERSION

# /home/ubuntu/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
ansible.utils     2.9.0  
community.general 6.5.0  
kubernetes.core   2.4.0  

CONFIGURATION

CONFIG_FILE() = /etc/ansible/ansible.cfg

OS / ENVIRONMENT

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Not environment dependent.

STEPS TO REPRODUCE

File test.yaml:

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2023-03-08T10:06:46Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: extension-apiserver-authentication-reader
    namespace: kube-system
    resourceVersion: "184"
    uid: 99df70e6-6d4f-442a-8b95-bc4b2138b589
  rules:
  - apiGroups:
    - ""
    resourceNames:
    - extension-apiserver-authentication
    resources:
    - configmaps
    verbs:
    - get
    - list
    - watch
kind: List
metadata:
  resourceVersion: ""

Playbook to run:

---
- name: playbook test remove_keys
  hosts: localhost
  tasks:
    - name: debug file content without remove_keys
      ansible.builtin.debug:
        msg: "{{ lookup('file','test.yaml') | from_yaml }}"

    - name: debug file content with remove_keys
      ansible.builtin.debug:
        msg: "{{ lookup('file','test.yaml') | from_yaml | ansible.utils.remove_keys(target=['creationTimestamp','resourceVersion']) }}"

EXPECTED RESULTS

remove_keys should not remove the empty string from the apiGroups list:

TASK [debug file content with remove_keys] **************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": {
        "apiVersion": "v1",
        "items": [
            {
                "apiVersion": "rbac.authorization.k8s.io/v1",
                "kind": "Role",
                "metadata": {
                    "annotations": {
                        "rbac.authorization.kubernetes.io/autoupdate": "true"
                    },
                    "labels": {
                        "kubernetes.io/bootstrapping": "rbac-defaults"
                    },
                    "name": "extension-apiserver-authentication-reader",
                    "namespace": "kube-system",
                    "uid": "99df70e6-6d4f-442a-8b95-bc4b2138b589"
                },
                "rules": [
                    {
                        "apiGroups": [
                            ""
                        ],
                        "resourceNames": [
                            "extension-apiserver-authentication"
                        ],
                        "resources": [
                            "configmaps"
                        ],
                        "verbs": [
                            "get",
                            "list",
                            "watch"
                        ]
                    }
                ]
            }
        ],
        "kind": "List",
        "metadata": {}
    }
}

ACTUAL RESULTS

remove_keys remove the empty string from the apiGroups list (the ouput from the filter from_yaml is OK).
Same result with matching_parameter=regex

TASK [debug file content with remove_keys] **************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": {
        "apiVersion": "v1",
        "items": [
            {
                "apiVersion": "rbac.authorization.k8s.io/v1",
                "kind": "Role",
                "metadata": {
                    "annotations": {
                        "rbac.authorization.kubernetes.io/autoupdate": "true"
                    },
                    "labels": {
                        "kubernetes.io/bootstrapping": "rbac-defaults"
                    },
                    "name": "extension-apiserver-authentication-reader",
                    "namespace": "kube-system",
                    "uid": "99df70e6-6d4f-442a-8b95-bc4b2138b589"
                },
                "rules": [
                    {
                        "apiGroups": [],
                        "resourceNames": [
                            "extension-apiserver-authentication"
                        ],
                        "resources": [
                            "configmaps"
                        ],
                        "verbs": [
                            "get",
                            "list",
                            "watch"
                        ]
                    }
                ]
            }
        ],
        "kind": "List",
        "metadata": {}
    }
}

network_in_network test fails for /31 IPv4 and /127 IPv6 networks

SUMMARY

When using /31 IPv4 and /127 IPv6 networks for point-2-point connections the network_in_network filter fails when trying to validate the second IP address in the network.

Trying the same with network_in_usable works because this function seems to have special handling for a network size of two addresses. As network_in_usable is by definition a subset of network_in_network the latter should also succeed when using this kind of networks.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipaddr

ANSIBLE VERSION

ansible 2.10.16
  config file = /Users/CENSORED/test/ansible.cfg
  configured module search path = ['/Users/CENSORED/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/CENSORED/test/venv/lib/python3.9/site-packages/ansible
  executable location = /Users/CENSORED/test/venv/bin/ansible
  python version = 3.9.9 (main, Nov 21 2021, 03:16:13) [Clang 13.0.0 (clang-1300.0.29.3)]

COLLECTION VERSION

# /Users/CENSORED/test/collections/ansible_collections
Collection        Version
----------------- -------
ansible.netcommon 2.5.0

CONFIGURATION

Not dependent on config

OS / ENVIRONMENT

MacOS 12.1

STEPS TO REPRODUCE

- hosts: localhost
  tasks:
    - name: Test filter
      debug:
        msg:
          - "network_in_network IPv4: {{ '192.168.1.0/31'|ansible.netcommon.network_in_network('192.168.1.1') }}"
          - "network_in_network IPv6: {{ '2001:db8::/127'|ansible.netcommon.network_in_network('2001:db8::1') }}"

EXPECTED RESULTS

This should return True both times because the last IP address in a /31 is valid.

ACTUAL RESULTS

The result is False

PLAY [localhost] *************************************************************************************************************

TASK [Test ipaddr filters] ***************************************************************************************************
ok: [localhost] => {}

MSG:

['network_in_network IPv4: False', 'network_in_network IPv6: False']

PLAY RECAP *******************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

ansible.utils filters should raise AnsibleFilterError

SUMMARY

With the changes from PR149 ansible.utils raises the base AnsibleError for invalid input values instead of AnsibleFilterError. Where possible the more specific error should be raised.

An example of this can be seen with ansible-lint where the AnsibleError triggers the jinja[invalid] linting rule. If the filters would use AnsibleFilterError it would see that the problem lies with a filter, not with the template itself.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.utils filters

ANSIBLE VERSION

ansible [core 2.13.4]
  config file = None
  configured module search path = ['/home/harm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/harm/.virtualenvs/ansible-tmp/lib/python3.10/site-packages/ansible
  ansible collection location = /home/harm/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/harm/.virtualenvs/ansible-tmp/bin/ansible
  python version = 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
  jinja version = 3.1.2
  libyaml = True

ansible-lint 6.7.0 using ansible 2.13.4

COLLECTION VERSION

Collection    Version
------------- -------
ansible.utils 2.6.1 

OS / ENVIRONMENT

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

STEPS TO REPRODUCE

$ pip install ansible ansible-lint netaddr
$ ansible-galaxy collection install ansible.utils

```yaml
---
- name: Play
  hosts: localhost
  gather_facts: false

  tasks:
    # This PASS
    - name: Debug
      ansible.builtin.debug:
        msg: "{{ '127.0.0.1' | ansible.utils.ipaddr }}"

    # This FAILS
    - name: Debug
      ansible.builtin.debug:
        msg: "{{ somevar | ansible.utils.ipaddr }}"
      vars:
        somevar: '127.0.0.1'
...

EXPECTED RESULTS

$ ansible-lint play.yml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: play.yml

ACTUAL RESULTS

$ ansible-lint play.yml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: play.yml
WARNING  Listing 1 violation(s) that are fatal
jinja: Unrecognized type <<class 'ansible.template.AnsibleUndefined'>> for ipaddr filter <value> (jinja[invalid])
play.yml:13 Task/Handler: Debug

You can skip specific rules or tags by adding them to your configuration file:
# .config/ansible-lint.yml
warn_list:  # or 'skip_list' to silence them completely
  - jinja[invalid]  # Rule that looks inside jinja2 templates.

Finished with 1 failure(s), 0 warning(s) on 1 files.

Failure when using ipaddr after updating to Ansible community 5.6.0

SUMMARY

After updating to Ansible community 5.6.0, the ipaddr filter gives an error in one of our playbooks. The error did not occur in 5.5.0 and, per semver rules, 5.6.0 should be backwards compatible.

Originally reported in ansible/ansible#77480; was told to submit here instead.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipaddr

ANSIBLE VERSION

$ ansible --version
ansible [core 2.12.4]
  config file = None
  configured module search path = ['/Users/riendeau/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/riendeau/venvs/ansible5x/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/riendeau/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/riendeau/venvs/ansible5x/bin/ansible
  python version = 3.9.10 (v3.9.10:f2f3f53782, Jan 13 2022, 17:02:14) [Clang 6.0 (clang-600.0.57)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

$ ansible-galaxy collection list ansible.utils

# /Users/riendeau/venvs/ansible5x/lib/python3.9/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.5.2

CONFIGURATION

$ ansible-config dump --only-changed
COLOR_DEBUG(env: ANSIBLE_COLOR_DEBUG) = bright gray

OS / ENVIRONMENT

Mac OS

STEPS TO REPRODUCE

---
- name: Test playbook
  hosts: localhost
  tasks:
  - name: Set fact 1
    set_fact:
      ip1: "172.20.0.1"
  - name: Set fact 2
    set_fact:
      ip2: "{{ ((ip1 | ipaddr('int')) + 6) | ipaddr }}"
  - debug: msg="ip2={{ ip2 }}"

EXPECTED RESULTS

Expected behavior seen in Ansible community 5.5.0:

$ ansible-playbook ~/Desktop/ipaddr-filter-test.yml 
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Test playbook] *********************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set fact 1] ************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set fact 2] ************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] *****************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ip2=172.20.0.7"
}

PLAY RECAP *******************************************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 

ACTUAL RESULTS

$ ansible-playbook ~/Desktop/ipaddr-filter-test.yml 
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Test playbook] *********************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set fact 1] ************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set fact 2] ************************************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
fatal: [localhost]: FAILED! => {"msg": "Unrecognized type <<class 'int'>> for ipaddr filter <value>"}

PLAY RECAP *******************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

fact_diff filter or lookup

SUMMARY

The fact_diff module is great for comparing facts, but I often use the asset module to compare vars. It would be useful to be able to use a fact_diff type function in a fail message of an assert task.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

fact_diff

ADDITIONAL INFORMATION

Add new plugin resource_list_compare that generate the final resource list after comparing base and provided resource list and bangs

SUMMARY

Add new plugin resource_list_compare that generates the final resource list after comparing base and provided resource list and bangs.

EXAMPLE:

- set_fact:
    resources: resources:
        - '!all'
        - 'interfaces'
        - 'l2_interfaces'
        - 'l3_interfaces'
 - set_fact
      network_resource_list:
          modules:
              - 'acl_interfaces'
              - 'acls'
              - 'bgp_address_family'
              - 'bgp_global'
              - ' interfaces'
              - 'l2_interfaces'
              - 'l3_interfaces'
              - 'lacp',
              - 'lacp_interfaces'
              - 'lag_interfaces'
              - 'lldp_global',
              - 'lldp_interfaces'
              - 'logging_global'
              - 'ospf_interfaces'
              - 'ospfv2'
              - 'ospfv3'
        "prefix_lists",
        "route_maps",
        "static_routes",
        "vlans"

- name: set facts for data and criteria
  set_fact:
    final_network_resources: "{{ network_resources_list['modules']|resource_list_compare(resources) }}"

RESULT:

final_network_resources=['interfaces', 'l2_interfaces', 'l3_interfaces']

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

IP address filter to get the last or first X bits of an IP address

SUMMARY

Please implement a new filter that returns either the first or last X bits of an IP address.
Example:
I have an IP address like 1234:4321:abcd:dcba::17. Now I would like to just get the last 80 bits of the address for my firewall rule.
1234:4321:abcd:dcba::17 > expanded: 1234:4321:abcd:dcba:0000:0000:0000:17 -> filtered: dcba:0:0:0:17

ISSUE TYPE

• Feature Request

COMPONENT NAME

ansible.utils.ipaddr

ADDITIONAL INFORMATION

In my case I need to derive a portion of an IP address to make a firewall rule from it.
I don't want to use the IP address itself, as it wouldn't match (Complicated IPv6 story).
So I just want to have the network portion and the interface identifier.

1234:4321:abcd:dcba::17 | ansible.utils.ipcut(last, 80)
dcba:0:0:0:17

1234:4321:abcd:dcba::17 | ansible.utils.ipcut(first, 64)
1234:4321:abcd:dcba

Edit:
As I need that for nftables, the result needs to be in expanded format.

subnet_of_test does not work with IPv6 addresses

SUMMARY

subnet_of_test does not work with IPv6 addresses

ISSUE TYPE

• Bug Report

COMPONENT NAME

subnet_of_test

ANSIBLE VERSION

ansible [core 2.14.3]
  config file = /workspaces/ansible.cfg
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /workspaces/.pyenv/versions/3.10.4/envs/ansible.venv/lib/python3.10/site-packages/ansible
  ansible collection location = /workspaces/collections
  executable location = /workspaces/.pyenv/versions/ansible.venv/bin/ansible
  python version = 3.10.4 (main, Mar  2 2023, 17:18:28) [GCC 10.2.1 20210110] (/workspaces/.pyenv/versions/3.10.4/envs/ansible.venv/bin/python3.10)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /workspaces/collections/ansible_collections
Collection    Version   
------------- ----------
ansible.utils 2.10.0-dev

# /workspaces/.pyenv/versions/3.10.4/envs/ansible.venv/lib/python3.10/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.9.0  

CONFIGURATION

CACHE_PLUGIN(/workspaces/ansible.cfg) = ansible.builtin.jsonfile
CACHE_PLUGIN_CONNECTION(/workspaces/ansible.cfg) = .cache.facts.json
CACHE_PLUGIN_TIMEOUT(/workspaces/ansible.cfg) = 86400000
COLLECTIONS_PATHS(/workspaces/ansible.cfg) = ['/workspaces/collections']
CONFIG_FILE() = /workspaces/ansible.cfg
DEFAULT_BECOME(/workspaces/ansible.cfg) = True
DEFAULT_BECOME_FLAGS(/workspaces/ansible.cfg) = -l
DEFAULT_BECOME_METHOD(/workspaces/ansible.cfg) = su
DEFAULT_FORCE_HANDLERS(/workspaces/ansible.cfg) = True
DEFAULT_FORKS(/workspaces/ansible.cfg) = 50
DEFAULT_HOST_LIST(/workspaces/ansible.cfg) = ['/workspaces/inventory.ini']
DEFAULT_JINJA2_NATIVE(/workspaces/ansible.cfg) = True
DEFAULT_MANAGED_STR(/workspaces/ansible.cfg) = This file is managed by ansible - local changes will be lost
DEFAULT_REMOTE_USER(/workspaces/ansible.cfg) = root
DEFAULT_TRANSPORT(/workspaces/ansible.cfg) = ssh
DEFAULT_VAULT_PASSWORD_FILE(/workspaces/ansible.cfg) = /workspaces/.vpass
DEPRECATION_WARNINGS(/workspaces/ansible.cfg) = True
INVENTORY_ENABLED(/workspaces/ansible.cfg) = ['ini']
SYSTEM_WARNINGS(/workspaces/ansible.cfg) = True

OS / ENVIRONMENT

Debian 11 in dev container with pinned python and dependency versions for reproducible runs in CI/CD.

STEPS TO REPRODUCE

'fe80::1234/64' is ansible.utils.subnet_of 'fe80::/10'

EXPECTED RESULTS

Evaluating to true

ACTUAL RESULTS

Evaluating to false

Valid IPv6 input may cause 'ipsubnet' filter to attempt to create 2 ^ 128 values

SUMMARY

The "2600:1f1c:1b3:8f00::/56" | ipsubnet(120, 0) filter never returns. Internally, the filter invokes the netaddr.subnet function, which attempts to create 18446744073709551616 values.

The ipsubnet filter creates a IPNetwork with "2600:1f1c:1b3:8f00::/56" and prefix length 120.
The ipsubnet filter invokes the netaddr.subnet function at https://github.com/ansible-collections/ansible.utils/blob/main/plugins/filter/ipsubnet.py#L283
)

return str(list(value.subnet(query))[index])

In the netaddr module, the subnet() function calculates the max number of subnets:

width = self._module.width
max_subnets = 2 ** (width - self.prefixlen) // 2 ** (width - prefixlen)

If the network prefix is /56 and the prefix length is /120, then max_subnets = 18446744073709551616.

In the worst case, the network address is /0 and the prefix length is 128, then max_subnets = 2 ^ 128 // 2 ^ 0, which is 2 ^ 128. If the count argument is set to 0, this will cause unbounded CPU usage and memory allocation.
So netaddr.subnet never returns.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipsubnet filter

ANSIBLE VERSION

I have pasted the output below, but the problem is reproduced when running unit tests in the main branch of ansible.utils module as of 01/24/2022.

ansible [core 2.12.1]
  config file = None
  configured module search path = ['/home/serosset/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/serosset/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/serosset/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible
  python version = 3.8.11 (default, Sep  1 2021, 12:33:46) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

Because the problem is exposed when running the test_ipsubnet.py unit tests inside a docker image, the applicable collections and configuration is what's inside the docker container. I'm not sure how to get that information. But see summary, I think the problem can be root caused to a specific line in the ipsubnet function in the main branch.

ansible-test units --python=3.10 --docker -vvvv tests/unit/plugins/filter/test_ipsubnet.py

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

Checkout the code from #131, then run the following command:

# ansible-test units --python=3.10 --docker -vvvv tests/unit/plugins/filter/test_ipsubnet.py

The following filter never completes: "2600:1f1c:1b3:8f00::/56" | ipsubnet(120, 0)

The problem gets worse when the network prefix has a smaller value and the first argument of ipsubnet has a higher value. In the worst case, this would cause 2 ^ 128 iterations.

EXPECTED RESULTS

"2600:1f1c:1b3:8f00::/56" | ipsubnet(120, 0) should return 2600:1f1c:1b3:8f00::/120
"2600:1f1c:1b3:8f00::/56" | ipsubnet(120, 4) should return 2600:1f1c:1b3:8f00::400/120

This could be calculated in constant time. Instead, the ipsubnet filter attempts to create all possible subnets, then get the subnet at index X. However, the list of all subnets could be 2 ^ 128 in the worst case.

ACTUAL RESULTS

The ipsubnet filter never returns. It iterates through a very large number of elements and never returns.

Add xml_to_json and json_to_xml filter plugins

SUMMARY

Add xml_to_json filter plugin to convert a xml string to json structure using xmltodict as default parsing engine.

{{ some_xml_string_variable | xml_to_json(engine='xmltodict') }}

Add json_to_xml filter plugin to convert a json string to xml structure using xmltodict as default parsing engine.

{{ some_json_string_variable | json_to_xml(engine='xmltodict') }}

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

update_fact will prevent variable replacement under certain conditions

SUMMARY

under certain conditions, update_facts will prevent variables being substituted within variables it manipulates

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.utils.update_fact

ANSIBLE VERSION

ansible [core 2.11.4]
  config file = /Users/pookey/.ansible.cfg
  configured module search path = ['/Users/pookey/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/4.5.0/libexec/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/pookey/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.9.7 (default, Sep  3 2021, 04:31:11) [Clang 12.0.5 (clang-1205.0.22.9)]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

ansible-galaxy collection list ansible.utils

# /opt/homebrew/Cellar/ansible/4.5.0/libexec/lib/python3.9/site-packages/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.4.0

# /Users/pookey/.ansible/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 2.4.3

CONFIGURATION

OS / ENVIRONMENT

Tested on OSX and Linux

STEPS TO REPRODUCE

---

- hosts: localhost
  vars:
    env_name: moo
    datadog_checks:
      mysql:
        - host: "{{ env_name }}.moo"
      codedeploy:
        logs: []
  tasks:
    - debug:
        var: datadog_checks.mysql[0].host
    - name: update fact
      ansible.utils.update_fact:
        updates:
          - path: "datadog_checks['codedeploy']['logs']"
            value: "{{ datadog_checks['codedeploy']['logs'] + ['/path/to/logfile'] }}"
      register: new_dd
    - name: replace the datadog_checks array
      set_fact:
        datadog_checks: "{{ new_dd.datadog_checks }}"
    - debug:
        var: datadog_checks.mysql[0].host

EXPECTED RESULTS

the second debug should print 'moo.moo', in the same way the first debug does.

ACTUAL RESULTS

I get {{ env_name }}.moo as the output.

❯ ansible-playbook ./test.yaml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *****************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [debug] *********************************************************************************************************************************************************************
ok: [localhost] => {
    "datadog_checks.mysql[0].host": "moo.moo"
}

TASK [update fact] ***************************************************************************************************************************************************************
changed: [localhost]

TASK [replace the datadog_checks array] ******************************************************************************************************************************************
ok: [localhost]

TASK [debug] *********************************************************************************************************************************************************************
ok: [localhost] => {
    "datadog_checks.mysql[0].host": "{{ env_name }}.moo"
}

PLAY RECAP ***********************************************************************************************************************************************************************
localhost                  : ok=5    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

"Invalid" values for ipaddr filters

SUMMARY

Hi.
If the point of ipaddr filters is to parse/modify valid values and return False (or discard in lists) for invalid, then why some invalid values produce warnings (and even errors)?
The doc contains '' example which is shown to be discarded silently, but in actuality produces a warning.

Chaining multiple ipaddr family and other filters becomes hard as one needs to take care to manually filter out anything that might break it, instead of just relying on ipaddr filter to parse what is parseable. Even chaining two ipaddr filters together will result in warning if the value does not pass through the first because False is now invalid.

This behavior was introduced at some point, but why? and what is the plan for "breaking change in future"?

Just placing this before singular values in ipaddr filters could simplify a lot of stuff:

if isinstance(value, (int, str)) and value:
    # proceed with filtering
    pass
else:
    return False

ISSUE TYPE

• Bug Report

COMPONENT NAME

ipaddr

ANSIBLE VERSION

ansible [core 2.14.2]
  config file = None
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.1 (main, Dec 31 2022, 10:23:59) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

ansible.utils 2.8.0

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

Debian testing

STEPS TO REPRODUCE

---
- hosts: localhost
  gather_facts: no
  vars:
    potential_ips:
      - 127.0.0.1
      - 192.168.0.0/24
      - 192.168.0.14/24
      - 8.8.8.8
      - 8.8.8.0/27
      - ''
      - some string
      - []
      - yes
      - no
      - 0
      - - 1
        - 2
        - noip
        - ''
        - False
      - 1
      #- 1.12
      #- {key: value}
  tasks:
    - name: ipaddr per item
      debug:
        msg: |-
          {{ item }} => {{ item | ansible.utils.ipaddr }}
      loop: "{{ potential_ips }}"
    - name: ipaddr on list
      debug:
        msg: |-
          {{ potential_ips | ansible.utils.ipaddr }}

EXPECTED RESULTS

Everything invalid is silently discarded, ints and valid stings are parsed

TASK [ipaddr per item] *********************************************************
ok: [localhost] => (item=127.0.0.1) => {
    "msg": "127.0.0.1 => 127.0.0.1"
}
ok: [localhost] => (item=192.168.0.0/24) => {
    "msg": "192.168.0.0/24 => 192.168.0.0/24"
}
ok: [localhost] => (item=192.168.0.14/24) => {
    "msg": "192.168.0.14/24 => 192.168.0.14/24"
}
ok: [localhost] => (item=8.8.8.8) => {
    "msg": "8.8.8.8 => 8.8.8.8"
}
ok: [localhost] => (item=8.8.8.0/27) => {
    "msg": "8.8.8.0/27 => 8.8.8.0/27"
}
ok: [localhost] => (item=) => {
    "msg": " => False"
}
ok: [localhost] => (item=some string) => {
    "msg": "some string => False"
}
ok: [localhost] => (item=[]) => {
    "msg": "[] => []"
}
ok: [localhost] => (item=True) => {
    "msg": "True => False"
}
ok: [localhost] => (item=False) => {
    "msg": "False => False"
}
ok: [localhost] => (item=0) => {
    "msg": "0 => False"
}
ok: [localhost] => (item=[1, 2, 'noip', '', False]) => {
    "msg": "[1, 2, 'noip', '', False] => ['0.0.0.1', '0.0.0.2']"
}
ok: [localhost] => (item=1) => {
    "msg": "1 => 0.0.0.1"
}
ok: [localhost] => (item=1) => {
    "msg": "1.12 => False"
}
ok: [localhost] => (item=1) => {
    "msg": "{key: value} => False"
}
TASK [ipaddr on list] **********************************************************
ok: [localhost] => {
    "msg": [
        "127.0.0.1",
        "192.168.0.0/24",
        "192.168.0.14/24",
        "8.8.8.8",
        "8.8.8.0/27",
        [
            "0.0.0.1",
            "0.0.0.2"
        ],
        "0.0.0.1"
    ]
}

ACTUAL RESULTS

TASK [ipaddr per item] *********************************************************
ok: [localhost] => (item=127.0.0.1) => {
    "msg": "127.0.0.1 => 127.0.0.1"
}
ok: [localhost] => (item=192.168.0.0/24) => {
    "msg": "192.168.0.0/24 => 192.168.0.0/24"
}
ok: [localhost] => (item=192.168.0.14/24) => {
    "msg": "192.168.0.14/24 => 192.168.0.14/24"
}
ok: [localhost] => (item=8.8.8.8) => {
    "msg": "8.8.8.8 => 8.8.8.8"
}
ok: [localhost] => (item=8.8.8.0/27) => {
    "msg": "8.8.8.0/27 => 8.8.8.0/27"
}
[WARNING]: The value '' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
ok: [localhost] => (item=) => {
    "msg": " => False"
}
ok: [localhost] => (item=some string) => {
    "msg": "some string => False"
}
ok: [localhost] => (item=[]) => {
    "msg": "[] => []"
}
[WARNING]: The value 'True' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
ok: [localhost] => (item=True) => {
    "msg": "True => False"
}
[WARNING]: The value 'False' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
ok: [localhost] => (item=False) => {
    "msg": "False => False"
}
[WARNING]: The value '0' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
ok: [localhost] => (item=0) => {
    "msg": "0 => False"
}
ok: [localhost] => (item=[1, 2]) => {
    "msg": "[1, 2] => ['0.0.0.1', '0.0.0.2']"
}
ok: [localhost] => (item=1) => {
    "msg": "1 => 0.0.0.1"
}

TASK [ipaddr on list] **********************************************************
[WARNING]: The value '' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
[WARNING]: The value 'True' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
[WARNING]: The value 'False' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
[WARNING]: The value '0' is not a valid IP address or network, passing this
value to ipaddr filter might result in breaking change in future.
ok: [localhost] => {
    "msg": [
        "127.0.0.1",
        "192.168.0.0/24",
        "192.168.0.14/24",
        "8.8.8.8",
        "8.8.8.0/27",
        [
            "0.0.0.1",
            "0.0.0.2"
        ],
        "0.0.0.1"
    ]
}

If float or dict values are uncommented, it results in error.

fatal: [localhost]: FAILED! => {"msg": "Unrecognized type <<class 'float'>> for ipaddr filter <value>"}
fatal: [localhost]: FAILED! => {"msg": "Unrecognized type <<class 'dict'>> for ipaddr filter <value>"}

Ipaddr filters to be moved to utils from netcommon

SUMMARY

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

Sub-plugin documentation

SUMMARY

We do not currently publish the documentation strings from sub-plugins in this collection. Should we make it available somewhere? Move it into the related module plugins?

Existing sub-plugins
cli-parser sub-plugin docstring
cli-parser module plugin docstring
cli-parser module published documentation

ISSUE TYPE

• Documentation Report

COMPONENT NAME

All sub-plugins

ANSIBLE VERSION

Ansible 4 and later

update_fact: Support creation of vars and sub-fields

SUMMARY

Currently update_fact will report error if the var or subfield doesn't exist. It would be really nice if the missing var or subfield can be created automatically. To avoid breaking existing patterns, an additional argument could be added to update_fact such that by default it is still the old behavior.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

update_fact

ADDITIONAL INFORMATION

Update `validate` to use 2.11 `ArgumentSpecValidator` if available

Something like this....

try:
 from ansible.module_utils.common.arg_spec import ArgumentSpecValidator

 HAS_ANSIBLE_ARG_SPEC_VALIDATOR = True
except ImportError:
 HAS_ANSIBLE_ARG_SPEC_VALIDATOR = False
 
 def validate(self):
     """The public validate method
     check for future argspec validation
     that is coming in 2.11, change the check according above
     """
     if HAS_ANSIBLE_ARG_SPEC_VALIDATOR:
         if self._schema_format == "doc":
             self._convert_doc_to_schema()
         conditionals = self._schema_conditionals or {}
         validator = ArgumentSpecValidator(self._schema['argument_spec'], **conditionals)
         result = validator.validate(self._data)
         valid = not bool(result.error_messages)
         return valid, result.error_messages, result.validated_parameters
     else:
         return self._validate()