This role provides the foundation for building network roles by providing modules and plugins that are common to all Ansible Network roles.
License: GNU General Public License v3.0
![softwarefactory-project-zuul[bot] avatar](https://avatars.githubusercontent.com/in/6924?v=4 "softwarefactory-project-zuul[bot]")
The update method on a dict is only a single level. There is an _update now in the template/__init__.py that should be used at https://github.com/ansible-network/network-engine/blob/devel/action_plugins/text_parser.py#L148
Not sure how that will need to be edited to get to that method since it's now in a plugin. Might need a util structure to put common methods into.
To make the network-engine role easier to use and easier to describe, in 2.5.1 we plan to update several terms used in the role:
text_parser module to yaml_parsercommand_parser, provide aliastextfsm module to textfsm_parser, provide aliasfile parameter to parser_template in both included modules, provide aliasescontent parameter to command_output in both included modules, provide aliasesdir and src parameters will remain as they are.ansible --version
ansible 2.7.8
ansible-galaxy list | grep ansible.network
ansible-network.network-engine, v2.7.5
Broken Link at (https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/command_parser.md#sample-parser-templates) for "Parser templates for the command_parser module in the Network Engine role use YAML syntax. To write a parser template, follow the parser_directives documentation."
Currently - parser_directives documentation
Should - parser_directives documentation
Simple fix, looks like you just need to remove /user_guide/docs from the URL.
Broken link.
Consolidate the existing platform specific capabilities modules into a single net_facts module under network-engine and upstream to ansible/ansible. The generalized form of the capabilities should return the device_info key from [cliconf|netconf] plugins
ansible --version
ansible 2.7.0
config file = /Users/z003gn3/ansible_config/ansible.cfg
configured module search path = [u'/Users/z003gn3/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Users/z003gn3/virtualenvs/venv_lan_ansible/lib/python2.7/site-packages/ansible
executable location = /Users/z003gn3/virtualenvs/venv_lan_ansible/bin/ansible
python version = 2.7.10 (default, Oct 6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
ansible-galaxy list | grep ansible.network
- ansible-network.network-engine, v2.7.0
- ansible-network.cisco_ios, v2.6.2
I'm trying to use a textfsm template with network-engine to populate ansible_facts. I cannot find documentation how to supply the fact name to supply, and the facts are not showing in ansible_facts.
I added show_interfaces_status.template to parser_templates/cli folder. I'm just not sure how to register that json dictionary now.
# I added this to vars/get_facts_command_map.yaml
- command: show interfaces status
parser: show_interfaces_status.template
groups:
- all
- default
engine: textfsm_parser
I expect ansible facts to be populated.
command runs without issue, but ansible_facts aren't updated. It's pretty clear I'm not setting the facts through the course of this, but I don't know how I'm supposed to using textfsm vs command_parser.
I don't beleive the README.md should advise users not to invoke the network-engine role directly in a playbook. I suggest the removal of this wording.
It's a common practice to invoke the role at the beginning of the playbook, this wording also doesn't match with the later wording.
Any of the provided functions can be implemented in Ansible playbooks directly.
If you can use the functions in the playbook why can't you invoke the role in the playbook?
This role provides the foundation for building network roles by providing modules and plugins that are common to all Ansible Network roles. Typically this role should not be directly invoked in a playbook.
ansible --version
ansible 2.7.8
config file = /var/lib/awx/projects/net_compliance/ansible.cfg
configured module search path = [u'/var/lib/awx/projects/net_compliance/library', u'/usr/lib/python2.7/site-packages/napalm_ansible/modules', u'/usr/lib/python2.7/site-packages/ntc-ansible/library']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
ansible-galaxy list | grep ansible.network
- ansible-network.config_store, (unknown version)
- ansible-network.network-engine, v2.7.5
- ansible-network.cisco_ios, v2.7.1
Docker Container (awx_task):
[root@4567e492e296 net_compliance]# uname -a
Linux 4567e492e296 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
network-engine 2.7.5, the error message received when the command_parser template is "missing or invalid" no longer provides the file path/name within the message, rather only "None" is provided. As seen below, the filename and path are provided when going back to network-engine 2.7.3
Use command_parser with any filename that doesn't actually exist (or not in the path):
- name: PARSE THE RAW OUTPUT
command_parser:
file: "{{ parser_template_dir }}/ios/show_run_interface_switch1.yaml"
content: "{{ ansible_net_config }}"Output similar to ansible-network.network-engine v2.7.3
fatal: [switch_hostname]: FAILED! => {}
MSG:
src [/var/lib/awx/projects/net_compliance/parser_templates/ios/show_run_interface_switch1.yaml] is either missing or invalid
Paste verbatim command output between quotes below -->
ansible-network.network-engine v2.7.5
fatal: [switch_hostname]: FAILED! => {}
MSG:
src [None] is either missing or invalid
ansible --version
2.7.6
ansible-galaxy list | grep ansible.network
2.7.3
NXOS 9K
The following spec file fails with TypeError
meta/setup.yaml
---
required_together:
- [ 'create_service_account', 'service_account_name', 'service_account_password']
- [ 'remove_service_account', 'service_account_name' ]
required_one_of:
- [ 'remove_service_account', 'create_service_account' ]The full traceback is:
Traceback (most recent call last):
File "/home/vagrant/dev-workspace/ansible/lib/ansible/executor/task_executor.py", line 142, in run
res = self._execute()
File "/home/vagrant/dev-workspace/ansible/lib/ansible/executor/task_executor.py", line 644, in _execute
result = self._handler.run(task_vars=variables)
File "/home/vagrant/.ansible/roles/ansible-network.network-engine/action_plugins/validate_role_spec.py", line 91, in run
validated_spec = basic.AnsibleModule(**spec)
TypeError: __init__() takes at least 2 arguments (3 given)
fatal: [frr-dummy]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
N/A
N/A
It looks as though the functionality from this has migrated its way into the modern network libraries within Ansible. This should be documented on the repository and also on Ansible Galaxy to shift and guide to the proper location for parsing specifically.
$ ansible --version
ansible 2.5.2
config file = /home/v-jamigh/git/Ansible-Networking/ansible.cfg
configured module search path = [u'/home/v-jamigh/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
We're unable to use when: in a json_template. This is holding us back from updating to the plugin version. I saw the comment in the plugin code, making this issue to track it a little easier.
- name: template interface state
register: interfaces_state
loop: "{{ contents.TABLE_interface.ROW_interface }}"
export: yes
export_as: dict
json_template:
template:
- key: interfaces
object:
- key: interface
object:
- key: "{{ item.interface }}"
object:
- key: ethernet
object:
- key: state
object:
- key: hw-mac-address
value: "{{ item.eth_hw_addr }}"
- key: duplex-mode
value: "{{ item.eth_duplex }}"
- key: port-speed
value: "{{ item.eth_speed }}"
when: "'ethernet' in (item.interface | lower)"Only load the object when the conditional passes.
when isn't check
ansible 2.9.0
config file = None
configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/user/.local/lib/python3.6/site-packages/ansible
executable location = /home/user/.local/bin/ansible
python version = 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0]
ansible-network.network-engine, v2.7.5
All
Nearly all Network OS report a short name for interfaces. It would be useful for interface_split or a similar filter to spit such short names out given a full interface name.
{{ 'Ethernet1/1' | interface_split('shortname') }} returns 'Et1/1'
{{ 'Loopback0' | interface_split('shortname') }} returns 'Lo0'
{{ 'TenGigabitEthernet1/0/1' | interface_split('shortname') }} returns 'Te1/0/1'
{{ 'Tunnel1' | interface_split('shortname') }} returns 'Tu1'
validate_config test plugin will validate the input config against a yaml spec file derived from yang module. The output will indicate if the input config conforms with spec file or not
Ref: #54
Feature request:
The search path to include the parsers directory adjacent to the playbook directory
It would be preferable to use the parser file name and default to parser_templates/{{ ansible_network_os }}/file.[yml|yaml]
We have with the ansible network engine role, the ability to parse the output of "show commands" executed on a network device (e.g. 'show version'). This is with using the 'command_parser' or 'textfsm_parser'. The present feature request is for the ability to parse the configuration text of a network device and to make the capabilities in the custom module at the following link, native to the ansible network engine role.
Custom Module, which we woild like to see native to Ansible Network Engine: https://cidrblock.github.io/section-snipping-module-for-ansible.html
Create a new task cli for execting cli commands against a network device. The task should be pinned to using only the network_cli connection. The task should support the following input parameters:
command - the command to run
parser - the path to the parser to pass the command through
engine - the command parsing engine to use
ansible --version
ansible 2.7.9
config file = /Users/hectorisaza/ansible.cfg
configured module search path = [u'/etc/ansible/library']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.16 (default, Mar 4 2019, 09:01:38) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]
ansible-galaxy list | grep ansible.network
- ansible-network.network-engine, v2.7.5
There is a minor bug that has a comma next to the regex key, that will not display the captured data.
- name: match hardware
pattern_match:
regex: "Hardware is (\\S+)," <---- bug ","
content: "{{ item }}"
register: typeok: [csr1] => {
"interface": {
"ansible_facts": {
"interface_facts": [
{
"GigabitEthernet1": {
"config": {
"description": null,
"mtu": "1500",
"name": "GigabitEthernet1",
"type": "CSR" <----- Expect to see the type value to be populated
}
}
},
ok: [csr1] => {
"interface": {
"ansible_facts": {
"interface_facts": [
{
"GigabitEthernet1": {
"config": {
"description": null,
"mtu": "1500",
"name": "GigabitEthernet1",
"type": null <--------- Comes back as null
}
}
},
Documentation Report
docs/user_guide/filter-plugins.md
Compress a vlan list
1,2,3,4,5 becomes 1-5
1,2,4,5 becomes 1-2,4-5
1,2,3,5 becomes 1-3,5
The _coerce_to_native function in
network-engine/lib/network_engine/plugins/template/__init__.py
incorrectly casts values to type integer.
Any string value that looks like an integer is turned into an integer which causes issues. Many network constructs should not be converted from type string. For example, an ip prefix-list defined on a Cisco device could have a name similar to "0012345678" however once the that value goes through this process the prefix-list name is change to 12345678 of type integer.
The line value = int(value) is changing the variable type of objects that should be strings.
def _coerce_to_native(self, value):
if not isinstance(value, bool):
try:
value = int(value)
except Exception:
if value is None or len(value) == 0:
return None
pass
return value
text_parser:
file: "ios_prefix_list_parser.yml"
content: "{{ prefix_list_output.stdout[0] }}"
prefix_list["0012345678"] is defined
prefix_list[12345678] is defined
Expand a vlan string to its elements
vlan1,3-5,7 expands to 1,3,4,5,7
vlan1-5 expands to 1,2,3,4,5
ansible --version
ansible 2.10.7
ansible-galaxy list | grep ansible.network
- ansible-network.network-engine, v2.7.5
After upgrade ansible from 2.9.18 to 2.10, I was no longer able to run my playbooks with network-engine.
The error was "ModuleNotFoundError: No module named 'ansible.module_utils.network_common'"
In file
~/.ansible/roles/ansible-network.network-engine/action_plugins/command_parser.py
I had to change line 26
from ansible.module_utils.network_common import to_list
to
from ansible_collections.ansible.netcommon.plugins.module_utils.network.common.utils import to_list
And in file:
~/.ansible/roles/ansible-network.network-engine/lib/network_engine/utils.py
I had to change line 13
from ansible.module_utils.network.common.utils import sort_list
to
from ansible_collections.ansible.netcommon.plugins.module_utils.network.common.utils import sort_list
---
- hosts: branches
gather_facts: no
tasks:
- name: import the network-engine role
import_role:
name: ansible-network.network-engine
- name: Generate global_facts as JSON
command_parser:
file: my_parsers.yml
content: "{{ inventory_hostname }} "No errors at all.
ERROR! Unexpected Exception, this is probably a bug: No module named 'ansible.module_utils.network_common'
the full traceback was:
Traceback (most recent call last):
File "~/.ansible/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 23, in <module>
from ansible.module_utils.network.common.utils import to_list
ModuleNotFoundError: No module named 'ansible.module_utils.network'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "~/.venvs/ansible-3.1/bin/ansible-playbook", line 123, in <module>
exit_code = cli.run()
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/cli/playbook.py", line 129, in run
results = pbex.run()
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/executor/playbook_executor.py", line 169, in run
result = self._tqm.run(play=play)
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 281, in run
play_return = strategy.run(iterator, play_context)
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/plugins/strategy/linear.py", line 267, in run
action = action_loader.get(task.action, class_only=True, collection_list=task.collections)
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/plugins/loader.py", line 782, in get
return self.get_with_context(name, *args, **kwargs).object
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/plugins/loader.py", line 802, in get_with_context
self._module_cache[path] = self._load_module_source(name, path)
File "~/.venvs/ansible-3.1/lib/python3.7/site-packages/ansible/plugins/loader.py", line 766, in _load_module_source
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "~/.ansible/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 26, in <module>
from ansible.module_utils.network_common import to_list
ModuleNotFoundError: No module named 'ansible.module_utils.network_common'
$ ansible --version
ansible 2.5.2
config file = /home/v-jamigh/git/Ansible-Networking/ansible.cfg
configured module search path = [u'/home/v-jamigh/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
We're unable to use variables with export_as. We'd like to be able to pass in a variable that can be evaluated at runtime.
export_as: "{{ export_type | default('dict') }}"Read export_type or default to 'dict'
fatal: [host]: FAILED! => {
"msg": "invalid value for export_as, got {{ export_type | default('dict') }}"
}
ansible 2.7.5
ansible-network.network-engine, v2.7.2
ios
ios-xe
ios-xr
nxos
Currently one cli command output can be linked with one command parser instance (YAML file) to produce a structured json template structure based on the unstructured ASCII response of that single command. The "extend" directive has been added, which allows to extend an existing facts json structure, but always with a "separate key", right? (hope I did not miss something here)
An extend facts structure might look like this:
The command based on the first fact list items is "show interface", the second command is "show interfaces status". Here we see the same key ("0/1") twice, because I extracted the interface ID as root key for both commands...
"ansible_facts": {
"device": {
"extension": {
"interface_facts": [
{
"0/1": {
"config": {
"description": "*** ROUTER ***",
"mac": null,
"mtu": "1500",
"name": "0/1"
},
"stats": {
"drops_out": "21059",
"input_errors": "0",
"input_rate": "433000 bits/sec",
"output_errors": "0",
"output_rate": "38000 bits/sec"
}
}
},
[output of other interfaces omitted]
{
"0/1": {
"config": {
"status": "connected",
"type": "10/100/1000BaseTX",
"vlan": "trunk"
}
}
},
[output of other interfaces omitted]
Especially on network interfaces, one show command often do not give all required information for further ansible processing in efficient way. In above example the first command does not indicate if the interface operates as access port or trunk - the second command provides this information.
All these interface related commands have at least one common key together - the interface ID or Name. A new "combine" directive could match on a common key like that and merges the json structure based on that to unique data sets. This makes everything a lot easier for futher processing in ansible (like loops a.s.o)
E.g. a "show cdp neighbor" could also be included as a third show command - it would just be great to have CDP neighbors under the same interface related key. "show mac address-table" is another example.
If the command parser would allow to create a combined json structure based on one common key (like interface ID) using multiple commands - ansible could do things quite out of the box, that no other network mgmt tools can do until today!
"ansible_facts": {
"device": {
"extension": {
"interface_facts": [
{
"0/1": {
"config": {
"description": "*** ROUTER ***",
"mac": null,
"mtu": "1500",
"name": "GigabitEthernet0/1"
},
"mac_addr": {
[
"000c.29e3.3204",
"709e.2998.b20c"
]
},
"cdp_neig": {
"name": router01,
"interface": "GigabitEthernet1/1/1",
"type": "C819G-4G"
}
"stats": {
"drops_out": "21059",
"input_errors": "0",
"input_rate": "433000 bits/sec",
"output_errors": "0",
"output_rate": "38000 bits/sec"
}
}
},
[further output omitted]
With this comprehensive facts I could do maximum flexible conditional configuration tasks, for example: Configure only active (up) trunk interfaces with routers as neighbors with setting "spanning-tree portfast trunk" - there are countless usecases...
I apologize my second Feature request in a short time, I currently do deep Ansible eval for network automation and this role is very promising! It makes great things possible without need to develop own modules (which is currently yet a kind of show stopper in regards to Ansible for Networkers)
Currently the parse will accept (and silently fail) if content: is passed a list (or anything else that isn't a string).
This can cause confusion if you pass in (say) a list and you don't get any matches.
ansible 2.7.6
Ansible Tower 3.3.1
ansible-network.network-engine, v2.7.4
I am attempting to utilize network-engine's command_parser to extract key facts, such as network device version number, from the device.
Because different versions of NXOS and IOS seem to present the data in a slightly different way, I was hoping to use a regex branch reset group to apply 'OR' logic to match the different variants of how the data is displayed to the same regex group.
To reproduce, attempt to use a branch reset group under 'regex' within pattern_match.
`- name: MATCH PATTERN
pattern_match:
regex: "(?|(doggies)|(Cisco Nexus Operating))"
match_all: yes
register: section
- name: GENERATE JSON DATA STRUCTURE
json_template:
template:
- key: "show version"
object:
- key: version
value: "{{ item.matches.0 }}"
loop: "{{ section }}"
extend: device_facts
export: yes
register: device_facts`The above should match either 'doggies' or 'Cisco Nexus Operating' to regex group 1. This works fine on online Regex editors, but throws an error when I try this through command_parser.
regex101.com: Link to expected result example
TASK [Parsing command data into JSON format: show version] *********************
task path: /var/lib/awx/projects/_69__network_data_collection/data-discovery/show-nxos.yml:30
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 140, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 612, in _execute
result = self._handler.run(task_vars=variables)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 187, in run
res = self._process_directive(task)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 391, in _process_directive
return meth(**args)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 406, in do_pattern_match
return parser.match(regex, match_all, match_until, match_greedy)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/parser/pattern_match.py", line 39, in match
return self._match_all(content, regex)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/parser/pattern_match.py", line 44, in _match_all
match = self.re_matchall(pattern, content)
File "/var/lib/awx/projects/_69__network_data_collection/data-discovery/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/parser/pattern_match.py", line 153, in re_matchall
regex = re.compile(regex)
File "/usr/lib64/python2.7/re.py", line 190, in compile
return _compile(pattern, flags)
File "/usr/lib64/python2.7/re.py", line 242, in _compile
raise error, v # invalid expression
error: unexpected end of pattern
fatal: [S022DC3N5212]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
yang2spec lookup plugin will transform given yang model to a yaml spec which can be used to validate input json config.
ansible 2.7.5
ansible-network.network-engine, v2.7.2
ios
ios-xe
ios-xr
nxos
The current implementation of the netcfg_diff lookup plugin allows to create a diff based on a "have" and a "want" configuration - the result is basically the lines that are missing in the "have" configuration.
Network engineers also need to make sure, that certain configurations/features are never configured/active on certain network devices. It would be great to extend the scope if this plugin (or create a new one) that allow to compare a network configuration against a set of config parameters that should never occur in a network device configuration.
Compare a "have" configuration to a "must not" template and if the "have" configuration contains any of the "must not" parameters, generate a "no parameter" template as a report.
E.g. these global configurations should never be activated on any Cisco device:
service config
service tcp-small-servers
service udp-small-servers
service pad
service finger
Also on interfaces, there are some configurations, that should never occur on certain important ports e.g. if they are uplink interfaces or Server ports:
interface range <RANGE>
no logging event link-status
no snmp trap link-status
Another simple example, only ssh should be enabled on any network device - never telnet
line vty 0 15
transport input telnet
Trying to use a parser defined in roles/ansible-network.network-engine/parsers/nxos/
I tried the following without success:
show_version.ymlnxos/show_version.ymlparsers/nxos/show_version.yml~/.ansible/roles/ansible-network.network-engine/parsrs/nxos/show_version.ymlException:
fatal: [sw01]: FAILED! => {
"msg": "src [~/.ansible/roles/ansible-network.network-engine/parsers/nxos/show_version.yml] is either missing or invalid"
}
Using the absolute path did work, but I would expect:
parsers directory adjacent to the tasks directory~ would be acceptable to use as my home directoryIt would be preferable to use the parser file name and default to parsers/{{ ansible_network_os }}/file.[yml|yaml]
ansible --version
ansible 2.7.6
config file = /home/4032956/repos/network-automation/ansible.cfg
configured module search path = ['/home/4032956/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/4032956/.pyenv/versions/3.7.1/envs/network-automation/lib/python3.7/site-packages/ansible
executable location = /home/4032956/.pyenv/versions/network-automation/bin/ansible
python version = 3.7.1 (default, Dec 4 2018, 14:17:28) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
$ ansible-galaxy list -p roles/ | grep ansible-network
- ansible-network.cisco_ios, v2.7.0
- ansible-network.juniper_junos, v2.7.2
- ansible-network.network-engine, v2.7.2
- ansible-network.cisco_nxos, v2.7.1
- ansible-network.config_manager, v2.6.2
- ansible-network.network-engine, v2.7.1
Not relevant
When you construct a json_template that references a non-existing (or perhaps null) parsed variable, ansible hangs. It should fail explicitly with a useful message.
From inside ansible-network.cisco_ios
ansible-playbook tests/test_parser_templates.yaml -i tests/inventoryWhen network engine tries to construct the key restart_time (link) which is null / doesn't exist because the previous regex didn't match anything, it should produce a meaningful error.
TASK [lukedrussell.cisco_ios : run command and parse output] ****************************************************************
task path: /home/4032956/repos/network-automation/roles/lukedrussell.cisco_ios/includes/run_cli.yaml:6
<snip> attempting to start connection
<snip> using connection plugin network_cli
<snip> found existing local domain socket, using it!
<snip> updating play_context for connection
<snip>
<snip> local domain socket path is /home/4032956/.ansible/pc/c96ac4390e
Traceback (most recent call last):
File "/home/4032956/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/4032956/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle StrictUndefined objectsThe playbook hangs at this point for all hosts, and must Ctrl+C to break it.
As a user, If I make a mistake in a matching directive or in the content: parameter, I may end up with one or more null values in the final variable output. For example, this playbook:
---
- hosts: ios
connection: network_cli
gather_facts: no
tasks:
- name: Collect version information from device
ios_command:
commands: "show version"
register: ios_version_output
- import_role:
name: ansible-network.network-engine
- name: Generate version facts as JSON
text_parser:
file: "my_parsers/ios_show_version.yaml"
content: ios_version_output['stdout_lines'][0]
# in case anyone find this while looking for a *correct* example
# the correct formulation would be
# content: "{{ ios_version_output.stdout.0 }}"
- name: Display matches (system_facts)
debug:
var: system_facts succeeds, but returns this final variable:
ok: [ios01] => {
"system_facts": {
"image_file": null,
"memory": {
"free": null,
"total": null
},
"model": null,
"uptime": null,
"version": "null"
}
}The error in this example is in the content line, but a similar situation could occur if one or more match rules were incorrect. If I run the playbook with -vvvv I can see the directives:
processing directive: match version
processing directive: match model
processing directive: match image
processing directive: match uptime
processing directive: match total memory
processing directive: match free memory
processing directive: export system facts to playbook
but there's no information about which ones find matches.
To help debug issues with content lines and issues with match definitions, could we:
ansible 2.7.8
ansible tower 3.4.1
ansible-network.network-engine 2.7.4
N/A
Currently in command_parser.py after source path is constructed tasks are still loaded from src [tasks = self._loader.load_from_file(src)], but with recent changes I think tasks should now load from src_path [tasks = self._loader.load_from_file(src_path)]
- name: parse the show run vrf output
command_parser:
file: "filename.yml"
content: "{{output.stdout[0]}}"
Command_parser finishes parsing successfully
on ansible tower playbook fails with
fatal: [XXXXXXXX]: FAILED! => {"msg": "Unable to retrieve file contents\nCould not find or access '/var/lib/awx/projects/_project_name/filename.yml' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
Ref: #54
Playbook:
text_parser:
file: "~/roles/network-engine/parsers/ios/show_interfaces.yaml"
content: "{{ lookup('file', './output/ios/show_interfaces.txt') }}"
Result:
fatal: [ios]: FAILED! => {
"msg": "src [~/roles/network-engine/parsers/ios/show_interfaces.yml] is either missing or invalid"
}
Documentation Report
docs/user_guide/lookups.md
ansible --version
ansible 2.6.5
config file = /Users/me/.ansible.cfg
configured module search path = [u'/Users/me/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Users/me/Documents/git/local_ansible/venv/lib/python2.7/site-packages/ansible
executable location = /Users/me/Documents/git/local_ansible/venv/bin/ansible
python version = 2.7.10 (default, Oct 6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
I have to populate my parser_templates folder, complete with all templates, in two locations for the playbook to succeed.
I have the following folder structure.
local_ansible
-inventory
temp
- playbooks
show_version.yaml
I call my playbook via
ansible-playbook -i inventory/temp playbooks/show_version.yaml -k
Under this folder struct the playbook works
local_ansible
-inventory
temp
- parser_templates
ios
nxos
junos
- playbooks
- parser_templates
ios
nxos
junos
show_version.yaml
Parser templates is in two locations
local_ansible/parser_templates
local_ansible/playbooks/parser_templates
The playbook succeeds under this setup.
If I remove parser_templates from the root folder leaving it only defined in the playbooks folder the playbook fails.
Folder structure is now
local_ansible
-inventory
temp
- playbooks
- parser_templates
ios
nxos
junos
show_version.yaml
The following error is provided
TASK [Generate Version Fact as JSON] ************************************************************************************************
fatal: [myhost]: FAILED! => {"msg": "src [parser_templates/nxos/show_version.yaml] is either missing or invalid"}
Under a different setup the parser_templates folder is in the root directory. But I have removed it from the playbooks directory.
This is the structure
local_ansible
-inventory
temp
- parser_templates
ios
nxos
junos
- playbooks
show_version.yaml
I then get this error instead
TASK [Generate Version Fact as JSON] ************************************************************************************************
fatal: [myhost]: FAILED! => {"msg": "Unable to retrieve file contents\nCould not find or access '/Users/me/Documents/git/local_ansible/playbooks/parser_templates/nxos/show_version.yaml' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
If I return the state of having the folder in both locations the playbook succeeds. Any ideas? Seems like the module is using the same relative path from two contexts thereby causing this problem.
Here is the actual playbook
- hosts: nxos
connection: network_cli
gather_facts: false
tasks:
- name: Collect Version Information
ios_command:
commands:
- show version
register: nxos_version_output
- name: import the network-engine role
import_role:
name: ansible-network.network-engine
- name: Generate Version Fact as JSON
command_parser:
file: "parser_templates/nxos/show_version.yaml"
content: "{{ nxos_version_output.stdout.0 }}"
- hosts: junos
connection: netconf
gather_facts: false
tasks:
- name: Collect Version Information
junos_command:
commands:
- show version
register: junos_version_output
- name: import the network-engine role
import_role:
name: ansible-network.network-engine
- name: Generate Version Fact as JSON
command_parser:
file: "parser_templates/junos/show_version.yaml"
content: "{{ junos_version_output.stdout.0 }}"
- hosts: all
gather_facts: false
tasks:
- name: Display Facts
debug:
msg: "{{ [system_facts.version] }}"
ansible --version
ansible 2.6.3
config file = /Users/4032956/Documents/network-automation/ansible.cfg
configured module search path = ['/Users/4032956/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Users/4032956/.pyenv/versions/3.6.6/envs/network-automation/lib/python3.6/site-packages/ansible
executable location = /Users/4032956/.pyenv/versions/network-automation/bin/ansible
python version = 3.6.6 (default, Jul 30 2018, 09:47:46) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
ansible-galaxy list | grep ansible.network
v2.6.3
Running a submodule currently on commit 654729f3931d6d9f6485a2a0458753752905e01b
When using the template argument repeat_for and repeat_var I was getting python exceptions. With help from a developer here he got me to change json_template.py and it started working. I'm not sure really what is happening with the code.
See this gist or my fork of cisco_ios
Complete gist
Traceback (most recent call last):
File "/Users/4032956/.pyenv/versions/3.6.6/envs/network-automation/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 139, in run
res = self._execute()
File "/Users/4032956/.pyenv/versions/3.6.6/envs/network-automation/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 577, in _execute
result = self._handler.run(task_vars=variables)
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 159, in run
resp = self._process_directive(task)
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 395, in _process_directive
return meth(**args)
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/command_parser.py", line 413, in do_json_template
return self.template.run(template, self.ds)
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/template/json_template.py", line 71, in run
val = self.run(items, variables)
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/template/json_template.py", line 56, in run
templated_value.append(self.run(items, variables))
File "/Users/4032956/Documents/network-automation/roles/ansible-network.network-engine/action_plugins/../lib/network_engine/plugins/template/json_template.py", line 24, in run
key = self.template(item['key'], variables)
TypeError: string indices must be integers
fatal: [localhost]: FAILED! => {}
MSG:
Unexpected failure during module execution.
to retry, use: --limit @/Users/4032956/Documents/network-automation/retry/test.retry
Hi,
I have written some parser templates for Cisco IOS. I tried to find if someone had already written such templates before I wrote mine, but couldn't find any.
I would like to publish my templates so that others in the same situation don't have to start from zero. I would also be happy to get improvements to my templates from others.
What is the best way to publish templates so that others will find them?
What is the "best" folder structure for the templates? I think that there should be a standard way of doing things.
Thanks for all the work with network-engine!
ansible --version
2.5
ios
iosxr
eos
nxos
junos
vyos
Create a lookup plugin that generates a configuration diff between two configuration input (eg: candidate and running). The diff is generated based on the options given as input to the lookup plugin.
I am creating templates for some Cisco IOS devices.
Unfortunately Cisco is not consistent with the output on some of their commands across IOS versions and devices.
For example the command "show vtp status" outputs:
VTP version running : 2
on some switches. And this on others:
VTP Version : 2
Is it possible to handle this with an if statement or or?
(regexp like this, if null then regexp like this instead).
Or something similar to this:
pattern_match:
regex: "VTP Version capable : (.*)" | "VTP Version : (.*)"
Also the amounts of whitespace between names and variables differ on different devices:
Number of existing VLANs : 34
Number of existing VLANs : 8
I haven't found any examples or documentation on how to handle these situations.
EDIT:
I solved this particular problem with this regex:
regex: "VTP \\Sersion [r|\\s][u|\\s][n|\\s][n|\\s][i|\\s][n|\\s][g|\\s] : (.*)"
Still I would like to know if there is a possibility to do an if statement.
Also more documentation would be very helpful.
Summary
Textfsm returns only the last dictionary entry from facts as ansible_facts. The engine should be able to parse all the vlans from content.
Component name
action_plugins/textfsm.py
Steps to Reproduce
$ cat vlan/show.yaml
tasks:
- name: Parse from contents and generate json structure
textfsm:
file: parsers/nxos/show_vlan.yaml
contents: "{{ lookup('file', 'output/nxos/show_vlan.txt') }}"
$ cat vlan/parsers/nxos/show_vlan.yaml
Value VLAN_ID (\d+)
Value NAME (\w+)
Value STATE (\w+)
Start
^\d+\s+enet\s+CE
^${VLAN_ID}\s+${NAME}\s+${STATE}\s+ -> Record
$ cat vlan/output/nxos/show_vlan.txt
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Eth1/1, Eth1/2, Eth1/7, Eth1/8
Eth1/9, Eth1/10, Eth1/11
102 app02 active
103 app03 active
1001 test1001 active
2096 test active
VLAN Type Vlan-mode
---- ----- ----------
1 enet CE
102 enet CE
103 enet CE
1001 enet CE
2096 enet CE
Remote SPAN VLANs
-------------------------------------------------------------------------------
Primary Secondary Type Ports
------- --------- --------------- -------------------------------------------
Actual Result
ok: [nxos9k-02] => {
"ansible_facts": {
"NAME": "test",
"STATE": "active",
"VLAN_ID": "2096"
},
"changed": false
}
Expected Result
Should parse all the vlans
2.6.4
Tower 3.3
Arista vEOS: 4.20.1F
validate_role_spec is not defining default values properly.
---
argument_spec:
mlag_setup:
type: dict
options:
state:
type: str
required: false
default: present
choices:
- "present"
- "absent"
description:
- "Desired state: present, absent"
peer:
type: str
required: true
description:
- "The MLAG peer switch hostname (inventory_hostname)"
vlan:
type: int
required: false
default: "4094"
description:
- "VLAN to use for peer communication."
- "VLAN is used to create the interface vlan for MLAG_CONTROL_PLANE."
domain:
type: int
required: false
default: "1"
description:
- "Represents the MLAG domain."
- "The ID must be the same only for members of the same MLAG domain."
- "Domain is used to create the MLAG_CONTROL_PLANE VRF with RD <site>:<domain>"
site:
type: int
required: false
default: "1"
description:
- "Represents the site where the domain is located."
- "Site is used to create the MLAG_CONTROL_PLANE interface with address 169.254.<domain>.<site><device>"
- "Site is used to create the MLAG_CONTROL_PLANE VRF with RD <site>:<domain>"
device:
type: int
required: false
default: "1"
description:
- "Unique ID to represent the device member of the MLAG domain"
- "The ID is used to create the peer communication address 169.254.<domain>.<site><device>"
linkagg:
type: int
required: false
default: "1"
description:
- "Linkagg to use for the peer-to-peer communication."
vlans:
type: list
required: false
default:
- "all"
description:
- "List of vlans member of the MLAG domain"
- "VLANs are NOT created if they do not exist"
interlinks:
type: list
required: false
default:
- "Ethernet1"
description:
- "List of physical interfaces used by linkagg as interlinks for the peer-to-peer communication"
heartbeat:
type: int
default: "10000"
required: false
description:
- "Timer for heartbeat in ms"
reload_delay:
type: int
default: "150"
required: false
description:
- "Delay to bring MLAGs up after reload."
- "Helps to ensure that MLAGs comes up only when adjacency is done and tables are pre-populated."
- "Prevents blackhole of traffic when device is converging and tables being populated after a reboot."
ipv4:
type: str
required: true
description:
- "IPV4 address to use for the MLAG intercommunication"
vrf:
type: str
required: false
default: "mlag_control"
description:
- "The VRF name for the MLAG intercommunication"
extension:
type: dict
required: false
description:
- "Extension of specs with platform-specific keys/values"Undefined variables and subkeys being defined with the value defined in default.
Neither variables or subkeys are defined with the default values specified in the spec file.
If a pattern_match doesnt find what's being looked for, the returned container matches is undefined. This makes us unable to reference the matches.0 and use filters on it since matches is undefined.
Example pattern_match
- name: match duplex
pattern_match:
regex: "duplex (\\S+)"
contents: "{{ item }}"
register: duplex_modeIf duplex isn't found and I try to default it with something like, but it will always return null
item.duplex_mode.matches.0 | default('auto')My current workaround is an inline conditional
'auto' if item.duplex_mode.matches is undefined else item.duplex_mode.matches.0Minor lint issues found in network-engine:
ansible --version
ansible 2.6.1
ansible-galaxy list | grep ansible.network
- ansible-network.network-engine, v2.6.0
n/a for this report.
% find ./roles/ansible-network.network-engine -type f -name \*.yml -print0 | while read -d $'\0' -r YML
do
yamllint "$YML"
ansible-lint "$YML"
done
./roles/ansible-network.network-engine/tasks/main.yml
2:32 error no new line character at the end of file (new-line-at-end-of-file)
./roles/ansible-network.network-engine/tests/json_template/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/netcfg_diff/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/text_parser/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/vlan_compress/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/textfsm_parser/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/interface_range/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/command_parser/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/textfsm/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/interface_split/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/tests/vlan_expand/test.yml
1:1 warning missing document start "---" (document-start)
./roles/ansible-network.network-engine/meta/main.yml
4:81 error line too long (154 > 80 characters) (line-length)
13:81 error line too long (84 > 80 characters) (line-length)
19:81 error line too long (81 > 80 characters) (line-length)
22:4 warning missing starting space in comment (comments)
./roles/ansible-network.network-engine/.travis.yml
12:5 error wrong indentation: expected 6 but found 4 (indentation)
29:61 error no new line character at the end of file (new-line-at-end-of-file)
[ANSIBLE0008] deprecated sudo feature
./roles/ansible-network.network-engine/.travis.yml:2
{'sudo': False}
./roles/ansible-network.network-engine/handlers/main.yml
2:35 error no new line character at the end of file (new-line-at-end-of-file)
Lint checks should not report any results. Almost all of the items above are very minor and trivial to fix.
The one deprecated feature may take longer to resolve.
https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/
% ansible-test sanity
zsh: command not found: ansible-test
I was not able to test with your published sanity checker. The above commands were available, and did produce reasonable items, so they were used instead.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.