ciscotestautomation / yang

NETCONF/YANG related test framework

NETCONF/YANG Test Framework

This repository homes two distinct but related packages:

Connector

The yang.connector package features a series of classes that connect to Data Model Interfaces (DMI), in particular, an implementation of NETCONF client and GNMI client for Cisco pyATS.

See Connector Documentation for more details.

Ncdiff

The yang.ncdiff package features classes that allows the user to download, compile, and diff NETCONF configs for automation purposes, such as calculating new config states based on current configuration state, and any new edit-configs.

See Ncdiff Documentation for more details.

Installation

Both packages are featured in the Python Package Index and can be directly installed into your Python environment.

# connector
pip install yang.connector

# ncdiff
pip install yang.ncdiff

Support

Everyone is welcomed to contribute to this git repository. For support questions and issues, open an issue under this git repository and it will be attended to.

Copyright (c) 2018 Cisco Systems, Inc. and/or its affiliates

yang's Issues

Stabilize version requirements

Hitting version requirement issue (again) mainly with regard to grpcio. Previously yang.connector was grpcio<=1.28.1 and my library requires yang.connector so getting version conflict again. Can you explain why grpcio is restricted to less than 1.36.1 in yang.connector now? I control cisco-gnmi and grpcio is required there and open to any version.

yang.connector:

    # package dependencies
    install_requires =  [
        'paramiko >= 1.15.1',
        'lxml >= 3.3.0',
        'ncclient >= 0.6.6, < 0.6.10',
        'grpcio <= 1.36.1',
        'cisco-gnmi >= 1.0.13, < 2.0.0',
    ],

yang.ncdiff:

yang.ncdiff uses cisco-gnmi.proto now so yang/ncdiff/src/yang/ncdiff/proto/* modules are no longer needed. The proto is installed from grpcio. I switched yang.ncdiff to using cisco-gnmi.proto because I wanted yang.connector and yang.ncdiff to use the same "proto". On another note, why does ncdiff require yang.connector >= 3.0.0?

    install_requires =  [
        'pyang >= 1.7.3',
        'ncclient >= 0.6.3, < 0.6.10',
        'requests >= 2.18.4',
        'cisco-gnmi',
        'xmljson >= 0.1.9',
        'yang.connector >= 3.0.0',
    ],

The best solution going forward would be to remove grpcio from the yang.connector setup.py requirement.

NETCONF does not work with EnXR simulator as target test device.

EnXR simulator communicates through Unix sockets instead of a network connection. The pyATS Unicon library works with CLI connections but not NETCONF and gNMI.

Error from xpath_util.xml_path_to_path_elem()

If a namespace is passed in that does not match Cisco or openconfig namespace this traceback happens:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/miott/ysuite/install/yangsuite/venv/lib/python3.7/site-packages/yang/connector/xpath_util.py", line 272, in xml_path_to_path_elem
    if module:
UnboundLocalError: local variable 'module' referenced before assignment

RESTCONF support?

Is there a plan to add support for the RESTCONF protocol?
Are there other python packages that support YANG with RESTCONF?

rpc-reply should be presented in the log with proper indentation and newlines

The rpc-reply text in the logs have no linefeeds or spaces making the message awkward to read.

Logging module "settings.py" causes pyats install requirement

Most modules in yang.connector can be instantiated as standalone. The from pyats.datastructures import AttrDict import in settings.py requires pyats install for all libraries that import yang.connector.

yang-connector (19.7) or ncclient (0.6.6) fail to create_subscription to NSO 5.2

yang-connector (19.7) or ncclient (0.6.6) generate different XML versus the one generated by netconf-console (which work fine), ie:

NETCONF CONSOLE (CLIENT):

MBERETTA-M-K1L8:~ mberetta$ netconf-console -u CISCO15 -p Otbu+2019 --host 10.58.230.45 --port 830 -i
netconf> create-subscription ncs-events
<ok xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0"/>

netconf> 

NETCONF CONSOLE (SERVER):

<nc:rpc message-id="urn:uuid:5854c9a7-5b4f-4d2c-921b-628d0d218533" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ns0:create-subscription xmlns:ns0="urn:ietf:params:xml:ns:netconf:notification:1.0">
    <ns0:stream>ncs-events</ns0:stream>
  </ns0:create-subscription>
</nc:rpc>

6-Nov-2019::16:11:30.378 **< sess:336 write:
<rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:5854c9a7-5b4f-4d2c-921b-628d0d218533" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

PYATS (CLIENT):

>>> svo.create_subscription(stream_name="ncs-events")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gtandi/pyats/lib/python3.6/site-packages/yang/connector/__init__.py", line 410, in execute
    return super().execute(cls, *args, **kwargs)
  File "/Users/gtandi/pyats/lib/python3.6/site-packages/ncclient/manager.py", line 236, in execute
    huge_tree=self._huge_tree).request(*args, **kwds)
  File "/Users/gtandi/pyats/lib/python3.6/site-packages/ncclient/operations/subscribe.py", line 64, in request
    return self._request(node)
  File "/Users/gtandi/pyats/lib/python3.6/site-packages/ncclient/operations/rpc.py", line 349, in _request
    raise self._reply.error
ncclient.operations.rpc.RPCError: {'type': 'protocol', 'tag': 'unknown-element', 'severity': 'error', 'info': '<?xml version="1.0" encoding="UTF-8"?><error-info xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0"><bad-element>stream</bad-element>\n</error-info>\n', 'path': '\n    /rpc/p36:create-subscription\n  ', 'message': None}
>>>

PYATS (SERVER):

<nc:rpc message-id="urn:uuid:90297abb-c47d-486b-91fb-e3f6ed632dd9" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ns0:create-subscription xmlns:ns0="urn:ietf:params:xml:ns:netconf:notification:1.0">
    <nc:stream>ncs-events</nc:stream>
  </ns0:create-subscription>
</nc:rpc>
 
6-Nov-2019::11:03:42.743 **< sess:131 write:
<rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:90297abb-c47d-486b-91fb-e3f6ed632dd9" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <rpc-error>
    <error-type>protocol</error-type>
    <error-tag>unknown-element</error-tag>
    <error-severity>error</error-severity>
    <error-path xmlns:p36="urn:ietf:params:xml:ns:netconf:notification:1.0">
    /rpc/p36:create-subscription
  </error-path>
    <error-info>
      <bad-element>stream</bad-element>
    </error-info>
  </rpc-error>
</rpc-reply>

Conditional imports for gnmi.py not in 23.11 wheel file.

#110 release 23.11 was created. I assume that a yang.connector-23.11.whl file was supposed to be uploaded to pypi.org. #107 adding some conditional imports was committed 3 weeks prior to #110. Even though the conditional imports are correct in the repository, if I pip install yang.connector==23.11 the conditional imports are not there. It appears the wrong wheel file was uploaded to pypi.org.

Cannot use yang.connector library without installing pyats

Created a virtualenv and installed yang.connector. Tried to import a yang.connector class and an error that unicon is missing was reported. Tried adding unicon and got an error saying pyats is missing.

(yang) ~/pyats/install # pip list
Package        Version
-------------- -------
bcrypt         4.0.1
cffi           1.16.0
cryptography   41.0.5
grpcio         1.59.2
lxml           4.9.3
ncclient       0.6.15
paramiko       3.3.1
pip            23.3.1
protobuf       4.25.0
pycparser      2.21
PyNaCl         1.5.0
setuptools     68.2.2
six            1.16.0
wheel          0.41.3
yang.connector 23.10
(yang) ~/pyats/install # python
Python 3.8.16 (default, Dec  7 2022, 01:27:54) 
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from yang.connector import Gnmi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/__init__.py", line 22, in <module>
    from .gnmi import Gnmi
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/gnmi.py", line 10, in <module>
    from unicon.sshutils import sshtunnel
ModuleNotFoundError: No module named 'unicon'
>>> from yang.connector import Netconf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/__init__.py", line 22, in <module>
    from .gnmi import Gnmi
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/gnmi.py", line 10, in <module>
    from unicon.sshutils import sshtunnel
ModuleNotFoundError: No module named 'unicon'
>>> 

Installed unicon and tried again.

Installing collected packages: wcwidth, pyyaml, PrettyTable, dill, unicon.plugins, unicon
Successfully installed PrettyTable-3.9.0 dill-0.3.7 pyyaml-6.0.1 unicon-23.10 unicon.plugins-23.10 wcwidth-0.2.9
(yang) ~/pyats/install # python            
Python 3.8.16 (default, Dec  7 2022, 01:27:54) 
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from yang.connector import Netconf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/__init__.py", line 24, in <module>
    from .grpc import Grpc
  File "/Users/miott/pyats/install/yang/lib/python3.8/site-packages/yang/connector/grpc/__init__.py", line 6, in <module>
    from pyats.connections import BaseConnection
ModuleNotFoundError: No module named 'pyats.connections'
>>> 

Installed yang.connector==23.9 and issue went away.

Installing collected packages: yang.connector
  Attempting uninstall: yang.connector
    Found existing installation: yang.connector 23.10
    Uninstalling yang.connector-23.10:
      Successfully uninstalled yang.connector-23.10
Successfully installed yang.connector-23.9
(yang) ~/pyats/install # python                          
Python 3.8.16 (default, Dec  7 2022, 01:27:54) 
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from yang.connector import Netconf
>>> 

After investigating, in 23.10, "grpc" was added which has imports required from pyats and the connector/gnmi.py added an import from unicon.

Need documentation added for gnmi feature.

A gnmi.py module was added containing a new Gnmi class. The usage for this class is well documented in the docstrings of the module but this needs to be added to connector documentation.

yang_snapshot_restore causes traceback using NetconfEnxr class.

yang_snapshot_restore is accessing a locked member in the NetconfEnxr class that does not exist.

Please update readthedocs documentation

Hi, Jonathan.

We have made documentation updates but don't have the procedure to update readthedocs documentation. We updated this package to support the new credentials feature in pyATS v19.7. Could you either update these docs or provide us a way to do so?

Thanks.

breakage seen with newly released pyats v19.5 packages

yang.connector is not handling the case where self.connectioninfo now contains the "credentials" member. The credentials feature was released as part of pyATS v19.5.

$ pip list
Package Version

---

asn1crypto 0.24.0
asyncssh 1.16.1
bcrypt 3.1.4
certifi 2018.10.15
cffi 1.11.5
chardet 3.0.4
cryptography 2.6.1
defusedxml 0.5.0
dill 0.2.9
distro 1.4.0
httplib2 0.11.3
idna 2.7
Jinja2 2.10
jira 2.0.0
junit-xml 1.8
lxml 4.2.5
MarkupSafe 1.1.0
ncclient 0.5.3
netconf-console 2.1.1
oauthlib 3.0.1
paramiko 2.4.2
pathspec 0.5.9
pbr 5.1.3
pexpect 4.7.0
pip 19.1.1
psutil 5.4.8
ptyprocess 0.6.0
pyasn1 0.4.4
pyats 19.5.0
pyats.aereport 19.5.0
pyats.aetest 19.5.0
pyats.async 19.5.0
pyats.connections 19.5.0
pyats.datastructures 19.5.0
pyats.easypy 19.5.0
pyats.examples 19.5.0
pyats.kleenex 19.5.0
pyats.log 19.5.0
pyats.results 19.5.0
pyats.tcl 19.5.0
pyats.templates 19.5.0
pyats.topology 19.5.0
pyats.utils 19.5.0
pycparser 2.19
PyJWT 1.7.1
PyNaCl 1.3.0
PyYAML 3.13
requests 2.20.0
requests-oauthlib 1.2.0
requests-toolbelt 0.9.1
setproctitle 1.1.10
setuptools 39.0.1
six 1.11.0
unicon 19.5.0
urllib3 1.24.1
xmltodict 0.11.0
yamllint 1.15.0
yang.connector 3.0.0
(pyats) MBERETTA-M-K1L8:~ mberetta$ cat svo/temp/svo_100.yaml
devices:
ncs:
type: nso
os: linux
tacacs:
username: admin
passwords:
tacacs: admin
enable: admin
line: admin
connections:
defaults:
class: 'unicon.Unicon'
via: netconf
admin:
protocol: https
ip: 10.58.228.93
port: 8443
username: CISCO15
password: otbu+2019
host:
protocol: ssh
ip: 10.58.228.93
port: 22
username: root
password: lablab
netconf:
class: yang.connector.Netconf
ip : 10.58.228.100
port: 830
username: CISCO15
password: otbu+2019
hostkey_verify: False
look_for_keys: False
timeout: 120
custom:
host_type: Card
(pyats) MBERETTA-M-K1L8:~ mberetta$
(pyats) MBERETTA-M-K1L8:~ mberetta$
(pyats) MBERETTA-M-K1L8:~ mberetta$ python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from ats.topology import loader
testbed = loader.load('svo/temp/svo_100.yaml')
svo = testbed.devices['ncs']
svo.connect(via='netconf')
/Users/mberetta/pyats/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.ecdsa_curve.curve_class(), pointinfo
Traceback (most recent call last):
File "", line 1, in
File "src/pyats/connections/manager.py", line 305, in pyats.connections.manager.ConnectionManager.connect
File "/Users/mberetta/pyats/lib/python3.6/site-packages/yang/connector/init.py", line 324, in connect
self.session.connect(**defaults)
TypeError: connect() got an unexpected keyword argument 'credentials'

Invalid proto descriptor for file "gnmi.proto".

If a library using grpcio is already installed the following error occurs when using yang.connector or yang.ncdiff.

2020-06-10T23:00:06: %EASYPY-ERROR: TypeError: Couldn't build proto file into descriptor pool!
2020-06-10T23:00:06: %EASYPY-ERROR: Invalid proto descriptor for file "gnmi.proto":
2020-06-10T23:00:06: %EASYPY-ERROR:   gnmi.proto: A file with this name is already in the pool.

The grpcio-tools library can be used to compile gNMI protobufs. If a proto file is already compiled and installed, this error occurs. Need a solid way to avoid this issue.

Latest version of yangsuite has --https option so ngnx is no longer needed.

The yangsuite --https option will activate https in the internal web server. The docker-compose implementation should take advantage of that option and remove the nginx container as a requirement.

Gnmi notification tests do not work

Max timeout is always taking default value.
Subscribe works but nothing in stream is evaluated.

Using ncclient 0.6.4+ the rpc-reply no longer shows up in the log

Updating ncclient to 0.6.7, rpc-reply messages only show up in the DEBUG log which is very chatty. This was the result of a change to logging that came in 0.6.4.

Add yang.verifiers documentation to developer.cisco.com.

The yang.connector docs are published at https://developer.cisco.com/docs/yang-connector/. The new YANG verifier feature, yang.verifiers, should also have documentation published at https://developer.cisco.com/docs/yang-verifiers/.

yangsuite-gnmi fails to load because of unicon error

Docker error:

docker-yangsuite-1  | ERROR:root:Error in loading YANG Suite app "yangsuite-gnmi": No module named 'unicon'

Issue is caused by changes made in yang.connector version 23.10 and an issue has been opened.
#104

Workaround:

docker exec -it docker-yangsuite-1 bash
root@c1e82c5fc8c7:/# pip install yang.connector==23.9
exit
docker stop docker-yangsuite-1
docker start docker-yangsuite-1

gNMI should use new method of credential password modeling.

The gNMI protocol should use the new method of credential password modeling to allow users to hide passwords and usernames on target test devices.