Cisco WLC support about netmiko

[jayswan]

When using this device type I get socket.error: Socket is closed from paramiko before "Interactive SSH session established" about 50% of the time. Re-running the script always works eventually.

Traceback (most recent call last):
  File "xmit_power.py", line 115, in <module>
    main()
  File "xmit_power.py", line 90, in main
    with ssh_manager(device) as conn:
  File "c:\Anaconda\lib\contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "xmit_power.py", line 20, in ssh_manager
    conn = SSHClient(**net_device)
  File "c:\Anaconda\lib\site-packages\netmiko\base_connection.py", line 44, in __init__
    self.establish_connection(verbose=verbose, use_keys=use_keys)
  File "c:\Anaconda\lib\site-packages\netmiko\cisco\cisco_wlc_ssh.py", line 64, in establish_connection
    self.remote_conn.sendall(self.password + '\n')
  File "c:\Anaconda\lib\site-packages\paramiko\channel.py", line 744, in sendall
    sent = self.send(s)
  File "c:\Anaconda\lib\site-packages\paramiko\channel.py", line 698, in send
    return self._send(s, m)
  File "c:\Anaconda\lib\site-packages\paramiko\channel.py", line 1058, in _send
    raise socket.error('Socket is closed')
socket.error: Socket is closed

Code is at https://github.com/jayswan/wlc_scripting

I've tested on both Windows and OS X to make sure it's not a Windows-specific paramiko problem.

[ktbyers]

@jayswan I recall that the WLC puts up two login prompts that I had to put a workaround in for.

Can you see if you can capture and print out what happens on the SSH session up to this point (i.e. what the WLC sends as output before the SSH session closes)?

[jayswan]

How would you suggest doing this? The exception occurs at line 64 of cisco_wlc_ssh.py, when the password is sent. The immediate previous call, self.remote_conn.sendall(self.username + '\n'), returns None.

[ktbyers]

Let me look into it a bit more.

Kirk

On Tue, Aug 18, 2015 at 8:29 AM, Jay Swan [email protected] wrote:

How would you suggest doing this? The exception occurs at line 64 of
cisco_wlc_ssh.py, when the password is sent. The immediate previous call, self.remote_conn.sendall(self.username

• '\n'), returns None.

—
Reply to this email directly or view it on GitHub
#66 (comment).

Kirk Byers
CCIE #6243 emeritus
[email protected]
http://pynet.twb-tech.com http://pynet.twb-tech.com/blog
[phone]
Simplify your network management through automation

[ktbyers]

@jayswan I think this was probably being caused by the code that tries to handle the extra 'login as' that some WLCs display at login.

I have improved this code and hopefully fixed the problem. I couldn't test it, however, because the WLC that I was testing on (running 7.6.130.0) only displayed 'User:' and 'Password:' on login.

Let me know if you still see the issue.

[ktbyers]

@jayswan I am somewhat doubtful that the fix that I put in fixes what you were seeing, but I am unable to reproduce your behavior. If you see it again, let me know.

[mbs91]

@ktbyers,
I'm having the same issue as jayswa. I'm getting "socket.error: Socket is closed" when using 'cisco_wlc' as the device_type 100% of the time. the WLC I'm testing on is running the 8.0.110.0 version.

[ktbyers]

@mahabs What version of Netmiko are you using?

import netmiko
netmiko.__version__

[mbs91]

@ktbyers,

netmiko.version
'0.3.4'

[ktbyers]

@mahabs I just pushed Netmiko 0.4.3 to pypi. Can you re-test and see if you are still seeing the same error?

[mbs91]

@ktbyers, I'm getting the same error.

net_connect = ConnectHandler(device_type="cisco_wlc", ip=wlc_ip, username=username, password=password)
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/netmiko/ssh_dispatcher.py", line 88, in ConnectHandler
return ConnectionClass(_args, *_kwargs)
File "/Library/Python/2.7/site-packages/netmiko/base_connection.py", line 68, in init
self.establish_connection(verbose=verbose, use_keys=use_keys, key_file=key_file)
File "/Library/Python/2.7/site-packages/netmiko/base_connection.py", line 182, in establish_connection
self.special_login_handler()
File "/Library/Python/2.7/site-packages/netmiko/cisco/cisco_wlc_ssh.py", line 31, in special_login_handler
self.remote_conn.sendall(self.username + '\n')
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 744, in sendall
sent = self.send(s)
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 698, in send
return self._send(s, m)
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 1058, in _send
raise socket.error('Socket is closed')
socket.error: Socket is closed

[ktbyers]

@mahabs Can you show me what the session looks like when you manually SSH into it (i.e. capture all the text that happens on a manual SSH)?

Please strip out anything that is confidential password, IP address, etc.

[mbs91]

@ktbyers ,

[ktbyers]

Okay, we are probably going to need to try to get some additional information. Can you add the print statement to netmiko/netmiko/cisco/cisco_wlc_ssh.py (this will be on line 30):

        while i <= 12:
            if self.remote_conn.recv_ready():
                output = self.remote_conn.recv(MAX_BUFFER).decode('utf-8', 'ignore')
                print(output)       # new statement

Let me know what you see for output after you do this?

[tima-fey]

Hi!
I have the same problem
I've added print statement and
my script is just 3 lines

from netmiko import ConnectHandler 
cisco_wlc = { 'device_type':'cisco_wlc', 'ip':'x.x.x.x', 'username':'test-user', 'password':'test'}
net_connect = ConnectHandler(**cisco_wlc)

my output is

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-9-d206b184aa37> in <module>()
----> 1 net_connect = ConnectHandler(**cisco_wlc)

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/netmiko/ssh_dispatcher.pyc in ConnectHandler(*args, **kwargs)
     86                          'currently supported platforms are: {0}'.format(platforms_str))
     87     ConnectionClass = ssh_dispatcher(kwargs['device_type'])
---> 88     return ConnectionClass(*args, **kwargs)
     89 
     90 

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/netmiko/base_connection.pyc in __init__(self, ip, host, username, password, secret, port, device_type, verbose, global_delay_factor, use_keys, key_file, ssh_strict, system_host_keys, alt_host_keys, alt_key_file, ssh_config_file)
     66         self.ssh_config_file = ssh_config_file
     67 
---> 68         self.establish_connection(verbose=verbose, use_keys=use_keys, key_file=key_file)
     69         self.session_preparation()
     70 

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/netmiko/base_connection.pyc in establish_connection(self, sleep_time, verbose, timeout, use_keys, key_file)
    180         self.remote_conn = self.remote_conn_pre.invoke_shell()
    181         self.remote_conn.settimeout(timeout)
--> 182         self.special_login_handler()
    183         if verbose:
    184             print("Interactive SSH session established")

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/netmiko/cisco/cisco_wlc_ssh.pyc in special_login_handler(self, delay_factor)
     29                 output = self.remote_conn.recv(MAX_BUFFER).decode('utf-8', 'ignore')
     30                 print(output)       # new statement
---> 31                 if 'login as' in output or 'User' in output:
     32                     self.remote_conn.sendall(self.username + '\n')
     33                 elif 'Password' in output:

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/paramiko/channel.pyc in sendall(self, s)
    742         """
    743         while s:
--> 744             sent = self.send(s)
    745             s = s[sent:]
    746         return None

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/paramiko/channel.pyc in send(self, s)
    696         m.add_byte(cMSG_CHANNEL_DATA)
    697         m.add_int(self.remote_chanid)
--> 698         return self._send(s, m)
    699 
    700     def send_stderr(self, s):

/Users/markovtimofey/anaconda/lib/python2.7/site-packages/paramiko/channel.pyc in _send(self, s, m)
   1056             if self.closed:
   1057                 # this doesn't seem useful, but it is the documented behavior of Socket
-> 1058                 raise socket.error('Socket is closed')
   1059             size = self._wait_for_send_window(size)
   1060             if size == 0:

error: Socket is closed

And thanks a lot for Great job!

[mbs91]

@ktbyers ,

I did as requested and here is the output I got:

(Cisco Controller)
User:
User:Raw mode will not be supported, Closing connection.

User:
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/netmiko/ssh_dispatcher.py", line 88, in ConnectHandler
return ConnectionClass(_args, *_kwargs)
File "/Library/Python/2.7/site-packages/netmiko/base_connection.py", line 68, in init
self.establish_connection(verbose=verbose, use_keys=use_keys, key_file=key_file)
File "/Library/Python/2.7/site-packages/netmiko/base_connection.py", line 182, in establish_connection
self.special_login_handler()
File "/Library/Python/2.7/site-packages/netmiko/cisco/cisco_wlc_ssh.py", line 32, in special_login_handler
self.remote_conn.sendall(self.username + '\n')
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 744, in sendall
sent = self.send(s)
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 698, in send
return self._send(s, m)
File "/Library/Python/2.7/site-packages/paramiko/channel.py", line 1058, in _send
raise socket.error('Socket is closed')
socket.error: Socket is closed

Thanks!

[ktbyers]

@tima-fey @mahabs Okay, I will have to look at this. Remind me in mid-May if I forget (I probably won't get to it before then).

[petermoorey]

I'm also getting socket closed when connecting over higher latency links 250ms. This is remediated by modifying the delay factor. WLCs are running 8.1.131.0, 8.0.131.0 and 7.6.130.0. Netmiko version 0.4.3.

Traceback (most recent call last):
File "wlc-capture.py", line 60, in
net_connect = ConnectHandler(device_type='cisco_wlc', ip=ip, username=usr, password=pwd)
File "C:\Python27\lib\site-packages\netmiko\ssh_dispatcher.py", line 88, in ConnectHandler
return ConnectionClass(_args, *_kwargs)
File "C:\Python27\lib\site-packages\netmiko\base_connection.py", line 68, in init
self.establish_connection(verbose=verbose, use_keys=use_keys, key_file=key_file)
File "C:\Python27\lib\site-packages\netmiko\base_connection.py", line 182, in establish_connection
self.special_login_handler()
File "C:\Python27\lib\site-packages\netmiko\cisco\cisco_wlc_ssh.py", line 33, in special_login_handler
self.remote_conn.sendall(self.password + '\n')
File "C:\Python27\lib\site-packages\paramiko\channel.py", line 761, in sendall
sent = self.send(s)
File "C:\Python27\lib\site-packages\paramiko\channel.py", line 715, in send
return self._send(s, m)
File "C:\Python27\lib\site-packages\paramiko\channel.py", line 1075, in _send
raise socket.error('Socket is closed')
socket.error: Socket is closed

Please let me know if I can help with any testing.

Pete.

[ktbyers]

I wonder if we just increase the delay whether it all gets better.

Can someone test changing line 35 in ./netmiko/netmiko/cisco/cisco_wlc_ssh.py

    def special_login_handler(self, delay_factor=.5):
        '''
        WLC presents with the following on login (in certain OS versions)
        login as: user
        (Cisco Controller)
        User: user
        Password:****
        '''
        delay_factor = self.select_delay_factor(delay_factor)
        i = 0
        while i <= 12:
            if self.remote_conn.recv_ready():
                output = self.remote_conn.recv(MAX_BUFFER).decode('utf-8', 'ignore')
                if 'login as' in output or 'User' in output:
                    self.remote_conn.sendall(self.username + '\n')
                elif 'Password' in output:
                    self.remote_conn.sendall(self.password + '\n')
                    break
                time.sleep(1)           # test where sleep time is increased
            else:
                self.remote_conn.sendall('\n')
                time.sleep(delay_factor * 2)
            i += 1

[petermoorey]

Changing line 35 to '1' results in success for a the WLC 250ms away, login is consistent and configuration capture works.

For a WLC 650ms away (satellite link) the capture fails, I found that changing the base delay factor (line 13) from '.5' to '1' results in success every time. Maybe we don't care too much about this scenario, it's niche...

Thanks,
Pete.

[ktbyers]

@petermoorey Okay, I pushed a commit up to the master branch

a62ea88

Can you test if this fixes the issue for you (in both case 250mS delay and 650mS delay).

Also let me know if it is really slow to login.

[ktbyers]

@mahabs @tima-fey Either of you willing to test this fix. Just need to clone from the master branch in GitHub.

Let me know if it fixes your issue or not.

[petermoorey]

Hi, I can test it in a couple of weeks, I just left for vacation...

Pete

[ktbyers]

@petermoorey No problem...

[ktbyers]

From @petermoorey

I'm finding I get quite a few instances of router prompt not found when logging into Cisco WLCs with latency >200ms, I think it's a timing issue.  Do you happen to know which delay factor I could try adjust, I think it's a similar issue this one: https://github.com/ktbyers/netmiko/issues/66
 GitHub
Cisco WLC support · Issue #66 · ktbyers/netmiko · GitHub
Do I need to re-enable paging before I disconnect from device (i.e. is the paging change actually stored in config)? Verify WLC has no enable mode. Verify WLC has no config mode. Prompt typically e... 



[2:33]  
I think it's affecting roughly one in fifteen devices, not huge but when I am applying commands to 150 devices I either have to run it twice, or manually deal with those offenders.  Usually on the second try they are fine...

[2:34]  
let me know if there is a debug I can run, happy to provide any information.  I can dig around the code myself and test some stuff when I get time

bedecarroll [3:56 PM]  
@petermoorey: I found the same thing, I put a delay_factor of 0.5 and it got much better

[ktbyers]

Pushed this change which I hope improves this:

770cadc

[ktbyers]

I am going to close this. If there are still issues with Cisco WLC will create a new issue to track it.

[xxddpac]

@ktbyers
netmiko version 2.3.3
connect Cisco WLC output “socket.error: Socket is closed”

[ktbyers]

@xxddpac There is not really anything I can do with that...there is not enough context to give you any guidance.

If you create a separate issue, post your code, and your full exception stack we might be able to help you.

Regards, Kirk