Error 111 - Connection Refused about spur.py

[mwilliamson]

Hmm, the fact that the connection is refused suggests that it's failing to even start a TCP connection. The stack trace from the original error from paramiko has been cut off by spur, which isn't very helpful -- I'll see if I can fix that and release a new version. In the meantime:

• Can you connect to localhost using a private key?
• Could you paste in the command you used to manually connect?

[tgrosinger]

I have tried a few combinations for connecting manually.

ssh -o 'StrictHostKeyChecking no' -i /home/tony/.ssh/grosinger-west [email protected]

ssh -q -i /home/tony/.ssh/grosinger-west [email protected]

When trying to connect locally using a key I receive this error:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    result = shell.run(["echo", "-n", "hello"])
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 71, in run
    return self.spawn(*args, **kwargs).wait_for_result()
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 80, in spawn
    channel = self._get_ssh_transport().open_session()
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 176, in _get_ssh_transport
    raise self._connection_error(error)
spur.ssh.ConnectionError: Error creating SSH connection
Original error: Server 'localhost' not found in known_hosts

[mwilliamson]

I've just uploaded spur 0.3.2, which should hopefully include the traceback for the original error. Could you wrap your original code in a try...except block and see what the value of original_traceback is? In other words, something like:

import spur
import spur.ssh

try:
    shell = spur.SshShell(
        hostname=host,
        username=settings['ami_default_username'],
        private_key_file=settings['aws_ssh_key'],
        missing_host_key=spur.ssh.MissingHostKey.accept
    )
    shell.run(["true"])
except spur.ssh.ConnectionError as error:
    print error.original_traceback
    raise

Also, have you tried hard-coding the arguments to SshShell? Something like:

import spur
import spur.ssh

try:
    shell = spur.SshShell(
        hostname="ec2-54-245-69-99.us-west-2.compute.amazonaws.com",
        username="ubuntu",
        private_key_file="/home/tony/.ssh/grosinger-west",
        missing_host_key=spur.ssh.MissingHostKey.accept
    )
    shell.run(["true"])
except spur.ssh.ConnectionError as error:
    print error.original_traceback
    raise

What code did you use for trying to connect to localhost? I was thinking of something like:

import spur
import spur.ssh

try:
    shell = spur.SshShell(
        hostname="localhost",
        username="tony",
        private_key_file="/home/tony/.ssh/grosinger-localhost",
        missing_host_key=spur.ssh.MissingHostKey.accept
    )
    shell.run(["true"])
except spur.ssh.ConnectionError as error:
    print error.original_traceback
    raise

Where /home/tony/.ssh/grosinger-localhost is the path to a file that contains a private key that allows you to log in to your local machine.

[tgrosinger]

Here is the full error:

Traceback (most recent call last):
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 178, in _get_ssh_transport
    return self._connect_ssh().get_transport()
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 195, in _connect_ssh
    timeout=self._connect_timeout
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/paramiko/client.py", line 301, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/paramiko/util.py", line 278, in retry_on_signal
    return function()  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/paramiko/client.py", line 301, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

Traceback (most recent call last):
  File "deploy.py", line 222, in <module>
    downloadFiles(conn, instanceDNS)
  File "deploy.py", line 85, in downloadFiles
    shell.run(["true"])
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 72, in run
    return self.spawn(*args, **kwargs).wait_for_result()
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 82, in spawn
    channel = self._get_ssh_transport().open_session()
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/spur/ssh.py", line 180, in _get_ssh_transport
    raise self._connection_error(error)
spur.ssh.ConnectionError: Error creating SSH connection
Original error: [Errno 111] Connection refused

I updated the connection to localhost using your example. Here is the output.

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/paramiko/transport.py", line 1621, in run
  File "/home/tony/virtualEnvironments/aws/lib/python2.7/site-packages/paramiko/transport.py", line 1404, in _log
  File "/usr/lib/python2.7/logging/__init__.py", line 1203, in log
  File "/usr/lib/python2.7/logging/__init__.py", line 1257, in _log
  File "/usr/lib/python2.7/logging/__init__.py", line 1231, in makeRecord
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

[mwilliamson]

Hmm, definitely looks like it's failing entirely to connect to the port. Did you try hard-coding the arguments to SshShell?

It looks like the connection to localhost is working, but I realised that a successful output doesn't actually print anything. Could you try:

import spur
import spur.ssh

try:
    shell = spur.SshShell(
        hostname="localhost",
        username="tony",
        private_key_file="/home/tony/.ssh/grosinger-localhost",
        missing_host_key=spur.ssh.MissingHostKey.accept
    )
    with shell:
        print shell.run(["echo", "hello"]).output
except spur.ssh.ConnectionError as error:
    print error.original_traceback
    raise

The output should just be hello. Using the shell as a context manager should close the SSH connection, which should prevent the error that you saw when trying it out localhost.

Assuming that it still doesn't work with the hard-coded arguments, and that you can successfully connect to localhost with a private key, I'm not really sure what else there is to try.

[tgrosinger]

Alright, I have no idea what is going on. I believe the issue is with the host that I am specifying. I am retrieving it from the aws instance that is being started and sticking it in directly in the connection object.

When I hard code in the host instead it does seem to connect. Sadly, the whole idea for the program is creating and building an environment in EC2 so the hostname will be different every time.

[mwilliamson]

If you print out the value of host, what do you get?

[tgrosinger]

That's the very confusing part, it prints out exactly what I would expect. The host name changes every time, but I confirm it against the aws control panel and it does match. That is also what I use when testing connecting manually.

[mwilliamson]

What happens if you try something like:

print "ec2-54-245-69-99.us-west-2.compute.amazonaws.com" == host
print type(host)

(with the appropriate hard-coded string)

If it prints True and <type 'str'>, then I can't explain what's going on! If it prints False, then perhaps trying knocking together some code to tell you exactly what the difference is (or reuse an assert_equal function from a testing framework)

[mwilliamson]

Also, thinking a bit more about it: when are you starting the machines? Is it possible that you're trying to connect before SSH is ready? But by the time you try manually, the machine has loaded and SSH is ready?

[tgrosinger]

Well I made one last script for a definitive test (btw, really appreciate your help!)
It does return true and the type is unicode (tried casting to string with no difference in result)

https://gist.github.com/tgrosinger/4725887

The authentication error is confusing. Again, testing manually with the same host and key works fine. So confused.

Edit: I have tried on a machine that was existing for a few minutes, so I know that ssh was ready. I also had it wait until the machine reported 'running' + 10 seconds on top. I always remove the host from the known_hosts file before running a test since that is how it will be in the final product.

[mwilliamson]

What happens if you retry?

https://gist.github.com/mwilliamson/4726047

[tgrosinger]

Sadly it gives up after trying for the allotted minute. After connecting manually and adding the host to the known_hosts file the script still produces the same results. Connecting manually is successful.

[mwilliamson]

Is that still with an authentication error? I just noticed that the username argument in the gist is "Tony", whereas your earlier successful manual command used "ubuntu".

Otherwise, I'm afraid I'm stumped!

[tgrosinger]

That's pretty embarrassing. Changing the username on the script you wrote does cause it to work. I added the retry to my main program and it fails for about 10 seconds but then does in fact connect!

Thank you so much for your help, I will definitely mention you.

So it is working, but I thought I would show you this output to see if it makes more sense to you. The two [INFO, ERROR, ERROR, Original error] blocks at the beginning are it retrying. It then retries again and connects however the next several lines are not created by me. The word "hello" is the command succeeding and finally "Connected!" is when control is returned to my application.

INFO: Attempting to connect...
ERROR: Attempt failed
ERROR: Error creating SSH connection
Original error: [Errno 111] Connection refused
INFO: Attempting to connect...
ERROR: Attempt failed
ERROR: Error creating SSH connection
Original error: [Errno 111] Connection refused
INFO: Attempting to connect...
DEBUG: starting thread (client mode): 0x2401490L
INFO: Connected (version 2.0, client OpenSSH_5.9p1)
DEBUG: kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss', 'ecdsa-sha2-nistp256'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', '[email protected]'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', '[email protected]'] client mac:['hmac-md5', 'hmac-sha1', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', '[email protected]'] server compress:['none', '[email protected]'] client lang:[''] server lang:[''] kex follows?False
DEBUG: Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
DEBUG: using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
DEBUG: Switch to new keys ...
DEBUG: Trying key a16ad9c670909d2ad02c6d89ae017b52 from /home/tony/.ssh/grosinger-west
DEBUG: userauth is OK
INFO: Authentication (publickey) successful!
DEBUG: [chan 1] Max packet in: 34816 bytes
DEBUG: [chan 1] Max packet out: 32768 bytes
INFO: Secsh channel 1 opened.
DEBUG: [chan 1] Sesch channel 1 request ok
DEBUG: [chan 1] EOF received (1)
DEBUG: [chan 1] EOF sent (1)
hello

DEBUG: Dropping user packet because connection is dead.
DEBUG: Dropping user packet because connection is dead.
INFO: Connected!

[mwilliamson]

Excellent, glad I could help! The INFO and DEBUG lines are probably generated by paramiko, which is the library that spur uses for SSH.

[mwilliamson]

Oh, I forgot to say: if you're using that retry, depending on how quick each iteration round the loop is, you might want to consider sticking a time.sleep at the end of each loop iteration.

[tgrosinger]

Yes, I currently have it set to sleep for 0.1 seconds in between tries but I will probably turn that up to a quarter or even half when I go back and clean up all the code. Full project is here if you're curious what this is all about.