Bolt SSH Key Verification Failure about bolt

[ahpook]

At SF puppetizelive - @torrancew mentioned that he's running into this problem when there's an EC-DSA key that overlaps with an existing (erroneous) RSA key - net::ssh prefers the RSA key (even when configured not to), which causes the known_hosts mismatch error. there is an upstream bug open on this - net-ssh/net-ssh#627 but it's not getting much traction. either helping them fix it upstream or working around it in bolt would help a lot! 🥇

[darkn3rd]

That would be great to defer to ssh-config (or ~/.ssh/config) settings used by the underlying OpenSSH tool. I hope we can have more exposure in the inventory settings, and separate out a inventory name vs. actual hostname or IP address. This way I can meta generate per environment (local dev vagrant vs. google cloud vs. aws vs. ... ) on the fly, and not have to aggregate out explicit IP address or hostnames: when I have, prod-web-01 vs stage-web-01 vs. web-01.dev, but want to configure web-01 per environment.

[blackknight36]

Was this ever fixed in bolt? I'm getting the same error when I try to run a task on some of our CentOS nodes.

Failed on host.example.com:
  Host key verification failed for host.example.com: fingerprint 9c:e6:8c:c8:13:01:9a:42:c9:73:49:7f:48:bd:1e:7a is unknown for "host.example.com,192.168.0.5"

Adding --no-host-key-check to the bolt command works but should not be necessary.

[darkn3rd]

Cannot discern this from the debug output, whether ignoring Port, StrictHostKeyChecking, or other setting. In my setup, I also configured to use private IP addresses, and can use this as an alternative to get into the system, I would try --nodes 127.0.0.1 --port 2222, but there's no support for a port option.

bolt command run 'hostname' \
  --nodes 172.16.0.30 \
  --user vagrant \
  --private-key .vagrant/machines/tools.dev/virtualbox/private_key

[adreyer]

Bolt has it's own options for host key checking since net::ssh does not support that from ssh config.

Set --no-host-key-check on the cli or host-key-check: false in config or inventory.

[MikaelSmith]

You can also pass a port as part of --nodes, such as --nodes 127.0.0.1:2222.

[darkn3rd]

But it is ignoring this setting from the SSH configuration? If SSH is configured this way, why would we have to specify it again?

[adreyer]

Fair enough, It's a shortcoming of the underlying library we use but there is no reason we can't add it.

[donoghuc]

@blackknight36 Are you asking if picking up StrictHostKeyChecking from OpenSSH config files is supported? This is not currently available in the underlying net-ssh gem (https://net-ssh.github.io/net-ssh/classes/Net/SSH/Config.html ). We are working on documenting what OpenSSH options are configurable outside of bolt config ( https://tickets.puppetlabs.com/browse/BOLT-1227 ).

[blackknight36]

@donoghuc I do not have StrictHostKeyChecking enabled in my .ssh/config and the default setting is set to "ask" which works fine with the regular ssh command. Does net-ssh work with the /etc/ssh/known_hosts file? All of our host keys are exported by puppet and collected into this file any time the host key on a node changes.

[donoghuc]

Yes, net-ssh will respect known_hosts. I am not 100% if that location (/etc/ssh) is a default (for me its ~/.ssh/known_hosts). You can configure it with:

Host *example.com
  UserKnownHostsFile=/etc/ssh/known_hosts
  IdentityFile ~/.ssh/private_key
  User foo

Also, are you on bolt version >= 1.15.0? https://puppet.com/docs/bolt/latest/bolt_resolved_issues.html#known-hosts-weren-t-parsed-correctly-1-15-0 could be an issue.

[blackknight36]

@donoghuc Thanks for the help. I'm still not sure why bolt is giving an ssh key error when I connect to a host that is already in my known_hosts file and my ssh-agent is working correctly. I'm not having any issues connecting to hosts through ssh outside of bolt either.