How To Troubleshoot SSH Protocol Issues On Your Virtual Machine

After you determine when to troubleshoot an issue instead of migrating or redeploying, you can identify and resolve specific SSH errors based on which phase of a successful SSH connection you need to debug.

Once a connection is successfully established, the SSH protocol negotiates an encrypted connection with the client based on establishing trust. Before you can even verify who you are on the system, the system will attempt to establish an encryption method with your client. There are a few unique issues that can occur in this scope, and this tutorial covers how to identify them, how to resolve them, and additional resources to prevent them.

Prerequisites

To troubleshoot SSH issues, you will need to make sure your virtual machine is responding normally and has a working network configuration

Errors

Below are some common SSH protocol errors you might encounter.

Host Key Verification Issues

When you connect to a server through an SSH client, the server attempts to identify itself using a host key. In situations where the host key changes, you may see a warning such as:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...

In PuTTY, you get an equally ominous warning:

WARNING - POTENTIAL SECURITY BREACH!

The server's host key does not match the one PuTTY has
cached in the registry. This means that either the
Server administrator has changed the host key, or you
...

To resolve this, you can clear your host keys.

Connection Closed or Reset By Peer

In some situations, connectivity is established at the socket level but disconnects during the host key verification phase. In PuTTY you may see an error reported as:

Server Unexpectedly closed network connection

The OpenSSH client may report:

Connection closed by 203.0.113.0 port 22

Some common issues that may cause this error include:

  • The SSH service has crashed or is experiencing bugs.
  • The SSH service is not able to initialize a connection due missing server host keys.

Unable to Negotiate with Host

When initiating the SSH protocol, a shared secret is generated through a cipher negotiated between the client and the host. When there is a mismatch, you may see errors in PuTTY like this:

Couldn't agree a client-to-server cipher (available: aes128-ctr, aes192-ctr, aes256-ctr)

An OpenSSH client may report:

Unable to negotiate with 203.0.113.0: no matching key exchange method found.
Their offer: diffie-hellman-group1-sha1

Some common issues that may cause this error include:

  • You have a modern OpenSSH client implementation and an older host that does not support a common cipher with the client.
  • You have customized your SSH client cipher list to only allow a selection that the server does not support.

To resolve this issue, you need to customize the supported ciphers in your SSH client.

Solutions

Below are some troubleshooting methods and solutions to common SSH protocol errors.

Clearing Out Host Keys from Known Hosts

Host keys are typically stored in the ~/.ssh/known_hosts file on OpenSSH client implementations. PuTTY typically stores these in the Windows Registry (HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys).

In PuTTY, you can use the dialogue box to select Yes to allow the connection and update the registry. Alternatively, you can manually remove the entry.

On an OpenSSH client, you can can find the host entry in the ~/.ssh/known_hosts file and manually remove it. Another option is to use the ssh-keygen command:

ssh-keygen -R 203.0.113.0

This will attempt to access and clear the matching host entry in the known_hosts file:

# Host 203.0.113.0 found: line 2
/home/user/.ssh/known_hosts updated.
Original contents retained as /home/user/.ssh/known_hosts.old

Checking and Regenerating SSH Host Keys

If your SSH host doesn’t have its own private key to generate a shared secret, the connection will be aborted. To check that it does, look in the /etc/ssh directory for a group of files named like sshd_host_*_key with corresponding .pub files.

If they’re not found, you’ll need to regenerate them. You can do this using ssh-keygen as root or with sudo:

ssh-keygen -A

The output will report the actions taken, including a list of the keys generated:

ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519

Customizing Supported SSH Ciphers

You can customize the supported SSH ciphers on your client machine when you need support for a deprecated cipher like SHA1. This is not a very common issue. It typically happens in instances when you’re using a newer SSH client to connect to an old SSH server that hasn’t yet disabled weaker cyphers.

In OpenSSH, you can configure SHA1 using the KexAlgorithms option on the command line:

openssh -oKexAlgorithms=+diffie-hellman-group1-sha1 root@203.0.113.0

PuTTY should already include the Diffie-Hellman group 1 option in the Connection > SSH > Kex configuration.