Three Common Problems With SSH-Keychains

Part of keeping a secure network is periodically backing up data or system logs, but if you have more than a handful of computers then logging on to each individual workstation can be a hassle.  So what's a lazy (I prefer "efficient") Systems Administrator to do?

Why, automate it through scripts, of course.

Typically, in order to automate retrieving data from multiple workstations, you need to build a script that remotely logs on to each workstation using secure shell, or SSH. Unfortunately, the process of using SSH to access a workstation requires you to authenticate with each workstation, preventing you from automating it without either:

  • Embedding the username and password into your script
  • Disabling authentication completely
  • Manually typing in the username and password each time you run the script

Fortunately, there is an alternative:  SSH-Keychains.  SSH-Keychains utilize Public Key Infrastructure (PKI) to generate a public and private key to authenticate the server and account with each workstation without the use of a normal password.

A brief introduction of PKI can be found in a blog post I did back in August, or you can catch a more in-depth description in Understanding IT: Decoding Business and Technology.

The primary command used to pull this off is ssh-keygen and The Geek Stuff does a fairly good job of explaining the basic process, so rather than rehash it here, I'll just send you over there.

   Click the image for the blog post to learn how to generate RSA and DSA keys.


Click the image for the blog post to learn how to generate RSA and DSA keys.

What this, and most other posts on the subject, fail to mention are the common reasons why creating and copying your new ssh keys may not work right away. Which is where this post comes in handy; explaining three common problems with SSH-Keychains:



One common problem is that the destination-host (the workstation you're pulling data from) does not recognize the local-host (the server you want to pull data to) as a trusted server. In order for the destination-host to trust the local-host, it must check the known_hosts file in the /etc/ssh directory or the home directory of the user attempting to connect to it (e.g. ~/.ssh/known_hosts).  

This is an important step in the authentication process as the known_hosts file is a file managed on the destination-host to establish a list of trusted servers to prevent man-in-the-middle attacks so that if a compromised server attempts to SSH into a secure workstation, it would be denied access.

Hint: Use the -v flag when you ssh to the destination-host to see if the host is found in the known_hosts file. If it isn't, then this could be (one of) the problem(s).

A failure to use a public key because of the known_hosts file could be fairly benign (an IP address or Operating System change), and can be fixed by removing the old key from the known_hosts file by deleting the line starting with that server's name, and then attempting to SSH back into the server.


2.  Invalid permissions or ownership

Spend a little bit of time searching for answers on SSH-Keychains and you'll quickly find that the overwhelmingly most common error is invalid permissions or ownership of certain files. Some distributions of Linux/Unix may be more forgiving than others, but typically the local-host and destination-host must be set up with a pretty restrictive set of permissions (just blasting chmod 777 on everything won't work)!

Typically, you'll want the .ssh directory on your local-host to be 700 (drwx------), the public key to be 644 (-wr-r--r--), and the private key to be 600 (-rw------). This is a fairly common response on websites like stackoverflow, along with the recommendation that you double check to ensure that the /home/$USER directory (commonly referred to as "~" for short) and the ~/.ssh directories are owned by the $USER and no one else.

It's also worth mentioning that you should double check the permissions on ~/.ssh/. directory is owned by the $USER, and ~/.ssh/.. directory is owned by root as well.  This shouldn't be a problem (it should always be set to these), but if it somehow got changed, it can be something you might not think to check.

What happens if your permissions are too open? It errors out to prevent illegitimate or tampered keys from being inadvertently accepted by the system.

   Screen capture from (another excellent SSH tutorial  here ).


Screen capture from (another excellent SSH tutorial here).

Why?  If your private key is "world readable" then unauthorized users can impersonate you, giving them access they would not otherwise have.  As a result, if your private key is readable by more than just your account, then your SSH public key attempt will fail and you will be prompted for a password.


3. settings in /etc/shadow or /etc/ssh/sshd_config

One other common problem with using SSH-keychains is an improper setting in either /etc/shadow (or /etc/passwd) /etc/ssh/sshd_config files. Several problems could occur here, such as setting PermitEmptyPasswords in sshd_config to no, or having a hashed password, or *LK* on the second field of /etc/shadow.  Like the known_hosts file, this is a pretty quick fix once you know which files to manipulate, but if you've tried the other steps, it wouldn't hurt to double check your settings.



So, to recap:  Creating an SSH-Keychain should be a quick process using the ssh-keygen command and copying the public key ( to the ~/.ssh/authorized_keys file, but sometimes the process still hangs up.  So you'll need to troubleshoot the issue using ssh -v to determine why the SSH is failing.  It could fail because:  Destination is not found in the known_hosts, public key failed, or failed keyboard-interactive (configuration file settings).