How to Set Up SSH Key-Based Authentication on RHEL 7

SSH key-based authentication replaces password logins with a cryptographic key pair: a private key that stays on your local machine and a public key that you install on the remote server. When you connect, the server challenges your SSH client to prove it holds the private key, and the login succeeds without any password ever being transmitted across the network. This approach is significantly more secure than even a strong password — it eliminates brute-force risk entirely and protects against credential phishing. This tutorial walks you through generating a key pair, installing the public key on your RHEL 7 server, and locking down the SSH daemon to require key-based logins exclusively.

Prerequisites

  • A RHEL 7 server you can currently log into (with password or console access)
  • A local workstation (Linux, macOS, or Windows with OpenSSH) from which you will connect
  • Root or sudo access on the server
  • A non-root user account on the server to receive the key (see the user management tutorial if needed)

Step 1: Generate an SSH Key Pair on Your Local Machine

All key generation happens on your local workstation, not on the server. Open a terminal on your local machine and run:

ssh-keygen -t ed25519 -C "[email protected]"

The -t ed25519 flag selects the Ed25519 algorithm, which produces compact, fast, and highly secure keys. It is the recommended choice for new keys in 2024. If your environment requires RSA for compatibility with older systems, use a 4096-bit key instead:

ssh-keygen -t rsa -b 4096 -C "[email protected]"

You will be prompted for a file path to save the key pair:

Enter file in which to save the key (/home/localuser/.ssh/id_ed25519):

Press Enter to accept the default, or provide a custom path if you manage multiple keys.

Next, you will be prompted for a passphrase:

Enter passphrase (empty for no passphrase):

Always set a passphrase. The passphrase encrypts the private key file on disk, so even if the file is stolen it cannot be used without the passphrase. Use ssh-agent to cache the passphrase so you only enter it once per session.

After generation, two files are created:

  • ~/.ssh/id_ed25519 — your private key. Never share this file.
  • ~/.ssh/id_ed25519.pub — your public key. This is what you copy to servers.

View your public key to confirm it was created:

cat ~/.ssh/id_ed25519.pub

Step 2: Copy the Public Key to the RHEL 7 Server

The easiest way to install your public key on the remote server is with ssh-copy-id, which handles all the directory creation, file appending, and permission setting automatically:

ssh-copy-id -i ~/.ssh/id_ed25519.pub adminuser@your_server_ip

You will be prompted for the user’s current password one final time. After successful authentication, ssh-copy-id appends your public key to ~/.ssh/authorized_keys on the server and sets the correct permissions.

If ssh-copy-id is not available on your local machine, you can perform the same operation manually:

# On your local machine, display the public key:
cat ~/.ssh/id_ed25519.pub

# On the RHEL 7 server (logged in via password), create the .ssh directory and authorized_keys file:
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "paste_your_public_key_here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Alternatively, use SSH to do it in one command from your local machine:

cat ~/.ssh/id_ed25519.pub | ssh adminuser@your_server_ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Step 3: Verify Key-Based Login Before Changing SSH Settings

Before disabling password authentication, confirm that key-based login is working. From your local machine:

ssh adminuser@your_server_ip

If everything is configured correctly, you should be logged in without being prompted for a password (or you will be prompted for your key’s passphrase if you set one). If the connection still asks for a password, do not proceed to Step 4 — troubleshoot first.

Common troubleshooting steps:

# On the server, check file permissions (must be exact):
ls -la ~/.ssh/
# ~/.ssh must be 700 (drwx------)
# ~/.ssh/authorized_keys must be 600 (-rw-------)

# Check SELinux context on the .ssh directory:
ls -laZ ~/.ssh/

# If the SELinux context is wrong, restore it:
restorecon -R -v ~/.ssh

# Check sshd logs for authentication failure details:
journalctl -u sshd -n 50

Step 4: Harden sshd_config on the Server

Once you have verified key-based login works, log in to the server and edit the SSH daemon configuration to enforce key authentication and disable less secure methods. Make a backup first:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

Open /etc/ssh/sshd_config with a text editor and set or verify the following directives. Search for each line and change it (or add it if missing):

# Require public key authentication:
PubkeyAuthentication yes

# Disable password-based authentication:
PasswordAuthentication no

# Disable challenge-response auth (Kerberos passwords, etc.):
ChallengeResponseAuthentication no

# Disable root login entirely:
PermitRootLogin no

# Allow only specific users (optional but recommended):
AllowUsers adminuser deployuser

# Disable empty passwords:
PermitEmptyPasswords no

# Use only SSH protocol 2:
Protocol 2

Critical: Leave your current SSH session open while making these changes. Test connectivity in a separate terminal window before closing the original session.

Test the configuration file for syntax errors:

sudo sshd -t

If no errors are reported, restart the SSH daemon:

sudo systemctl restart sshd

Verify it came back up cleanly:

sudo systemctl status sshd

Step 5: Using ssh-agent to Cache Your Passphrase

If you set a passphrase on your private key (which you should), you can use ssh-agent to cache it in memory so you only enter it once per session:

# Start ssh-agent and set the required environment variables:
eval "$(ssh-agent -s)"

# Add your private key to the agent:
ssh-add ~/.ssh/id_ed25519

You will be prompted for the passphrase once. After that, any SSH connections or scp transfers during that session will use the cached key without re-prompting.

List the keys currently loaded in the agent:

ssh-add -l

Step 6: Managing Multiple Keys

If you connect to many servers, you may maintain several key pairs. Use ~/.ssh/config on your local machine to map hostnames to specific keys:

Host web01
    HostName 203.0.113.10
    User adminuser
    IdentityFile ~/.ssh/id_ed25519_web01
    Port 22

Host db-server
    HostName 203.0.113.20
    User dbadmin
    IdentityFile ~/.ssh/id_rsa_dbserver
    Port 22

Set strict permissions on this file:

chmod 600 ~/.ssh/config

With this configuration in place, you can connect with simply ssh web01 and SSH will use the correct key and settings automatically.

SSH key-based authentication is the cornerstone of secure remote server administration. By eliminating passwords from the remote login process, you remove the single largest attack vector against exposed SSH services. Combined with PermitRootLogin no and AllowUsers restrictions, your RHEL 7 server’s SSH daemon becomes highly resistant to both automated attacks and targeted credential theft. Keep your private keys secure, always use passphrases, and rotate keys periodically as part of your routine credential hygiene.