How to Set Up SSH Key-Based Authentication on RHEL 9

SSH key-based authentication is the gold standard for remote server access. Unlike passwords, SSH keys cannot be guessed by brute-force attacks. An Ed25519 private key is a 256-bit secret that would take longer than the age of the universe to crack by exhaustive search. When combined with a key passphrase, you have two-factor authentication: something you have (the key file) and something you know (the passphrase). This guide covers the full setup on RHEL 9, including generating modern Ed25519 and RSA-4096 keys, hardening the SSH daemon configuration, managing multiple keys with ssh-agent, configuring the local ~/.ssh/config file, and restricting individual keys with authorized_keys options.

Prerequisites

• RHEL 9 server with root or sudo access
• SSH server (openssh-server) installed and running: systemctl status sshd
• A local workstation with an SSH client (Linux or macOS have one built in; Windows users can use Windows Terminal or PuTTY)
• Current password-based SSH access to the server — you need this to install your key before disabling password auth

Step 1 — Generate an SSH Key Pair on Your Local Machine

Run this command on your local workstation, not the server. Choose Ed25519 for all modern systems.

# Ed25519 (recommended)
ssh-keygen -t ed25519 -C "[email protected]" -f ~/.ssh/web01_ed25519

If you need to connect to legacy systems that do not support Ed25519, use RSA-4096:

# RSA-4096 (use only for compatibility with very old SSH servers)
ssh-keygen -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/web01_rsa4096

You will be prompted for a passphrase. Always set one. It encrypts the private key on disk using AES-256, so the file is useless to an attacker without the passphrase.

This creates two files:

• ~/.ssh/web01_ed25519 — your private key. Never share this, never copy it to a server.
• ~/.ssh/web01_ed25519.pub — your public key. This is what you install on the server.

View your public key to confirm it was created correctly:

cat ~/.ssh/web01_ed25519.pub

A valid Ed25519 public key starts with ssh-ed25519 AAAA...

Step 2 — Copy the Public Key to the Server

The most reliable method is ssh-copy-id:

ssh-copy-id -i ~/.ssh/web01_ed25519.pub [email protected]

This appends your public key to ~/.ssh/authorized_keys on the server with correct permissions automatically. You will be prompted for the current password one last time.

If ssh-copy-id is not available (Windows without WSL), copy the key manually on the server:

# On the server, create the .ssh directory for the target user
mkdir -p /home/adminuser/.ssh
chmod 700 /home/adminuser/.ssh
chown adminuser:adminuser /home/adminuser/.ssh

# Paste your public key (replace the key value below)
echo "ssh-ed25519 AAAAC3NzaC1... adminuser@laptop" >> /home/adminuser/.ssh/authorized_keys
chmod 600 /home/adminuser/.ssh/authorized_keys
chown adminuser:adminuser /home/adminuser/.ssh/authorized_keys

Verify the SELinux context is correct (common cause of SSH failures on RHEL):

restorecon -Rv /home/adminuser/.ssh

Step 3 — Harden the SSH Daemon Configuration

Edit /etc/ssh/sshd_config to disable password authentication and enforce key-only login:

vi /etc/ssh/sshd_config

Set or update these directives:

# Disable password-based login entirely
PasswordAuthentication no
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no

# Enable public key authentication
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# Disable root login over SSH
PermitRootLogin no

# Use modern secure ciphers, MACs, and key exchange algorithms
Ciphers [email protected],[email protected],[email protected]
MACs [email protected],[email protected]
KexAlgorithms curve25519-sha256,diffie-hellman-group16-sha512

# Reduce attack surface
X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Add a warning banner
Banner /etc/issue.net

Test the configuration file for syntax errors before restarting:

sshd -t

No output means the configuration is valid. Restart SSH:

systemctl restart sshd

Step 4 — Test Key-Based Login in a New Terminal

Open a new terminal session on your local machine — keep the existing session open as a safety net:

ssh -i ~/.ssh/web01_ed25519 [email protected]

You should be prompted for your key passphrase, then logged in without entering a password. Only close your original session after confirming the new session works.

Enable verbose mode to debug connection issues:

ssh -v -i ~/.ssh/web01_ed25519 [email protected]

Step 5 — Configure ssh-agent to Avoid Repeated Passphrase Entry

ssh-agent holds decrypted keys in memory for the duration of your session, so you enter the passphrase only once per login:

# Start the agent (usually auto-started by desktop environments)
eval "$(ssh-agent -s)"

# Add your key — you will be prompted for the passphrase once
ssh-add ~/.ssh/web01_ed25519

# Confirm the key is loaded
ssh-add -l

On macOS, keys are stored in the system Keychain automatically. On Linux, add the following to ~/.bash_profile for automatic key loading:

if [ -z "$SSH_AUTH_SOCK" ]; then
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/web01_ed25519
fi

Step 6 — Configure ~/.ssh/config for Multiple Servers

A local SSH config file eliminates typing -i, -p, and usernames on every connection:

Host web01
    HostName 203.0.113.10
    User adminuser
    IdentityFile ~/.ssh/web01_ed25519
    Port 22
    ServerAliveInterval 60

Host db01
    HostName 10.0.0.5
    User dbadmin
    IdentityFile ~/.ssh/db01_ed25519
    # Route through web01 as a bastion host
    ProxyJump web01

With this config, connecting is simply ssh web01. The ProxyJump for db01 routes the SSH connection through web01 as a bastion host — the standard pattern for accessing servers on private networks without exposing them to the internet.

Step 7 — Restrict Keys with authorized_keys Options

Individual keys in authorized_keys can carry per-key restrictions:

# Allow a key only from a specific IP address
from="203.0.113.0/24" ssh-ed25519 AAAAC3Nz... adminuser@laptop

# Restrict a key to running one specific command (for automated backups via rsync)
command="/usr/bin/rsync --server --sender -avz . /backup",no-pty,no-agent-forwarding,no-port-forwarding ssh-ed25519 AAAAC3Nz... backup@ci-server

# Combine: allow only from specific IP and only one command
from="10.0.0.100",command="/usr/bin/df -h" ssh-ed25519 AAAAC3Nz... monitoring@nagios

Step 8 — Rotate SSH Keys

Rotate keys annually or when a team member leaves:

# Generate a new key
ssh-keygen -t ed25519 -C "adminuser@web01-2026" -f ~/.ssh/web01_ed25519_2026

# Copy the new key to the server (both keys will work temporarily)
ssh-copy-id -i ~/.ssh/web01_ed25519_2026.pub [email protected]

# Verify the new key works
ssh -i ~/.ssh/web01_ed25519_2026 [email protected]

# Remove the old key from authorized_keys on the server
grep -v "OLD_KEY_COMMENT" ~/.ssh/authorized_keys > /tmp/ak_new
mv /tmp/ak_new ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Verification

# Confirm password auth is disabled in running config
sshd -T | grep -E "passwordauthentication|pubkeyauthentication|permitrootlogin"

# Confirm SELinux context on .ssh directory
ls -laZ /home/adminuser/.ssh

# Check for failed login attempts
grep "Failed" /var/log/secure | tail -20

# Confirm which keys are installed
cat /home/adminuser/.ssh/authorized_keys

Troubleshooting

• Permission denied (publickey) — check /var/log/secure on the server: tail -f /var/log/secure. Common causes: wrong permissions on ~/.ssh (must be 700) or authorized_keys (must be 600), incorrect SELinux context, or the public key was not copied correctly.
• SELinux blocking authorized_keys — run restorecon -Rv ~/.ssh. The correct context for authorized_keys is ssh_home_t.
• Too many authentication failures — ssh-agent is offering too many keys. Use ssh -o IdentitiesOnly=yes -i ~/.ssh/web01_ed25519 ... to force only the specified key.

Security Considerations

• Use unique key pairs per server — a single compromised key should not grant access to your entire fleet.
• Never place SSH private keys on the server itself. Keys on the server can be stolen if the server is compromised.
• For production environments at scale, consider certificate-based SSH (using ssh-keygen -s and a CA) to avoid managing authorized_keys on every server manually.
• Revoke access immediately when staff leave: remove their public key from authorized_keys on every server they had access to.

Conclusion

Your RHEL 9 server now accepts only SSH key authentication, with password login fully disabled, modern ciphers configured, and root login blocked. You have set up ssh-agent for convenience, a ~/.ssh/config for connecting to multiple servers, and you understand how to restrict individual keys with authorized_keys options. Key rotation is now part of your regular security maintenance cycle.

Next steps: How to Configure Fail2Ban to Protect SSH on RHEL 9, How to Set Up Two-Factor Authentication for SSH on RHEL 9, and How to Harden SSH on RHEL 9.