PAM (Pluggable Authentication Modules) is the framework that sits between Linux applications and the underlying authentication mechanisms. Almost every program that authenticates users on RHEL 9 — SSH, su, sudo, login, and graphical sessions — goes through PAM. By configuring PAM you can enforce password complexity requirements, lock accounts after failed login attempts, set resource limits for user sessions, and control how authentication decisions are made. Understanding PAM is fundamental to hardening a RHEL 9 system to meet security policies and compliance requirements.

Prerequisites

  • RHEL 9 system with sudo or root access
  • Basic familiarity with Linux authentication concepts
  • A second active root or sudo session open before editing PAM files (to avoid lockout)

Step 1 — Understand the PAM Architecture

PAM configuration files live in /etc/pam.d/. Each file corresponds to a service (e.g., sshd, su, login). The file system-auth and password-auth are included by many service files and are the main files you edit for system-wide policy. Each line in a PAM file has four fields: module type, control flag, module path, and module arguments.

# PAM module types (stacks):
# auth     - verify identity (passwords, tokens)
# account  - check account validity (expiry, access controls)
# password - handle password changes
# session  - set up and tear down user sessions

# Control flags:
# required    - must succeed; failure noted but other modules still run
# requisite   - must succeed; failure causes immediate rejection
# sufficient  - if it succeeds and nothing required has failed, return success
# optional    - result is ignored unless it is the only module in the stack

# List PAM files for common services
ls /etc/pam.d/

# View the system-auth configuration
cat /etc/pam.d/system-auth

# View sshd PAM configuration
cat /etc/pam.d/sshd

Step 2 — Enforce Password Complexity with pam_pwquality

The pam_pwquality.so module enforces password quality rules. On RHEL 9 it is configured via /etc/security/pwquality.conf. Credit values control how character classes affect the minimum length: a positive credit means the class can be used to satisfy part of the minimum; a negative value means the class is required.

# Back up the original configuration
sudo cp /etc/security/pwquality.conf /etc/security/pwquality.conf.bak

# Configure password quality requirements
sudo tee /etc/security/pwquality.conf > /dev/null << 'EOF'
# Minimum password length
minlen = 14

# Require at least 1 digit (negative = required, not just credited)
dcredit = -1

# Require at least 1 uppercase letter
ucredit = -1

# Require at least 1 lowercase letter
lcredit = -1

# Require at least 1 other character (symbol)
ocredit = -1

# Number of characters from the old password that must not appear in new one
difok = 8

# Reject passwords that contain the username
usercheck = 1

# Reject passwords listed in a dictionary
dictcheck = 1

# Number of retries allowed before pam_pwquality returns failure
retry = 3
EOF

# Verify pam_pwquality.so is in the password stack of system-auth
grep pam_pwquality /etc/pam.d/system-auth

Step 3 — Configure Account Lockout with pam_faillock

The pam_faillock.so module counts failed authentication attempts and temporarily locks an account when the threshold is exceeded. This mitigates brute-force password attacks. On RHEL 9, pam_faillock replaces the older pam_tally2.

# pam_faillock is configured in /etc/security/faillock.conf on RHEL 9
sudo cp /etc/security/faillock.conf /etc/security/faillock.conf.bak

sudo tee /etc/security/faillock.conf > /dev/null << 'EOF'
# Lock after 5 failed attempts
deny = 5

# Count failures within this window (seconds)
fail_interval = 900

# Unlock after 15 minutes (900 seconds)
unlock_time = 900

# Apply lockout to root as well (remove this line to exclude root)
even_deny_root

# Log failures
audit
EOF

# Verify pam_faillock.so appears in the auth stack of system-auth
# On RHEL 9, authselect manages the PAM stack — use it instead of editing directly
sudo authselect list
sudo authselect select sssd with-faillock --force

# Confirm faillock lines appear in system-auth
grep faillock /etc/pam.d/system-auth

# Manually unlock a locked account
# sudo faillock --user username --reset

Step 4 — Apply authselect Profiles for System-Auth Management

On RHEL 9, the recommended way to manage system-auth and password-auth is via authselect rather than editing them directly. authselect generates the PAM files from a managed profile and prevents manual edits from being overwritten by system updates.

# List available authselect profiles
authselect list

# List optional features available for the sssd profile
authselect list-features sssd

# Apply the sssd profile with faillock and password quality features
sudo authselect select sssd 
  with-faillock 
  with-pwquality 
  --force

# Verify current profile
authselect current

# View the generated PAM file
cat /etc/pam.d/system-auth

Step 5 — Configure Session Resource Limits with pam_limits

The pam_limits.so module applies resource limits to user sessions at login using the configuration in /etc/security/limits.conf and files in /etc/security/limits.d/. Limits prevent a single user or process from exhausting system resources, which is both a stability and a security control (defense against fork bombs and resource exhaustion attacks).

# /etc/security/limits.conf format:
#         
# domain: username, @groupname, or * (all users)
# type:   soft (warning) or hard (enforced ceiling)

sudo tee /etc/security/limits.d/99-hardening.conf > /dev/null << 'EOF'
# Limit max open file descriptors for all users
*       soft    nofile      65536
*       hard    nofile      65536

# Limit max number of processes per user (prevents fork bombs)
*       soft    nproc       1024
*       hard    nproc       4096

# Disable core dumps for all users (prevents sensitive data leakage)
*       soft    core        0
*       hard    core        0

# Allow higher limits for specific service accounts if needed
# postgres  soft  nofile  131072
# postgres  hard  nofile  131072
EOF

# Verify pam_limits.so is active in the session stack
grep pam_limits /etc/pam.d/system-auth /etc/pam.d/password-auth

# Check limits applied to the current session
ulimit -a

Step 6 — Test and Validate PAM Configuration

Always test PAM changes before relying on them, and always keep a second privileged session open while making changes to avoid being locked out of the system.

# Test password complexity rejection (run as a regular user)
passwd testuser
# Try a short or simple password — it should be rejected with a quality error

# Test faillock — attempt to SSH with wrong password 5+ times
# Then check faillock status:
sudo faillock --user testuser

# Reset a locked account manually
sudo faillock --user testuser --reset

# Test resource limits — check limits for a specific user
sudo -u testuser bash -c "ulimit -a"

# Verify authselect profile integrity
sudo authselect check

# View the active PAM configuration for sshd
cat /etc/pam.d/sshd

Conclusion

You have configured PAM on RHEL 9 to enforce password complexity through pam_pwquality with minimum length and character class requirements, implement account lockout after repeated failures through pam_faillock, manage the PAM stack safely via authselect, and apply session resource limits through pam_limits. These controls are required by CIS RHEL 9 Benchmark sections 5.3 and 5.4, and directly address NIST 800-53 IA-5 and AC-7 controls.

Next steps: How to Configure Password Aging Policies on RHEL 9, How to Harden SSH Configuration on RHEL 9, and How to Configure sudo Policies and Logging on RHEL 9.