Rootkits are among the most dangerous forms of malware — they conceal themselves and other malicious software deep within the operating system, often modifying system binaries and kernel modules to hide their presence from standard tools. Detecting rootkits requires specialised scanners that check for known signatures, hidden files, suspicious processes, and tampered system binaries. This tutorial covers installing and using two complementary rootkit detection tools — rkhunter and chkrootkit — on RHEL 9, and setting up automated nightly scans with email alerts.

Prerequisites

  • RHEL 9 system with root or sudo access
  • EPEL repository enabled
  • A mail transfer agent configured (e.g., Postfix) if you want email alerts from automated scans
  • Internet access to download tool databases and packages

Step 1 — Install and Configure rkhunter

rkhunter (Rootkit Hunter) checks for known rootkits, backdoors, sniffers, and local exploits. It is available from the EPEL repository:

dnf install -y epel-release
dnf install -y rkhunter

# Verify the installation
rkhunter --version

After installing, update the rkhunter data file (signatures database) and build a baseline of file properties for your system. The baseline records SHA-256 hashes, permissions, and ownership of system binaries — deviations from this baseline in future scans will trigger warnings:

# Update the signatures database
rkhunter --update

# Build the initial file properties database (baseline)
# Run this immediately after a clean OS install, before connecting to the internet
rkhunter --propupd

Step 2 — Run a Full rkhunter Scan

Run a full system check. The --sk flag skips the keypress prompts between sections, making it suitable for automated use:

# Interactive scan (press Enter to continue between sections)
rkhunter --check

# Non-interactive scan (suitable for scripts and cron)
rkhunter --check --sk

# View the detailed log file after the scan
cat /var/log/rkhunter.log

# View only warnings from the log
grep -i "warning|found" /var/log/rkhunter.log | grep -v "^#"

The summary at the end shows counts for System checks, File properties checks, Rootkit checks, Application checks, and a final list of warnings. A clean system should show zero warnings in the Rootkit checks section.

Step 3 — Understand and Handle False Positives

rkhunter warnings are not always genuine threats. Common false positives occur after system updates (binaries change), when using non-standard SSH configurations, or because of tools rkhunter considers suspicious but which are legitimately installed on your system. Before investigating a warning as a real threat, check whether it is a known false positive:

# Common false positive: SSH using a non-default protocol version
# Warning: "The SSH and rkhunter SSH version strings differ"
# Fix: update rkhunter's configuration to match your sshd
grep "^Protocol" /etc/ssh/sshd_config

# After a system update that changes binary hashes, rebuild the baseline
dnf update -y
rkhunter --propupd

# Whitelist a specific file or directory in /etc/rkhunter.conf
# For example, to whitelist a custom script in /usr/local/bin:
echo 'SCRIPTWHITELIST=/usr/local/bin/my-custom-script' >> /etc/rkhunter.conf

Key rkhunter configuration options in /etc/rkhunter.conf:

# Allow sending reports by email
MAIL-ON-WARNING=root@localhost

# Specify a custom log file location
LOGFILE=/var/log/rkhunter.log

# Update mirrors
UPDATE_MIRRORS=1
MIRRORS_MODE=0

Step 4 — Install and Run chkrootkit

chkrootkit is a different tool that uses a script-based approach to detect rootkits by looking for known attack signatures in binary files, checking for hidden processes, and testing for suspicious network activity. It complements rkhunter well because they use different detection methods:

# chkrootkit is available from EPEL
dnf install -y chkrootkit

# Run a full scan
chkrootkit

# Run a specific test only (e.g., check for hidden processes)
chkrootkit -t ps

# Quiet mode: show only infected items
chkrootkit -q

# List all available tests
chkrootkit -l

A clean system produces output like INFECTED only for known false positives. The most common false positive on Linux systems is chkrootkit flagging the bindshell test. Verify any INFECTED result manually before treating it as a confirmed compromise.

Step 5 — Compare rkhunter and chkrootkit

Both tools have different strengths, and running both provides better coverage than either alone:

  • rkhunter — Signature database-driven, checks file property hashes against a baseline, tests for suspicious ports and processes, more actively maintained with regular database updates.
  • chkrootkit — Script-based pattern matching, checks specific rootkit signatures embedded in the tool itself, faster to run, useful as a quick second opinion.

Neither tool can guarantee detection of a sophisticated, previously unknown rootkit. They are most effective when run on a known-clean baseline system and used to detect changes over time. If you suspect a compromise, boot from trusted external media for analysis.

Step 6 — Set Up Automated Nightly Scans with Email Alerts

Automate both tools to run nightly and email results if warnings are found. First, configure rkhunter email alerts in its config file, then create a systemd timer (preferred over cron on RHEL 9) or a cron job:

# Configure rkhunter to email on warnings (edit /etc/rkhunter.conf)
sed -i 's/^#?MAIL-ON-WARNING=.*/MAIL-ON-WARNING=root@localhost/' /etc/rkhunter.conf

# Create a combined scan script
cat > /usr/local/bin/nightly-rootkit-scan.sh < "$LOG"
rkhunter --update --sk >> "$LOG" 2>&1
rkhunter --check --sk >> "$LOG" 2>&1

echo "" >> "$LOG"
echo "=== chkrootkit scan: ${DATE} ===" >> "$LOG"
chkrootkit -q >> "$LOG" 2>&1

# Email the log if any warnings were found
if grep -qi "warning|INFECTED" "$LOG"; then
    mail -s "ALERT: Rootkit scan warnings on $(hostname) - ${DATE}" root  /etc/cron.d/rootkit-scan
chmod 644 /etc/cron.d/rootkit-scan

Conclusion

You now have both rkhunter and chkrootkit installed and running automated nightly scans on your RHEL 9 system. rkhunter’s signature database and file property baseline provide continuous monitoring for known rootkits and unexpected system binary changes, while chkrootkit adds a fast script-based second layer of detection. Remember that rootkit detection tools are a supplement to, not a replacement for, solid preventative security — keep your system patched, use SELinux, restrict privileged access, and monitor logs consistently.

Next steps: How to Audit Linux Security with Lynis on RHEL 9, How to Configure nftables Firewall on RHEL 9, and How to Set Up Two-Factor Authentication for SSH on RHEL 9.