The Linux kernel’s systemd journal is a structured binary log that stores the output of every service, kernel message, boot sequence, and user session. Unlike traditional text-based syslog, the journal stores metadata alongside each log entry — the unit name, PID, UID, executable path, systemd unit state, and priority level — enabling far more precise querying. journalctl is the command-line interface to this journal, and mastering it is essential for debugging service crashes, tracing application errors, investigating security incidents, and auditing system behaviour on RHEL 9. This guide covers filtering by unit, time, priority, and keyword; following logs in real time; exporting and rotating journal data; and configuring journal retention.

Prerequisites

  • RHEL 9 server with root or sudo access
  • systemd running (default on RHEL 9)

Step 1 — Understand the Journal Location

By default on RHEL 9 the journal is persistent and stored in /var/log/journal/. If that directory does not exist, logs are written to a volatile store in /run/log/journal/ and are lost on reboot.

# Check where journal is stored
ls /var/log/journal/

# Verify journal disk usage
journalctl --disk-usage

Step 2 — Basic journalctl Usage

# Show all logs from newest to oldest (press q to quit)
journalctl

# Show logs from oldest to newest
journalctl -r

# Show last 50 lines
journalctl -n 50

# Follow new entries in real time (like tail -f)
journalctl -f

# Show current boot's messages only
journalctl -b

# Show previous boot's messages
journalctl -b -1

Step 3 — Filter by Systemd Unit

# All logs for a specific service
journalctl -u nginx.service

# Follow nginx logs in real time
journalctl -u nginx.service -f

# Combined logs for two services
journalctl -u nginx.service -u php-fpm.service

# Logs for a service from the previous boot
journalctl -u nginx.service -b -1

# Show unit logs since last restart of that unit
journalctl -u mariadb.service --since "1 hour ago"

Step 4 — Filter by Time Range

# Logs since a specific date and time
journalctl --since "2025-11-01 14:00:00"

# Logs between two timestamps
journalctl --since "2025-11-01 00:00:00" --until "2025-11-01 23:59:59"

# Human-friendly time references
journalctl --since "1 hour ago"
journalctl --since "yesterday"
journalctl --since "today"
journalctl --since "1 week ago" --until "2 days ago"

Step 5 — Filter by Priority (Log Level)

Systemd uses syslog-compatible priority levels from 0 (emergency) to 7 (debug). Filtering by priority reduces noise dramatically when investigating errors:

# Priority levels:
# 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug

# Show only errors and above (emerg, alert, crit, err)
journalctl -p err

# Show warnings and above
journalctl -p warning

# Show critical errors from the current boot
journalctl -b -p crit

# Show errors from nginx
journalctl -u nginx.service -p err

Step 6 — Search by Keyword and Field

# Search for a keyword in log messages (grep-style, case sensitive)
journalctl -g "connection refused"

# Case-insensitive grep
journalctl -g "failed" --case-sensitive=false

# Filter by PID
journalctl _PID=1234

# Filter by executable path
journalctl _EXE=/usr/sbin/sshd

# Filter by systemd unit combined with keyword
journalctl -u sshd.service -g "Failed password"

# Show authentication failures (useful for security auditing)
journalctl -u sshd.service -p notice --since today | grep "Failed"

Step 7 — Output Formats

# Default human-readable format
journalctl -u nginx.service

# Short format without hostname (cleaner for single-host)
journalctl -u nginx.service -o short

# JSON output for processing with jq
journalctl -u nginx.service -o json | jq '.MESSAGE'

# Verbose JSON with all metadata fields
journalctl -u nginx.service -o json-pretty | head -40

# Export for shipping to a log aggregator
journalctl -u nginx.service -o export > /tmp/nginx-journal-export.bin

# Cat format (message only, no timestamps)
journalctl -u nginx.service -o cat

Step 8 — Manage Journal Storage and Retention

vi /etc/systemd/journald.conf
[Journal]
# Maximum total journal size on disk
SystemMaxUse=2G

# Maximum size per journal file
SystemMaxFileSize=200M

# How long to keep journal entries (days, weeks, months, years)
MaxRetentionSec=30day

# Compress journal files
Compress=yes

# Persist journal across reboots (requires /var/log/journal/ directory)
Storage=persistent
systemctl restart systemd-journald

# Vacuum old journal entries manually
journalctl --vacuum-time=7d      # Remove entries older than 7 days
journalctl --vacuum-size=500M    # Remove oldest entries until under 500 MB

# Verify journal disk usage after vacuum
journalctl --disk-usage

Step 9 — Investigate a Service Crash

# Find when a service last crashed
journalctl -u myapp.service -b 0 -p err --no-pager | tail -30

# Look for OOM (out-of-memory) kills
journalctl -k -g "Out of memory" --since today

# Check kernel panics and hardware errors
journalctl -k -p crit --since "1 week ago"

# Find SELinux denials
journalctl -g "avc: denied" --since today

Verification Checklist

# Journal disk usage
journalctl --disk-usage

# Recent errors across all services
journalctl -p err --since "1 hour ago" --no-pager

# Recent nginx activity
journalctl -u nginx.service -n 20 --no-pager

# SSH authentication failures today
journalctl -u sshd.service --since today -g "Failed"

Conclusion

You now have a comprehensive journalctl toolkit for RHEL 9: filtering by unit, time, priority, PID, executable, and keyword; following logs in real time; outputting to JSON for automated processing; and managing journal storage. The combination of priority filtering (-p err), time ranges (--since today), and unit filtering (-u service) lets you cut through thousands of log entries to find the exact cause of a problem in seconds.

Next steps: How to Install and Configure vim and nano on RHEL 9, How to Perform a System Security Audit with auditd on RHEL 9, and How to Manage System Packages with dnf on RHEL 9.