RHEL 10 – journald persistent logs consume excessive disk space – Fix & Prevention

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Root filesystem pressure rises, risking application outages and failed updates.

Environment & Reproduction

Seen on verbose services with persistent journal enabled.

journalctl --disk-usage
df -h /var
cat /etc/systemd/journald.conf

Root Cause Analysis

Unbounded journal retention and high-volume logs create uncontrolled growth.

Quick Triage

Determine largest contributors and immediate reclaim potential.

journalctl --disk-usage
journalctl -p err --since yesterday | wc -l
systemctl status systemd-journald

Step-by-Step Diagnosis

Inspect retention settings and log volume sources.

grep -E 'SystemMaxUse|RuntimeMaxUse|MaxFileSec' /etc/systemd/journald.conf
journalctl -u noisy-service --since '24 hours ago' | wc -l
journalctl --verify

Solution – Primary Fix

Set limits, rotate, and vacuum old logs.

sed -i 's/^#SystemMaxUse=.*/SystemMaxUse=1G/' /etc/systemd/journald.conf
systemctl restart systemd-journald
journalctl --rotate
journalctl --vacuum-time=7d

Solution – Alternative Approaches

Forward logs to centralized system and reduce local persistence horizon.

sed -i 's/^#Storage=.*/Storage=auto/' /etc/systemd/journald.conf
systemctl restart systemd-journald

Verification & Acceptance Criteria

Journal usage stays within cap and /var has healthy free space.

journalctl --disk-usage
df -h /var
journalctl -b -p warning -n 30

Rollback Plan

Revert journald.conf changes if retention policy conflicts with audit requirements.

cp -a /etc/systemd/journald.conf.bak /etc/systemd/journald.conf
systemctl restart systemd-journald

Prevention & Hardening

Define retention standards and monitor journal growth trendlines.

systemctl is-enabled systemd-journald
journalctl --disk-usage

Related Errors & Cross-Refs

Can coincide with auditd backlog growth and runaway debug logging.

Related tutorial: View the step-by-step tutorial for rhel-10.

View all rhel-10 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

RHEL journald retention and logging architecture documentation.