RHEL 10 – journald Disk Usage Explodes and Fills Root FS – Log Retention Controls

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Root filesystem usage spikes, risking service outages and failed updates.

Environment & Reproduction

High log volume hosts with default or oversized journal retention.

df -h
journalctl --disk-usage

Root Cause Analysis

systemd-journald retains large binary logs due to permissive size/time settings.

Quick Triage

Measure journal share of disk and identify top noisy units.

journalctl --disk-usage
journalctl -p err -n 200 --no-pager

Step-by-Step Diagnosis

Review journald configuration and recurring high-volume service logs.

grep -E 'SystemMaxUse|SystemKeepFree|MaxRetentionSec' /etc/systemd/journald.conf
journalctl -u  -n 300 --no-pager

Solution – Primary Fix

Vacuum old journals and set bounded retention policy.

journalctl --vacuum-size=1G
vi /etc/systemd/journald.conf
systemctl restart systemd-journald

Solution – Alternative Approaches

Forward logs to rsyslog/SIEM and keep local journal footprint small.

systemctl enable --now rsyslog
journalctl --flush

Verification & Acceptance Criteria

Disk usage drops to target and remains stable over 24h.

df -h
journalctl --disk-usage

Rollback Plan

Restore previous journald.conf if retention becomes too aggressive for audit needs.

cp /etc/systemd/journald.conf.bak /etc/systemd/journald.conf
systemctl restart systemd-journald

Prevention & Hardening

Set monitoring alerts on filesystem and journal growth rate.

systemctl status systemd-journald
journalctl -u systemd-journald -n 80 --no-pager

Related Errors & Cross-Refs

No space left on device, failed to rotate journal files.

Related tutorial: View the step-by-step tutorial for rhel-10.

View all rhel-10 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

systemd-journald retention and log management recommendations.