RHEL 10 Root Filesystem Fills Rapidly from Journal and Logs

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution — Primary Fix
7. Solution — Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

System becomes unstable with no space left on device, impacting package and service operations.

Environment & Reproduction

Often triggered by verbose services writing continuously to journald and /var/log.

df -h ; sudo du -xh /var | sort -h | tail

Root Cause Analysis

Unbounded journald growth and missing logrotate coverage cause root volume exhaustion.

Quick Triage

Locate largest log consumers and check journald disk usage.

sudo journalctl --disk-usage ; sudo find /var/log -type f -size +200M -ls

Step-by-Step Diagnosis

Identify noisy unit and map growth trend over recent boots.

sudo journalctl -p warning -b ; sudo systemctl list-units --state=failed ; sudo journalctl -u  -n 200

Solution — Primary Fix

Vacuum oversized journal and set retention limits in journald config.

sudo journalctl --vacuum-time=7d ; sudo journalctl --vacuum-size=500M ; sudo sed -i 's/^#SystemMaxUse=.*/SystemMaxUse=500M/' /etc/systemd/journald.conf ; sudo systemctl restart systemd-journald

Solution — Alternative Approaches

Move /var/log to separate volume or ship logs to centralized collector.

Verification & Acceptance Criteria

Free space remains stable and log growth is capped over 24 hours.

df -h / ; sudo journalctl --disk-usage ; sudo logrotate -d /etc/logrotate.conf

Rollback Plan

Revert journald limits if troubleshooting requires temporary larger local retention.

sudo cp /etc/systemd/journald.conf.bak /etc/systemd/journald.conf ; sudo systemctl restart systemd-journald

Prevention & Hardening

Set explicit journal quotas on all hosts and alert at 80 percent disk use.

Related Errors & Cross-Refs

Frequently coupled with failed dnf transactions and systemd unit crashes due to full disk.

Related tutorial: View the step-by-step tutorial for rhel-10.

View all rhel-10 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

systemd-journald and logrotate administration guidance for RHEL.