Disk Fills Quickly Because Journald Logs Grow Unbounded

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

`/var` usage climbs rapidly and log retention pressure causes service instability.

Environment & Reproduction

Run `journalctl –disk-usage` and `du -sh /var/log/journal` to measure growth.

Root Cause Analysis

Confirm `systemd` and log forwarding components are at supported RHEL 9 patch level.

Quick Triage

Check `systemctl status systemd-journald` for errors and restart frequency.

Step-by-Step Diagnosis

Use `journalctl -p warning..alert –since -24h` to identify noisy units flooding logs.

Solution – Primary Fix

Backup `/etc/systemd/journald.conf` and current storage metrics before tuning.

Solution – Alternative Approaches

If forwarding logs remotely, verify path availability and `firewalld` policy for log transport.

Verification & Acceptance Criteria

Validate log directory labels after manual cleanup to prevent write failures.

Rollback Plan

Set `SystemMaxUse` and retention controls, then rotate with `journalctl –vacuum-time=7d`.

Prevention & Hardening

Confirm disk usage stabilizes and journald continues collecting entries normally.

Related Errors & Cross-Refs

Implement noisy-service detection and enforce log budgets per role.

Related tutorial: View the step-by-step tutorial for rhel-9.

View all rhel-9 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Provide journald config, top chatty units, and growth trend evidence.