RHEL 8 journalctl Storage Growth Consumes Root Disk

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Root filesystem gradually fills from persistent journal logs, leading to write failures and service instability.

Environment & Reproduction

Run journalctl –disk-usage on RHEL 8 hosts with high message volume and persistent logging.

Root Cause Analysis

journald size/retention defaults are too high for partition size or noisy services flood logs.

Quick Triage

Vacuum old logs and identify noisy units with journalctl -u and rate-based message analysis.

Step-by-Step Diagnosis

Review /etc/systemd/journald.conf, check forwarding behavior, and quantify per-unit log contribution.

Solution – Primary Fix

Set SystemMaxUse and RuntimeMaxUse, restart systemd-journald, and run journalctl –vacuum-size.

Solution – Alternative Approaches

Forward logs to rsyslog or centralized collector and keep smaller local retention footprint.

Verification & Acceptance Criteria

Disk usage stabilizes, journald stays within limits, and required audit logs remain accessible.

Rollback Plan

Restore previous journald.conf and restart journald if retention changes remove needed local history.

Prevention & Hardening

Alert on journal growth trends and fix noisy services rather than only truncating logs.

Related Errors & Cross-Refs

Related: No space left on device, failed to write entry, and log burst rate limiting warnings.

Related tutorial: View the step-by-step tutorial for rhel-8.

View all rhel-8 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Review systemd-journald and RHEL 8 logging architecture documentation.