journalctl Storage Growth Fills Root Filesystem

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Root volume usage grows rapidly, and /var/log/journal consumes large space.

Environment & Reproduction

Persistent journald retention limits are unset or too high for disk size.

Root Cause Analysis

Run journalctl –disk-usage and df -h to quantify pressure on the filesystem.

Quick Triage

Inspect /etc/systemd/journald.conf for SystemMaxUse and related retention controls.

Step-by-Step Diagnosis

Run journalctl –vacuum-time=14d or –vacuum-size=1G based on policy.

Solution – Primary Fix

Set SystemMaxUse and RuntimeMaxUse, then restart systemd-journald.

Solution – Alternative Approaches

Use journalctl -p warning..alert and unit filters to find noisy services.

Verification & Acceptance Criteria

Resolve root causes in failing services to prevent continual log flooding.

Rollback Plan

Forward critical logs to central storage before aggressive local retention trimming.

Prevention & Hardening

Disk exhaustion can stop applications, updates, and system services.

Related Errors & Cross-Refs

Alert on journal usage growth rate and low free space thresholds.

Related tutorial: View the step-by-step tutorial for rhel-9.

View all rhel-9 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Define journald retention baselines for each server role in RHEL 9 estates.