RHEL 8 Log Rotation Failure Fills /var and Breaks Services

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Disk usage on /var reaches critical levels, causing package manager failures, service crashes, and delayed logging. Incident response becomes difficult when logs stop writing.

Environment & Reproduction

Most visible on hosts with verbose application logs and custom logrotate rules. Rotation can silently fail after syntax or permission changes.

Root Cause Analysis

Invalid logrotate configs, missing postrotate scripts, or wrong ownership prevent rotation and compression. journalctl persistence settings can also consume unexpected space.

Quick Triage

Check df -h /var, run logrotate -d against main config, and review journalctl for logrotate timer failures. Identify largest directories with du summaries.

Step-by-Step Diagnosis

Validate each file in /etc/logrotate.d, test manual rotation, and inspect permission mismatches on target logs. Confirm systemd timers are active and executing.

Solution – Primary Fix

Fix malformed rotate stanzas, correct ownership, run logrotate -f as needed, and vacuum oversized journal files. Restart impacted services after freeing capacity.

Solution – Alternative Approaches

Move high-volume logs to dedicated partitions, centralize logs to external collectors, or adopt stricter retention and compression defaults.

Verification & Acceptance Criteria

Disk utilization returns to normal, scheduled rotations succeed, and services remain stable through at least one rotation cycle.

Rollback Plan

Revert recent logrotate changes from version control and restore prior retention policy if unexpected log loss occurs.

Prevention & Hardening

Monitor /var growth, lint logrotate configs in CI, and alert on timer failures. Enforce ownership standards for application log files.

Related Errors & Cross-Refs

Similar symptoms may stem from core dumps, runaway journal size, or temporary file leaks under /var/tmp.

Related tutorial: View the step-by-step tutorial for rhel-8.

View all rhel-8 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Use Red Hat logging and logrotate documentation for reliable retention management on RHEL 8.