CentOS Stream 10 – Common Problem 058 – Diagnosis and Fix

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

/var fills with /var/log/journal data, triggering `No space left on device` errors.

Environment & Reproduction

Caused by verbose service logging combined with default unlimited journal retention.

Root Cause Analysis

SystemMaxUse is unset so journald keeps growing until disk space is exhausted.

Quick Triage

Check disk usage with `journalctl –disk-usage` and `df -h /var/log/journal`.

Step-by-Step Diagnosis

Identify chatty units with `journalctl –since today | awk ‘{print $5}’ | sort | uniq -c`.

Solution – Primary Fix

Set retention with `journalctl –vacuum-size=1G` and configure SystemMaxUse in journald.conf.

Solution – Alternative Approaches

Forward journal to a remote log aggregator and clear local storage.

Verification & Acceptance Criteria

`/var/log/journal` stays within configured size after one rotation cycle.

Rollback Plan

Restore prior journald.conf if downstream queries depend on raw archived data.

Prevention & Hardening

Add disk pressure monitoring to alert before /var fills.

Related Errors & Cross-Refs

Often coincides with high-cardinality container logs and crash loops.

Related tutorial: View the step-by-step tutorial for centos-stream-10.

View all centos-stream-10 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

systemd-journald.conf manual for CentOS Stream 10.