Disk Full from Growing journal Logs

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Root filesystem fills unexpectedly, causing service failures and login/session instability.

Environment & Reproduction

Ubuntu 20.04 hosts with persistent journald enabled and high-volume logging workloads.

Root Cause Analysis

Journald retention settings permit unbounded growth relative to available disk capacity.

Quick Triage

Measure journal disk usage and identify top contributing units/services.

Step-by-Step Diagnosis

Review `journald.conf`, log rates, and storage policy for persistent/volatile modes.

Solution – Primary Fix

Vacuum old journals and set explicit size/time retention limits, then restart journald.

Solution – Alternative Approaches

Forward logs centrally and reduce local persistence for short-lived or stateless nodes.

Verification & Acceptance Criteria

Disk usage stabilizes and journal growth remains within configured thresholds.

Rollback Plan

Restore previous log retention settings if forensic requirements demand longer local history.

Prevention & Hardening

Set fleet-wide journald limits and alerts for early disk pressure detection.

Related Errors & Cross-Refs

`No space left on device`, failed service writes, and authentication/session anomalies.

Related tutorial: View the step-by-step tutorial for Ubuntu 20.04 LTS.

View all Ubuntu 20.04 LTS tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

systemd-journald retention management and Ubuntu logging operations guidance.