Ubuntu 20.04 LTS High CPU From journald Log Storm

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

CPU saturation from log handling slows application response and can cascade into watchdog restarts or node-level alert storms.

Environment & Reproduction

Frequently seen on noisy services emitting repeated error bursts, especially after deployment faults or failing dependencies.

Root Cause Analysis

Unbounded log volume overwhelms journald processing and disk write paths, amplifying overhead on already stressed hosts.

Quick Triage

Locate top logging processes and estimate event rates to separate root service failure from journal subsystem bottlenecks.

Step-by-Step Diagnosis

Use systemd-cgtop and journal statistics to map message sources, then confirm whether persistent storage and rotation policies are overloaded.

Solution – Primary Fix

Apply service-side log throttling, configure journald rate limits and retention settings, and restart affected units in controlled order.

Solution – Alternative Approaches

Offload logs to remote collectors, adopt structured sampling, or isolate noisy workloads into dedicated instances to reduce shared contention.

Verification & Acceptance Criteria

Resolution is confirmed when journald CPU normalizes, queue depth remains stable, and application latency returns to baseline targets.

Rollback Plan

Restore prior journald settings if essential telemetry is lost, then introduce safer staged limits with monitoring before full rollout.

Prevention & Hardening

Set per-service log budgets, alert on sudden message spikes, and test failure logging behavior in pre-production load scenarios.

Related Errors & Cross-Refs

Cross-reference disk I/O saturation and rsyslog bottleneck topics where log amplification drives broader system instability.

Related tutorial: View the step-by-step tutorial for Ubuntu 20.04 LTS.

View all Ubuntu 20.04 LTS tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Consult systemd-journald configuration docs, Linux logging design notes, and SRE guidance on high-cardinality log event control.