RHEL 10 – journalctl logs rotate too aggressively and lose incident evidence

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Administrators observe: RHEL 10 – journalctl logs rotate too aggressively and lose incident evidence. Services can become unavailable, maintenance windows extend, and production risk increases if the issue is not remediated quickly.

Environment & Reproduction

Issue appears on RHEL 10 hosts after updates, policy changes, or configuration drift. Reproduce by triggering the affected workflow and watching service and package manager behavior.

# Capture baseline context
cat /etc/redhat-release
uname -r
dnf repolist
systemctl --failed
journalctl -p err -n 50 --no-pager

Root Cause Analysis

The root cause for logging/journald-retention is usually a combination of stale state, dependency mismatch, policy enforcement, or runtime ordering. On RHEL 10, stricter defaults in SELinux, systemd, and firewalld can expose latent misconfiguration.

Quick Triage

Run a short triage pass to verify package integrity, service state, firewall policy, and SELinux mode before deeper changes.

# Quick triage
dnf check || true
systemctl status --no-pager
firewall-cmd --state
getenforce
journalctl -xe --no-pager | tail -n 80

Step-by-Step Diagnosis

Collect detailed diagnostics to isolate the exact failing layer and confirm whether package, policy, service ordering, or network filtering is responsible.

# Detailed diagnosis
journalctl -b --no-pager | tail -n 200
systemctl list-dependencies multi-user.target
firewall-cmd --list-all
semanage boolean -l | head -n 20
dnf history list | head -n 15

Solution – Primary Fix

Apply the primary fix path first. Execute in sequence and validate after each step to avoid masking the true cause.

# Primary fix sequence
dnf clean all && dnf makecache
dnf -y update
systemctl daemon-reload
systemctl restart firewalld
restorecon -Rv /etc /var
setsebool -P nis_enabled 1 || true

Solution – Alternative Approaches

If the primary approach fails, use targeted alternatives to address edge conditions.

# Alternatives
dnf distro-sync -y
ausearch -m avc -ts recent
sealert -a /var/log/audit/audit.log || true
firewall-cmd --runtime-to-permanent
systemctl reset-failed

Verification & Acceptance Criteria

Resolution is accepted only when service behavior, logs, and policy checks remain clean through restart and reload cycles.

# Verify
systemctl --failed
journalctl -p warning -n 30 --no-pager
firewall-cmd --list-all
getenforce
dnf check

Rollback Plan

If regression appears, roll back package and policy changes to last known-good state.

# Rollback
dnf history list | head -n 10
# Replace  with the transaction ID to undo
dnf history undo  -y
setenforce 0
systemctl restart affected-service

Prevention & Hardening

Prevent recurrence with staged updates, policy review, and continuous health checks.

# Hardening
dnf -y install dnf-automatic
systemctl enable --now dnf-automatic.timer
firewall-cmd --permanent --list-all
semanage fcontext -l | head -n 30
journalctl --vacuum-time=14d

Related Errors & Cross-Refs

Related patterns include package repository inconsistency, firewalld runtime/permanent drift, SELinux context mismatch, and systemd dependency loops.

Related tutorial: View the step-by-step tutorial for rhel-10.

View all rhel-10 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Red Hat Enterprise Linux 10 product documentation, dnf command reference, systemd unit and dependency documentation, firewalld administration guide, SELinux troubleshooting with audit logs and journalctl.