RHEL 7 auditd backlog overflow causing dropped security events

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Security audit trail contains gaps because events are dropped under load. Detection and forensic confidence decline, and compliance controls may be considered non-conformant.

Environment & Reproduction

Happens during high syscall activity, excessive audit rules, or slow log forwarding/storage. Reproduce with bursty workload while low backlog limits are configured in kernel/audit settings.

Root Cause Analysis

auditd cannot drain kernel audit queue quickly enough, leading to backlog overflow and dropped records. Disk throughput, rsyslog forwarding lag, or overbroad rule sets amplify the issue.

Quick Triage

Check auditctl status, auditd service state, and journalctl for backlog warnings. Verify disk I/O headroom and downstream log pipeline health before changing rule volume.

Step-by-Step Diagnosis

Measure event rate, inspect active audit rules for noise, and identify bottlenecks in local write or remote forwarding. Correlate overflow timestamps with workload spikes in journalctl.

Solution – Primary Fix

Increase backlog limits and rate settings appropriately, optimize audit rules, ensure auditd/rsyslog services are healthy via systemctl, and validate sustained event throughput without drops.

Solution – Alternative Approaches

Offload noisy rules, segment high-volume hosts, or deploy faster storage for audit paths. Use targeted rule scopes to preserve high-value events while reducing queue pressure.

Verification & Acceptance Criteria

No dropped event messages should appear during representative peak load. auditctl counters and journalctl logs must indicate stable queue behavior and complete event delivery.

Rollback Plan

If tuning destabilizes host performance, revert to prior limits and rule set from backup, then apply incremental adjustments with controlled load testing.

Prevention & Hardening

Continuously monitor audit queue health, review rule efficiency quarterly, and align log pipeline capacity with expected syscall volume. Keep SELinux, firewalld, and audit policy synchronized.

Related Errors & Cross-Refs

Related messages include audit: backlog limit exceeded and lost records. Cross-reference rsyslog forwarding delays, storage latency, and incident periods with unusual process activity.

Related tutorial: View the step-by-step tutorial for rhel-7.

View all rhel-7 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Refer to auditd, auditctl, and Red Hat security logging guidance. Include tested queue tuning baselines and escalation thresholds in security operations playbooks.