Ubuntu 22.04 LTS Node Exporter Missing Metrics After Service User Hardening

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Prometheus scrape target stays up but key node metrics disappear after hardening changes. Alerting becomes unreliable and capacity planning data is incomplete.

Environment & Reproduction

Ubuntu 22.04 LTS with node_exporter running under restricted systemd user. Reproduce by tightening filesystem and proc access without updating collector permissions.

Root Cause Analysis

Hardening directives like ProtectSystem and ReadOnlyPaths can block collectors from reading expected kernel and cgroup paths, causing silent metric gaps.

Quick Triage

Check exporter logs, scrape endpoint, and recent unit changes. Determine whether exporter itself is healthy but collectors are failing.

Step-by-Step Diagnosis

Inspect effective systemd sandbox settings and compare enabled collectors with denied paths shown in logs to isolate missing metric families.

Solution – Primary Fix

Relax only required sandbox restrictions, set explicit collector flags, and restart service. Keep least privilege while restoring necessary metric access.

Solution – Alternative Approaches

Disable problematic collectors, use textfile collector for custom metrics, or run exporter in container with controlled host mounts and read-only policies.

Verification & Acceptance Criteria

Previously missing metric families reappear, scrape duration remains stable, and alert rules evaluate correctly without false negatives.

Rollback Plan

Remove override file, daemon-reload, and restart service to revert to prior unit behavior if tightened permissions cannot be safely tuned quickly.

Prevention & Hardening

Test hardening profiles against observability requirements, codify supported collector set, and monitor metric cardinality/availability drift after service changes.

Related Errors & Cross-Refs

Related to cgroup v2 path changes, host PID namespace restrictions, and firewall blocks on scrape port 9100.

Related tutorial: View the step-by-step tutorial for Ubuntu 22.04 LTS.

View all Ubuntu 22.04 LTS tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Use Prometheus node_exporter docs, Ubuntu systemd hardening guidance, and man pages for systemd.exec(5) and systemctl(1).