SLES 16 – Common Problem 008 – Diagnosis and Fix

📖 ~5 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Administrators on SUSE Linux Enterprise Server 16 observe journalctl logs lost after reboot (volatile journal) appearing in production, often after a zypper update, SUSEConnect refresh, or systemd reload. Visible symptoms include failed services in systemctl status output, repeated entries in journalctl, and degraded user-facing behaviour such as failed logins, blocked traffic, or unreachable applications. Business impact ranges from minor latency to full outage for hosted workloads, especially when the affected node is part of a Pacemaker cluster or load-balanced tier. Capture the time, host, and exact error string before changing anything so root cause work has a stable starting point.

Environment & Reproduction

Reproduction targets SLES 16 GA and current maintenance level on x86_64 and aarch64, both physical and KVM/Hyper-V/Azure virtual machines. Confirm the channel state with `SUSEConnect –status-text` and `zypper lr -d`, and record kernel with `uname -r`. Reproduce by triggering the workflow that exposed journalctl logs lost after reboot (volatile journal): a fresh `zypper ref && zypper up`, a `systemctl restart` of the affected unit, or a network event such as `nmcli connection reload` or `wicked ifup`. Test on a non-production host first; on cloud images, snapshot the disk before iterating so each attempt starts from a known state.

Root Cause Analysis

Root cause for journalctl logs lost after reboot (volatile journal) typically traces back to one of: an outdated or pending repository metadata cache, a SUSEConnect token or proxy variable that no longer resolves, an AppArmor profile mismatch after a package upgrade, a firewalld zone that lost its assignment during reload, or a systemd unit whose ConditionPathExists no longer holds. Correlate the failure timestamp from `journalctl -b –since` with the audit log under /var/log/audit and with zypper history in /var/log/zypp/history. The pattern that proves the cause is a state change (package, profile, network) immediately preceding the first failing log entry.

Quick Triage

Within the first five minutes run a fixed triage block so handover stays consistent: `systemctl –failed`, `systemctl status -l`, `journalctl -xeu –no-pager | tail -n 200`, `firewall-cmd –state && firewall-cmd –list-all`, `aa-status | head`, `SUSEConnect –status-text`, and `zypper ps -s` to list services that need restart after a patch. If the host is unreachable over SSH, open a serial or cloud console and run the same commands locally. Decide immediately whether to roll back the last change (snapshot, package, profile) or to keep digging — do not do both at once.

Step-by-Step Diagnosis

Walk the stack from package to runtime. Step 1: `zypper history | tail -n 50` and `rpm -qa –last | head` to find the last change window. Step 2: validate repos with `zypper lr -E` and `zypper ref`; clear stale cache with `zypper clean -a` if metadata signatures look wrong. Step 3: inspect the unit with `systemctl cat ` and `systemd-analyze verify `. Step 4: check AppArmor with `aa-status` and `journalctl -k | grep -i apparmor`. Step 5: check firewalld with `firewall-cmd –get-active-zones` and confirm the interface is in the expected zone. Step 6: tail `journalctl -f` while reproducing the failure once more to capture a clean trace.

Solution – Primary Fix

Apply the canonical SUSE fix for journalctl logs lost after reboot (volatile journal). Refresh credentials and metadata: `SUSEConnect –cleanup && SUSEConnect -r -e ` if registration is involved, then `zypper ref -s`. Install or reinstall the affected package with `zypper in -f ` and run `zypper ps -sss | xargs -r systemctl restart` to bring services up on new binaries. For AppArmor regressions, regenerate the profile with `aa-genprof` or load the vendor profile via `apparmor_parser -r /etc/apparmor.d/`. For firewalld, place the interface in the correct zone with `firewall-cmd –permanent –zone=public –change-interface=eth0 && firewall-cmd –reload`. Verify with `systemctl status ` and one positive functional probe.

Solution – Alternative Approaches

If the primary fix is blocked by change control, fall back to a reversible workaround. Pin the previous package version with `zypper al ` and `zypper in –oldpackage -`, or boot the previous Btrfs snapshot from the GRUB snapper menu to restore last-known-good state. For network problems, switch the host temporarily from wicked to NetworkManager (or vice versa) with `systemctl disable –now wicked.service && systemctl enable –now NetworkManager.service` after backing up /etc/sysconfig/network. For service-specific issues, run the daemon in the foreground with `-d` or `-X` flags under a screen/tmux session to capture verbose output before the next maintenance window.

Verification & Acceptance Criteria

Acceptance is met when: the failing unit reports `active (running)` for at least 10 minutes with no restarts in `systemctl status`; `journalctl -p err -b` shows no related errors after the fix timestamp; a functional probe (HTTP 200, successful AD login, NFS mount with `df -h`, etc.) passes from a remote host; `zypper ps -s` is empty; and `firewall-cmd –list-all` shows the expected services and ports. Record the verification output, attach it to the change ticket, and re-run the original reproduction step once to confirm the symptom does not return.

Rollback Plan

Plan rollback before applying the fix. Snapshot the root subvolume with `snapper create -d ‘pre-fix-CP-008’` so a single `snapper rollback ` followed by reboot restores the state. Record the previous package set with `rpm -qa | sort > /root/pre-fix-rpms.txt` and the previous firewalld and AppArmor configuration by copying /etc/firewalld and /etc/apparmor.d into a timestamped tarball. If a service config changed, keep the prior file as `.bak-`. The decision to roll back triggers when verification fails twice or when a dependent service starts failing within 15 minutes of the fix.

Prevention & Hardening

Prevent recurrence by codifying the fix. Add a Salt or Ansible state that enforces the correct repository list, SUSEConnect status, firewalld zone assignment, and AppArmor profile set, and run it on a schedule. Enable `zypper-needs-restarting` and email alerts so pending service restarts after patching are not missed. Tighten change control: require a snapper snapshot before `zypper up` on any production host, and require a smoke-test script to run post-patch. Add a Prometheus or Zabbix check that asserts the unit is active and the relevant TCP/UDP port is reachable from a known monitoring host.

Related Errors & Cross-Refs

Closely related issues that share root cause patterns with journalctl logs lost after reboot (volatile journal): registration token expiry after `SUSEConnect –cleanup`, missing PackageHub or SLE Module enablement, transactional-update conflicts on SLE Micro hosts, kernel module load failures after Secure Boot enrolment, and Btrfs quota exhaustion masquerading as `ENOSPC`. When triaging, check the SUSE Customer Center for the relevant TID (Technical Information Document) and cross-reference the bsc# bug numbers found in `zypper info` advisories. Link the change ticket to the related runbook entries so future on-call engineers find them quickly.

Related tutorial: View the step-by-step tutorial for sles-16.

View all sles-16 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Authoritative sources: SUSE Linux Enterprise Server 16 Administration Guide, SUSE Customer Center Knowledge Base (search for the exact error string), `man zypper(8)`, `man firewall-cmd(1)`, `man systemd.unit(5)`, `man apparmor(7)`, and the SUSE Documentation portal sections on snapper, transactional-update, and SUSEConnect. Internal: the team runbook for SLES patching, the network zone matrix, and the post-mortem for the last incident with the same symptom signature. Keep the references alongside the fix in the knowledge base so the next engineer reaches resolution faster.