Ubuntu 16.04 — slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide

📖 ~4 min read  •  Source: Ubuntu Security Notice USN-8197-1

Related CVEs: CVE-2022-29501 CVE-2016-10030 CVE-2018-10995 CVE-2017-15566 CVE-2018-7033 CVE-2019-6438 CVE-2020-12693 CVE-2020-27745  +2 more

Upstream summary: It was discovered that Slurm did not properly handle access control when
dealing with RPC traffic through PMI2 and PMIx, which could allow an
unprivileged user to send data to an arbitrary unix socket on the host.
An attacker could possibly use this issue to execute arbitrary code as
the root user.

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

On Ubuntu 16.04 (xenial) hosts that have slurm-llnl installed, administrators report behaviour consistent with Ubuntu Security Notice USN-8197-1: apt reports pending security updates, services backed by slurm-llnl fail or restart unexpectedly, AppArmor denials appear in the kernel log, and — for security-rated advisories — the host is exposed to the vulnerability set above. Impact ranges from a single service-restart loop to wider availability incidents whenever slurm-llnl sits on the serving path.

Environment & Reproduction

Reproduction targets Ubuntu 16.04 (xenial). Confirm release and installed package:

lsb_release -a
cat /etc/os-release
dpkg -l slurm-llnl | tail -2
apt-cache policy slurm-llnl
uname -r

Trigger the workflow that exposes slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide while collecting:

sudo journalctl -u slurm-llnl -b --no-pager | tail -200
sudo journalctl -xe --no-pager | tail -200
sudo tail -200 /var/log/apt/history.log
sudo tail -200 /var/log/kern.log | grep -i apparmor

Root Cause Analysis

Root cause is documented in Ubuntu Security Notice USN-8197-1. Canonical security maintainers shipped fixes in the corresponding slurm-llnl update for Ubuntu 16.04; running an outdated build leaves the host exposed to the failure modes described in the advisory. On this release the fix typically arrives via the Ubuntu Pro ESM (esm-infra / esm-apps) channels rather than the standard archive. Correlate apt history with the journal:

grep -A2 -B2 slurm-llnl /var/log/apt/history.log
zgrep -A2 -B2 slurm-llnl /var/log/apt/history.log.*.gz 2>/dev/null
cat /proc/sys/kernel/tainted   # non-zero = tainted kernel / out-of-tree modules

Quick Triage

Run these on Ubuntu 16.04 to capture the current state of slurm-llnl:

dpkg -l slurm-llnl | tail -1                  # installed version
dpkg -V slurm-llnl                             # verify shipped files
sudo apt update && apt list --upgradable 2>/dev/null | grep -i security
systemctl is-active slurm-llnl
sudo ufw status verbose 2>/dev/null | head -20
sudo aa-status 2>/dev/null | head -20
# If slurm-llnl ships a service unit (unit/job name often differs from pkg name, e.g.
# bind9→named, apache2→apache2, postgresql-NN→postgresql@NN-main):
systemctl list-unit-files | grep -i slurm | head

On xenial the standard archive no longer ships security fixes. Verify Ubuntu Pro ESM coverage:

# `pro` CLI not available on this release; check the older `ubuntu-advantage-tools`:
sudo ua status --format=json 2>/dev/null | head
apt-cache policy | grep -i esm

Step-by-Step Diagnosis

1. List failing services.

   systemctl --failed --no-pager

2. Tail the journal / syslog for slurm-llnl.

   sudo journalctl -u slurm-llnl -f --no-pager
   sudo journalctl -xe -f --no-pager

3. Inspect UFW (Uncomplicated Firewall) state.

   sudo ufw status numbered
   sudo ufw show added
   sudo iptables -L -n -v | head -30

4. Surface AppArmor denials and switch the profile to complain mode if needed.

   sudo journalctl -k 2>/dev/null | grep -i 'apparmor="DENIED"' | tail -30
   sudo aa-status
   # /etc/apparmor.d/usr.bin.slurm-llnl or usr.sbin.slurm-llnl — inspect first
   sudo aa-complain /etc/apparmor.d/usr.bin.slurm-llnl 2>/dev/null || true

5. Verify slurm-llnl integrity and reinstall if anything is altered.

   sudo dpkg -V slurm-llnl
   sudo debsums -c slurm-llnl 2>/dev/null
   sudo apt install --reinstall -y slurm-llnl

6. Correlate findings with /var/log/apt/history.log, /var/log/dpkg.log, and Ubuntu Security Notice USN-8197-1 to pin the change that introduced slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide.

Solution – Primary Fix

Apply the corrective apt transaction referenced by Ubuntu Security Notice USN-8197-1, then reload the affected service:

sudo apt update
sudo apt -y install --only-upgrade slurm-llnl
sudo systemctl daemon-reload
# Service name may differ from pkg name; check first:
systemctl list-unit-files | grep -i slurm | head
sudo systemctl restart slurm-llnl
dpkg -l slurm-llnl | tail -1            # confirm new version
systemctl is-active slurm-llnl

On xenial the standard archive is past EoL for security; enable Ubuntu Pro ESM to receive the fix:

# Older releases use the `ua` command:
sudo ua attach <token>
sudo ua enable esm-infra
sudo ua enable esm-apps
sudo apt update
sudo apt -y install --only-upgrade slurm-llnl

For kernel / glibc / systemd / openssl advisories a reboot (or Livepatch) is required:

sudo apt install -y needrestart
sudo needrestart -r l       # list units that need restart
sudo systemctl reboot       # or: sudo shutdown -r now

Solution – Alternative Approaches

If the primary upgrade is not viable, pick from these:

• Hold the package so apt cannot upgrade it:

  sudo apt-mark hold slurm-llnl
  apt-mark showhold | grep slurm-llnl
  # Release the hold later with:
  sudo apt-mark unhold slurm-llnl

• Pin a known-good version via apt preferences:

  # /etc/apt/preferences.d/slurm-llnl.pref
  Package: slurm-llnl
  Pin: version <good-version>
  Pin-Priority: 1001

• Downgrade to an older version if a regression is suspected:

  apt-cache madison slurm-llnl
  sudo apt install --allow-downgrades -y slurm-llnl=<older-version>

• Investigate AppArmor blocking the new binary; switch to complain briefly, capture denials, then re-enforce:

  sudo aa-complain /etc/apparmor.d/usr.bin.slurm-llnl 2>/dev/null
  # reproduce the failure
  sudo journalctl -k | grep apparmor | tail
  sudo aa-enforce /etc/apparmor.d/usr.bin.slurm-llnl 2>/dev/null

• Take only the security pocket update and defer the full point-release upgrade:

  sudo apt -y install --only-upgrade -t xenial-security slurm-llnl

Verification & Acceptance Criteria

All of these should pass after the fix is applied:

dpkg -l slurm-llnl | tail -1                                  # expected fixed version
apt list --upgradable 2>/dev/null | grep -i security || echo OK
systemctl is-active slurm-llnl
sudo journalctl -u slurm-llnl --since "5 minutes ago" --no-pager | grep -iE "error|fail" || echo OK
sudo ufw status numbered | head
sudo aa-status 2>/dev/null | head -5

The original reproduction for slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide must not trigger across two consecutive runs.

Rollback Plan

Capture state before any change:

apt list --installed 2>/dev/null > /root/apt-pre.txt
dpkg --get-selections > /root/dpkg-pre.txt
# ZFS-on-root (Ubuntu 20.04+ default installer option):
sudo zfs snapshot rpool/ROOT/ubuntu@pre-slurm-llnl
# LVM-on-root:
sudo lvcreate -L 4G -s -n root_pre_patch /dev/<vg>/<root-lv>

To revert:

sudo apt install --allow-downgrades -y slurm-llnl=<old-version>
sudo systemctl daemon-reload
sudo systemctl restart slurm-llnl
# Kernel rollback: pick the prior kernel from the GRUB menu, then:
sudo systemctl reboot
# ZFS rollback (rolls the whole root dataset):
sudo zfs rollback -r rpool/ROOT/ubuntu@pre-slurm-llnl

Prevention & Hardening

Reduce the chance of this recurring on Ubuntu 16.04 (xenial):

• Enable scheduled security updates via unattended-upgrades:

  sudo apt install -y unattended-upgrades update-notifier-common
  sudo dpkg-reconfigure -plow unattended-upgrades
  # /etc/apt/apt.conf.d/50unattended-upgrades:
  Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; };

• Install needrestart so services restart automatically after library upgrades:

  sudo apt install -y needrestart
  # /etc/needrestart/needrestart.conf -> $nrconf{restart} = 'a';

• Attach Ubuntu Pro for ESM (mandatory on this past-EoL release) and Livepatch where supported:

  sudo ua attach <token>
  sudo ua enable esm-infra
  sudo ua enable esm-apps

• Subscribe to ubuntu-security-announce and watch ubuntu.com/security/cves.

• Monitor file integrity with debsums and AIDE:

  sudo apt install -y debsums aide
  sudo debsums -ca
  sudo aideinit && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
  sudo aide --check

• For estate-wide patching, manage with Canonical Landscape:

  sudo apt install -y landscape-client
  sudo landscape-config

• Keep AppArmor profiles in enforce mode and apply CIS Ubuntu Linux Benchmark hardening.

Related Errors & Cross-Refs

Issues that commonly surface alongside slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide: apt lock contention, broken dpkg state, systemd ordering cycles, AppArmor denials, and UFW rule drift. Useful triage:

sudo dpkg --configure -a
sudo apt --fix-broken install
systemd-analyze critical-chain
sudo journalctl -k 2>/dev/null | grep -i apparmor | tail
cat /proc/sys/kernel/tainted

View all ubuntu-16-04 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: Ubuntu Security Notice USN-8197-1. Manual pages useful on Ubuntu 16.04:

man apt
man apt-get
man apt-mark
man dpkg
man systemctl
man journalctl
man ufw
man apparmor
man aa-status
man unattended-upgrades
man ua

Other resources: Ubuntu Security Notices, Ubuntu CVE Tracker, Ubuntu upgrade notes, and per-package notes in /usr/share/doc/slurm-llnl/ for components implicated in slurm-llnl — multiple vulnerabilities (10 CVEs) — patch and remediation guide.