Ubuntu 20.04 — prometheus-alertmanager — vulnerability — patch and remediation guide

📖 ~4 min read  •  Source: Ubuntu Security Notice USN-6935-1

Related CVEs: CVE-2023-40577

Upstream summary: It was discovered that prometheus-alertmanager didn't properly sanitize
input it received through an API endpoint. An attacker with permission to
send requests to this endpoint could potentially inject arbitrary code.

On Ubuntu 20.04 LTS and Ubuntu 22.04 LTS, this vulnerability is only
present if the UI has been explicitly activated.

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

prometheus-alertmanager runs as a long-lived process on Ubuntu 20.04, started at boot and holding open sockets (visible in ss -lntp), so a flaw in it stays reachable for as long as the service is up. Exploitation usually means code execution in the daemon’s own security context, memory disclosure from a process holding credentials and cached client data, or a crash that takes the listener down. Workers started before the upgrade keep running the old binary until the unit is reloaded or restarted. prometheus-alertmanager is a genuine long-running service, so restart steps do apply — but confirm the unit name first with systemctl list-unit-files | grep -i prometheus-alertmanager, since it often differs from the package name (bind ships named.service).

Environment & Reproduction

Reproduction targets Ubuntu 20.04 (focal). Confirm release and installed package:

lsb_release -a
cat /etc/os-release
dpkg -l prometheus-alertmanager | tail -2
apt-cache policy prometheus-alertmanager
uname -r

Exercise the workload that uses prometheus-alertmanager while collecting:

sudo journalctl -u prometheus-alertmanager -b --no-pager | tail -200
sudo journalctl -xe --no-pager | tail -200
sudo tail -200 /var/log/apt/history.log
sudo tail -200 /var/log/kern.log | grep -i apparmor

Root Cause Analysis

Root cause is documented in Ubuntu Security Notice USN-6935-1. Canonical security maintainers shipped fixes in the corresponding prometheus-alertmanager update for Ubuntu 20.04; running an outdated build leaves the host exposed to the failure modes described in the advisory. Fixes land in the focal-security pocket of the main archive. Correlate apt history with the journal:

grep -A2 -B2 prometheus-alertmanager /var/log/apt/history.log
zgrep -A2 -B2 prometheus-alertmanager /var/log/apt/history.log.*.gz 2>/dev/null
cat /proc/sys/kernel/tainted   # non-zero = tainted kernel / out-of-tree modules

Quick Triage

Run these on Ubuntu 20.04 to capture the current state of prometheus-alertmanager:

dpkg -l prometheus-alertmanager | tail -1                  # installed version
dpkg -V prometheus-alertmanager                             # verify shipped files
sudo apt update && apt list --upgradable 2>/dev/null | grep -i security
systemctl is-active prometheus-alertmanager
sudo ufw status verbose 2>/dev/null | head -20
sudo aa-status 2>/dev/null | head -20
# If prometheus-alertmanager ships a service unit (unit/job name often differs from pkg name, e.g.
# bind9→named, apache2→apache2, postgresql-NN→postgresql@NN-main):
systemctl list-unit-files | grep -i prometheus | head

Step-by-Step Diagnosis

1. List failing services.

   systemctl --failed --no-pager

2. Tail the journal / syslog for prometheus-alertmanager.

   sudo journalctl -u prometheus-alertmanager -f --no-pager
   sudo journalctl -xe -f --no-pager

3. Inspect UFW (Uncomplicated Firewall) state.

   sudo ufw status numbered
   sudo ufw show added
   sudo iptables -L -n -v | head -30

4. Surface AppArmor denials and switch the profile to complain mode if needed.

   sudo journalctl -k 2>/dev/null | grep -i 'apparmor="DENIED"' | tail -30
   sudo aa-status
   # /etc/apparmor.d/usr.bin.prometheus-alertmanager or usr.sbin.prometheus-alertmanager — inspect first
   sudo aa-complain /etc/apparmor.d/usr.bin.prometheus-alertmanager 2>/dev/null || true

5. Verify prometheus-alertmanager integrity and reinstall if anything is altered.

   sudo dpkg -V prometheus-alertmanager
   sudo debsums -c prometheus-alertmanager 2>/dev/null
   sudo apt install --reinstall -y prometheus-alertmanager

6. Correlate findings with /var/log/apt/history.log, /var/log/dpkg.log, and Ubuntu Security Notice USN-6935-1 to pin the change that introduced the regression.

Solution – Primary Fix

Apply the corrective apt transaction referenced by Ubuntu Security Notice USN-6935-1, then reload the affected service:

sudo apt update
sudo apt -y install --only-upgrade prometheus-alertmanager
sudo systemctl daemon-reload
# Service name may differ from pkg name; check first:
systemctl list-unit-files | grep -i prometheus | head
sudo systemctl restart prometheus-alertmanager
dpkg -l prometheus-alertmanager | tail -1            # confirm new version
systemctl is-active prometheus-alertmanager

For kernel / glibc / systemd / openssl advisories a reboot (or Livepatch) is required:

sudo apt install -y needrestart
sudo needrestart -r l       # list units that need restart
sudo systemctl reboot       # or: sudo shutdown -r now
# Livepatch (Ubuntu Pro) avoids reboot for many kernel CVEs:
sudo canonical-livepatch status
sudo canonical-livepatch refresh

Solution – Alternative Approaches

If the primary upgrade is not viable, pick from these:

• Hold the package so apt cannot upgrade it:

  sudo apt-mark hold prometheus-alertmanager
  apt-mark showhold | grep prometheus-alertmanager
  # Release the hold later with:
  sudo apt-mark unhold prometheus-alertmanager

• Pin a known-good version via apt preferences:

  # /etc/apt/preferences.d/prometheus-alertmanager.pref
  Package: prometheus-alertmanager
  Pin: version <good-version>
  Pin-Priority: 1001

• Downgrade to an older version if a regression is suspected:

  apt-cache madison prometheus-alertmanager
  sudo apt install --allow-downgrades -y prometheus-alertmanager=<older-version>

• Investigate AppArmor blocking the new binary; switch to complain briefly, capture denials, then re-enforce:

  sudo aa-complain /etc/apparmor.d/usr.bin.prometheus-alertmanager 2>/dev/null
  # reproduce the failure
  sudo journalctl -k | grep apparmor | tail
  sudo aa-enforce /etc/apparmor.d/usr.bin.prometheus-alertmanager 2>/dev/null

• Apply Canonical Livepatch (Ubuntu Pro) to land kernel fixes without reboot:

  sudo canonical-livepatch status
  sudo canonical-livepatch refresh

• Take only the security pocket update and defer the full point-release upgrade:

  sudo apt -y install --only-upgrade -t focal-security prometheus-alertmanager

Verification & Acceptance Criteria

All of these should pass after the fix is applied:

dpkg -l prometheus-alertmanager | tail -1                                  # expected fixed version
apt list --upgradable 2>/dev/null | grep -i security || echo OK
systemctl is-active prometheus-alertmanager
sudo journalctl -u prometheus-alertmanager --since "5 minutes ago" --no-pager | grep -iE "error|fail" || echo OK
sudo ufw status numbered | head
sudo aa-status 2>/dev/null | head -5

The conditions described in the advisory must no longer be reported for prometheus-alertmanager across two consecutive runs.

Rollback Plan

Capture state before any change:

apt list --installed 2>/dev/null > /root/apt-pre.txt
dpkg --get-selections > /root/dpkg-pre.txt
# ZFS-on-root (Ubuntu 20.04+ default installer option):
sudo zfs snapshot rpool/ROOT/ubuntu@pre-prometheus-alertmanager
# LVM-on-root:
sudo lvcreate -L 4G -s -n root_pre_patch /dev/<vg>/<root-lv>

To revert:

sudo apt install --allow-downgrades -y prometheus-alertmanager=<old-version>
sudo systemctl daemon-reload
sudo systemctl restart prometheus-alertmanager
# Kernel rollback: pick the prior kernel from the GRUB menu, then:
sudo systemctl reboot
# ZFS rollback (rolls the whole root dataset):
sudo zfs rollback -r rpool/ROOT/ubuntu@pre-prometheus-alertmanager

Prevention & Hardening

Reduce the chance of this recurring on Ubuntu 20.04 (focal):

• Enable scheduled security updates via unattended-upgrades:

  sudo apt install -y unattended-upgrades update-notifier-common
  sudo dpkg-reconfigure -plow unattended-upgrades
  # /etc/apt/apt.conf.d/50unattended-upgrades:
  Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; };

• Install needrestart so services restart automatically after library upgrades:

  sudo apt install -y needrestart
  # /etc/needrestart/needrestart.conf -> $nrconf{restart} = 'a';

• Attach Ubuntu Pro to enable Livepatch and extended security coverage:

  sudo pro attach <token>
  sudo pro enable livepatch

• Subscribe to ubuntu-security-announce and watch ubuntu.com/security/cves.

• Monitor file integrity with debsums and AIDE:

  sudo apt install -y debsums aide
  sudo debsums -ca
  sudo aideinit && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
  sudo aide --check

• For estate-wide patching, manage with Canonical Landscape:

  sudo apt install -y landscape-client
  sudo landscape-config

• Keep AppArmor profiles in enforce mode and apply CIS Ubuntu Linux Benchmark hardening.

Related Errors & Cross-Refs

Issues that commonly surface alongside a prometheus-alertmanager update: apt lock contention, broken dpkg state, systemd ordering cycles, AppArmor denials, and UFW rule drift. Useful triage:

sudo dpkg --configure -a
sudo apt --fix-broken install
systemd-analyze critical-chain
sudo journalctl -k 2>/dev/null | grep -i apparmor | tail
cat /proc/sys/kernel/tainted

View all ubuntu-20-04 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: Ubuntu Security Notice USN-6935-1. Manual pages useful on Ubuntu 20.04:

man apt
man apt-get
man apt-mark
man dpkg
man systemctl
man journalctl
man ufw
man apparmor
man aa-status
man unattended-upgrades
man canonical-livepatch
man pro

Other resources: Ubuntu Security Notices, Ubuntu CVE Tracker, Ubuntu upgrade notes, and per-package notes in /usr/share/doc/prometheus-alertmanager/ for components implicated in this advisory.