VMware ESXi 6.7 — openslp — multiple ESXi vulnerabilities (2 CVEs) — VIB / vLCM patch and remediation guide

📖 ~4 min read  •  Source: VMware advisory VMSA-2021-0002

Related CVEs: CVE-2020-3992 CVE-2021-21974

Fixed image profile / build: ESXi670-202102401-SG

Upstream summary: ESXi contains a heap-overflow vulnerability in the OpenSLP service (CVE-2020-3992 / CVE-2021-21974) that allows a malicious actor residing in the management network with port 427 access to execute remote code. This advisory is the root cause behind the ESXiArgs ransomware wave that swept un-patched ESXi 6.7 and 7.0 hosts.

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

Nothing executes openslp on its own; its code runs inside whatever links against or imports it, so the exposed surface on a VMware ESXi 6.7 host is the union of every process that has loaded it — web workers, cron jobs, mail handlers, backup agents. The package manager replaces the files on disk, but processes already running keep the old build mapped, and virtualenvs, vendored copies and container images each carry their own copy, so a host can report itself patched while still running vulnerable code. openslp is a library — code loaded by other programs rather than a daemon — so there is no service to restart and /etc/init.d/ openslp will simply report no such service; restart its consumers instead.

Environment & Reproduction

Reproduction targets VMware ESXi 6.7. Confirm the running build, installed image profile and the affected component VIBs first — ESXi does not use a traditional package manager; everything is a VIB inside an image profile:

esxcli system version get
vmware -lv
esxcli software profile get
esxcli software vib list | grep -i openslp
esxcli software vib get -n openslp 2>/dev/null | head -40

Trigger the workflow that exposes openslp while tailing the hypervisor logs:

tail -200 /var/log/vmkernel.log
tail -200 /var/log/hostd.log
tail -200 /var/log/vpxa.log
tail -200 /var/log/syslog.log
# For an evidence bundle to attach to a Broadcom / VMware support case:
vm-support -w /vmfs/volumes/datastore1

Root Cause Analysis

Root cause is documented in VMware advisory VMSA-2021-0002. Broadcom / VMware shipped the corrective VIB inside a new ESXi image profile for VMware ESXi 6.7; hosts still running the pre-patch build retain the vulnerable openslp code path. Correlate the host history with logs to confirm:

esxcli software profile get
esxcli software vib list --rebooting-image 2>/dev/null | head
esxcli software vib get -n openslp 2>/dev/null | head -40
grep -i VMSA /var/log/esxupdate.log 2>/dev/null | tail
tail -200 /var/log/vmkernel.log

Quick Triage

Run these on VMware ESXi 6.7 (DCUI / SSH) to capture the current state of openslp and the host:

esxcli system version get                              # build & update level
esxcli software profile get                            # current image profile
esxcli software vib list | grep -i openslp
esxcli system maintenanceMode get
vim-cmd hostsvc/runtimeinfo | head -40                 # host runtime/config snapshot
esxcli network firewall ruleset list | head
esxcli vsan cluster get 2>/dev/null || echo 'vSAN not configured'
# DRS / cluster context (PowerCLI on a jump host):
# Connect-VIServer vc01 -User [email protected]
# Get-VMHost esx01 | Select Version, Build, ConnectionState

Step-by-Step Diagnosis

1. Confirm the openslp VIB and its origin.

   esxcli software vib list | grep -i openslp
   esxcli software vib get -n openslp 2>/dev/null | head -40

2. Tail the hypervisor logs while reproducing the issue.

   tail -F /var/log/vmkernel.log &
   tail -F /var/log/hostd.log &
   tail -F /var/log/vpxa.log &

3. Inspect the ESXi firewall posture (management network exposure for openslp / vpxa / hostd ports is the most common amplifier of these advisories).

   esxcli network firewall get
   esxcli network firewall ruleset list
   esxcli network firewall ruleset rule list | head -40

4. Check service state and cycle the affected daemon if it is hung.

   /etc/init.d/hostd status
   /etc/init.d/vpxa status
   /etc/init.d/hostd restart   # restarts management agent only — no VM impact
   /etc/init.d/vpxa restart

5. Confirm cluster context — DRS automation level, vMotion network and active maintenance-mode plans dictate the patch window.

   # PowerCLI on a jump host:
   Connect-VIServer vc01 -User [email protected]
   Get-Cluster Prod | Select Name, DrsEnabled, DrsAutomationLevel
   Get-VMHost esx01 | Get-View | Select Runtime | Format-List

6. Correlate findings with /var/log/esxupdate.log, /var/log/syslog.log and VMware advisory VMSA-2021-0002 to pin the build that introduced openslp.

Solution – Primary Fix

Apply the corrective ESXi image profile referenced by VMware advisory VMSA-2021-0002. The workflow is: evacuate workloads via DRS / vMotion, enter maintenance mode, pull the new profile (online from the VMware HCL or from an offline depot bundle), reboot, and exit maintenance mode.

# 1. (PowerCLI on jump host) Evacuate the host using DRS / vMotion:
Connect-VIServer vc01 -User [email protected]
Get-VM -Location esx01 | Move-VM -Destination esx02   # bulk vMotion
Set-VMHost esx01 -State Maintenance -Evacuate

# 2. On the ESXi host (SSH), enter maintenance mode from the CLI too:
esxcli system maintenanceMode set --enable=true

# 3a. Online patch direct from the VMware / Broadcom HCL depot:
esxcli software profile update 
  -p ESXi670-202102401-SG 
  -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

# 3b. Or offline patch from a downloaded depot ZIP on a datastore:
esxcli software profile update 
  -d /vmfs/volumes/datastore1/VMware-ESXi-8.0U3-depot.zip 
  -p ESXi670-202102401-SG

# 4. Reboot to commit the new profile:
reboot

# 5. After reboot — verify and exit maintenance mode:
esxcli system version get
esxcli software profile get
esxcli system maintenanceMode set --enable=false

For a whole vSphere cluster, use vSphere Lifecycle Manager (vLCM) image-based remediation — this is the only supported workflow for ESXi 7 and 8 fleets and handles DRS evacuation, sequenced reboots and post-checks automatically:

Connect-VIServer vc01 -User [email protected]
Get-Cluster Prod | Test-LcmClusterCompliance
Set-Cluster Prod -Remediate -Confirm:$false
# Or via govc:
govc cluster.remediate -cluster=Prod

Solution – Alternative Approaches

If the primary image-profile update is not viable right now, choose from these:

• Apply only the relevant VIB(s) from an offline depot instead of the full profile (smaller blast radius, but you lose image-profile consistency):

  esxcli software sources vib list 
    -d /vmfs/volumes/datastore1/VMware-ESXi-8.0U3-depot.zip | head
  esxcli software vib update 
    -d /vmfs/volumes/datastore1/VMware-ESXi-8.0U3-depot.zip 
    -n openslp

• Apply the VMware-documented workaround / mitigation listed on the VMSA page (for OpenSLP-class advisories the documented workaround is to disable the SLP service entirely on the management network):

  esxcli network firewall ruleset set --enabled=false --ruleset-id=CIMSLP
  /etc/init.d/slpd stop
  chkconfig slpd off
  esxcli network firewall ruleset list | grep -i slp

• Roll forward only the affected hosts and version-pin the rest of the cluster until a maintenance window opens — set the vLCM baseline on a per-host basis instead of cluster-wide:

  Connect-VIServer vc01 -User [email protected]
  $img = Get-LcmImage -Name 'ESXi-8.0U3-stable'
  Set-VMHost esx01 -LcmImage $img -Remediate

• For an emergency rebuild, use the new image profile to stateless-boot the host via Auto Deploy rules; the next PXE boot picks up the patched profile automatically:

  Connect-VIServer vc01 -User [email protected]
  Get-DeployRule | Select Name, ItemList
  Copy-DeployRule -DeployRule prod-rule -ReplaceItem $newImage

• If a guest-to-host escape is feared and no patch window is available, suspend untrusted VMs and isolate the host on a dedicated VLAN until the VIB is applied:

  vim-cmd vmsvc/getallvms
  vim-cmd vmsvc/power.suspend <vmid>
  esxcli network vswitch standard portgroup set -p 'Quarantine' -v 999

Verification & Acceptance Criteria

All of these should pass after the patch window:

esxcli system version get                                # expected fixed build
esxcli software profile get                              # expected fixed profile
esxcli software vib list | grep -i openslp
esxcli system maintenanceMode get                        # should be 'Disabled'
tail -100 /var/log/vmkernel.log | grep -iE "error|fail" || echo OK
tail -100 /var/log/hostd.log    | grep -iE "error|fail" || echo OK
esxcli network firewall ruleset list | head
# Cluster-level compliance check via PowerCLI:
Get-Cluster Prod | Test-LcmClusterCompliance

The original reproduction for openslp must not trigger across two consecutive runs, and the VMware advisory VMSA-2021-0002 must report the host as remediated.

Rollback Plan

Capture state before any change:

esxcli system version get        > /tmp/esxi-pre.txt
esxcli software profile get      >> /tmp/esxi-pre.txt
esxcli software vib list         >> /tmp/esxi-pre.txt
# Persist host config — survives a reboot/reinstall:
vim-cmd hostsvc/firmware/backup_config
# (Download the resulting bundle from /scratch/downloads/<id>/configBundle-*.tgz)

To revert if the new profile is bad:

# 1. Enter maintenance mode and reboot — ESXi keeps the previous boot bank:
esxcli system maintenanceMode set --enable=true
reboot
# 2. At the boot loader, press <Shift+R> to roll back to the previous image
#    profile (ESXi keeps two boot banks for exactly this).
# 3. Or, from a working SSH session, force the previous profile back online:
esxcli software profile update -p <previous-profile> 
  -d /vmfs/volumes/datastore1/<previous-depot>.zip
reboot
esxcli system maintenanceMode set --enable=false

Prevention & Hardening

Reduce the chance of this recurring on VMware ESXi 6.7:

• Drive every patch window through vSphere Lifecycle Manager (vLCM) image baselines — image-based remediation has been the supported workflow since vSphere 7 and is the only one that survives a Broadcom subscription audit:

  Connect-VIServer vc01 -User [email protected]
  New-LcmImage -Name 'ESXi-8.0U3-stable' -Version 8.0.3 -BaseImage <build>
  Set-Cluster Prod -LcmImage 'ESXi-8.0U3-stable'
  Get-Cluster Prod | Test-LcmClusterCompliance

• Subscribe to the VMware / Broadcom security advisories feed and forward to your SIEM:

  # Watch the VMSA feed (mirror or scrape via your own collector):
  curl -sSf https://www.vmware.com/security/advisories.xml | head

• Disable SSH and the DCUI on production hosts; enable them only for the patch window. Keep Lockdown Mode on:

  /etc/init.d/SSH stop
  vim-cmd hostsvc/disable_ssh
  vim-cmd hostsvc/lockdown_mode_enter

• Minimise the ESXi firewall surface — disable rulesets you do not need (CIMSLP, cimhttpserver, legacy slpd):

  esxcli network firewall ruleset list
  esxcli network firewall ruleset set --enabled=false --ruleset-id=CIMSLP
  esxcli network firewall ruleset set --enabled=false --ruleset-id=cimhttpserver

• Pin host configurations with a Host Profile and reapply after every patch:

  Connect-VIServer vc01 -User [email protected]
  Get-VMHostProfile
  Apply-VMHostProfile -Entity esx01 -Profile 'Prod-baseline'

• Enable Secure Boot + TPM 2.0 attestation so the hypervisor refuses to load unsigned VIBs or tampered kernel modules:

  esxcli system settings encryption get
  esxcli system security keypersistence enable

• Apply VMware vSphere Security Configuration Guide hardening for VMware ESXi 6.7 and remove unused VIBs.

Related Errors & Cross-Refs

Issues that commonly surface alongside a openslp update: hostd / vpxa disconnects from vCenter, ramdisk-full conditions on /var/log, vSAN object inaccessibility during reboots, stale DRS recommendations, and VIB acceptance-level mismatches. Useful triage:

esxcli system stats installtime get
esxcli software acceptance get
esxcli storage core device list | head
esxcli vsan health cluster get 2>/dev/null
vdf -h
tail /var/log/vmkernel.log

View all vmware-esxi-6-7 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: VMware advisory VMSA-2021-0002. Useful CLI documentation on VMware ESXi 6.7:

esxcli --help
esxcli software --help
esxcli software profile --help
esxcli system maintenanceMode --help
esxcli network firewall --help
vim-cmd help
vmware -lv
vm-support -h

Other resources: docs.vmware.com, VMware Security Advisories index, VMware / Broadcom Knowledge Base, and Broadcom support content notifications for components implicated in this advisory.

View all VMware ESXi 6.7 tutorials on the Tutorials Hub →