FreeBSD 12 — mcollective — vulnerability — patch and remediation guide

📖 ~4 min read  •  Source: FreeBSD VuXML

VuXML topic: mcollective — cert valication issue

Related CVEs: CVE-2014-3251

Upstream summary: Melissa Stone reports: The MCollective aes_security public key plugin does not correctly validate certs against the CA. By exploiting this vulnerability within a race/initialization window, an attacker with local access could initiate an unauthorized MCollective client connection with a server, and thus control the mcollective plugins running on that server. This vulnerability requires a collective be configured to use the aes_security plugin. Puppet Enterprise and open sourc

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

mcollective runs as a long-lived process on FreeBSD 12, started at boot and holding open sockets (visible in ss -lntp), so a flaw in it stays reachable for as long as the service is up. Exploitation usually means code execution in the daemon’s own security context, memory disclosure from a process holding credentials and cached client data, or a crash that takes the listener down. Workers started before the upgrade keep running the old binary until the service is restarted. mcollective is a genuine long-running service, so restart steps do apply — but confirm the service name first with service -e | grep mcollective, since it often differs from the package name (bind ships named).

Environment & Reproduction

Reproduction targets FreeBSD 12. Confirm release, installed package, and capture baseline state:

freebsd-version -kru
uname -a
pkg info mcollective
pkg query "%n-%v" mcollective
pkg audit -F
service -e

Exercise the workload that uses mcollective while collecting:

tail -200 /var/log/messages
dmesg -a | tail -200
tail -200 /var/log/pkg.log

Root Cause Analysis

Root cause is tracked at FreeBSD VuXML. The FreeBSD ports security team shipped a corrective mcollective port revision; hosts on an outdated build remain exposed. Correlate package logs with system logs and kernel state to isolate the change that triggered the failure mode:

tail -500 /var/log/pkg.log
tail -500 /var/log/messages
sysctl kern.lastpid
sysctl kern.osreldate     # numeric __FreeBSD_version, e.g. 1400097

Quick Triage

Run these checks on FreeBSD 12 to confirm the failure mode and current state of mcollective:

pkg version -v mcollective                # installed vs available version
pkg audit mcollective                     # advisory match for this package
tail -100 /var/log/messages
dmesg -a | tail -100
kldstat                              # kernel module state (for kernel/driver pkgs)
pfctl -sr 2>/dev/null || ipfw list   # only if pf/ipfw is enabled
# mcollective runs as the "mcollective" service - restart it once the upgrade is in place:
# e.g. bind918→named, php83→php-fpm), check it:
service -e | grep -i mcollective && service mcollective status

Step-by-Step Diagnosis

1. List enabled services (only relevant if the package provides one).

   service -e

2. Follow live logs.

   tail -F /var/log/messages
   dmesg

3. Validate firewall rules (skip if neither pf nor ipfw is enabled).

   pfctl -sr -v 2>/dev/null || ipfw show

4. Check package integrity for mcollective.

   pkg check -B mcollective
   pkg check -d mcollective    # verify shared-library deps

5. Reinstall mcollective if integrity check fails.

   pkg install -fy mcollective

6. Correlate findings with /var/log/pkg.log and FreeBSD VuXML to pin the commit that introduced the regression.

Solution – Primary Fix

Install the corrective mcollective port revision referenced by FreeBSD VuXML:

sudo pkg update
sudo pkg upgrade mcollective              # or: sudo pkg upgrade -y for the whole system
# mcollective runs as the "mcollective" service - restart it once the upgrade is in place:
sudo service mcollective restart
pkg audit mcollective                    # confirm no remaining advisory for this package

For ports-tree builders (FreeBSD 13.x and earlier used portsnap; on FreeBSD 14+ the ports tree is fetched with Git):

# FreeBSD 14+ (portsnap was removed):
sudo pkg install -y git-lite
sudo git clone --depth 1 https://git.FreeBSD.org/ports.git /usr/ports
# FreeBSD 13.x and earlier:
# sudo portsnap fetch update
cd /usr/ports/<category>/mcollective
sudo make deinstall reinstall clean

Reboot only if the package ships a kernel module or replaces a shared library used by long-running daemons.

Solution – Alternative Approaches

If the primary fix is not viable, choose from these alternatives:

• Lock the package until the fix is vetted:

  sudo pkg lock mcollective

• Downgrade to a known-good revision. pkg install pkgname-VERSION is not a real downgrade syntax — fetch a specific build instead:

  # 1. Discover available versions across configured repos:
  pkg search -e mcollective
  pkg rquery -r FreeBSD-quarterly '%n-%v' mcollective
  # 2. Install from a specific saved .pkg file:
  sudo pkg add -f /path/to/mcollective-<older-version>.pkg
  # 3. Or switch the host repo to the quarterly branch (see snippet below) and:
  sudo pkg upgrade -fr FreeBSD-quarterly mcollective

• Switch the pkg repository between quarterly and latest by editing /usr/local/etc/pkg/repos/FreeBSD.conf:

  FreeBSD: {
    url: "pkg+https://pkg.FreeBSD.org/${ABI}/quarterly",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/share/keys/pkg",
    enabled: yes
  }

• Isolate the affected service in a jail with stricter firewall rules:

  iocage create -n mcollective-jail -r 12.4-RELEASE
  iocage set allow_raw_sockets=0 mcollective-jail
  # or with Bastille:
  bastille create mcollective-jail 12.4-RELEASE 10.0.0.10

• Replace the service with a vendored static build for the period between exposure detection and full rollout.

Verification & Acceptance Criteria

All of these should pass after the fix:

pkg info mcollective                # shows the expected fixed version
pkg audit mcollective               # no advisory for this package (exit code 0)
tail -50 /var/log/messages   # no new errors after upgrade
# If mcollective ships a service, confirm it is running under its rc.d name:
# service mcollective status

The conditions described in the advisory must no longer be reported for mcollective across two consecutive runs.

Rollback Plan

Capture state before any change (only ZFS root has boot environments — UFS hosts skip bectl):

pkg query "%n-%v" > /root/pkg-pre.txt
# ZFS-on-root only:
sudo bectl create pre-mcollective-patch

To revert if the upgrade is bad, reinstall the previously saved .pkg file:

sudo pkg add -f /var/cache/pkg/mcollective-<previous-version>.pkg
# Or activate the pre-patch boot environment and reboot (ZFS-on-root only):
sudo bectl activate pre-mcollective-patch
sudo shutdown -r now

For kernel/loader changes on a UFS host, boot the previous kernel from the loader prompt (press 3 at the menu, then boot kernel.old).

Prevention & Hardening

Prevent recurrence on FreeBSD 12 hosts running mcollective:

• Enable the daily security pkg audit in /etc/periodic.conf:

  daily_status_security_pkgaudit_enable="YES"

• Subscribe to freebsd-security-notifications at lists.freebsd.org.

• Mirror through a local pkg repository managed by poudriere:

  poudriere jail -c -j 12amd64 -v 12.4-RELEASE
  poudriere ports -c -p default
  poudriere bulk -j 12amd64 -p default <category>/mcollective

• Version-pin sensitive packages:

  sudo pkg lock mcollective

• Take an automatic ZFS boot-environment snapshot before every upgrade (ZFS root only):

  sudo bectl create pre-upgrade-$(date +%Y%m%d)

• Monitor file integrity (create a baseline, verify against it later):

  # Create a baseline (use -c; target /usr/local/etc, /etc, /boot — NOT /):
  sudo mtree -c -K sha256digest -p /usr/local/etc > /var/db/usr-local-etc.mtree
  sudo mtree -c -K sha256digest -p /etc          > /var/db/etc.mtree
  # Verify later:
  sudo mtree -p /usr/local/etc < /var/db/usr-local-etc.mtree
  # Or with AIDE for a richer ruleset:
  sudo pkg install -y aide && sudo aide --init && sudo aide --check

• Harden jails with allow.* tunables in /etc/jail.conf:

  mcollective_jail {
    allow.raw_sockets = 0;
    allow.sysvipc    = 0;
    allow.mount      = 0;
    allow.chflags    = 0;
  }

Related Errors & Cross-Refs

Issues that commonly surface alongside a mcollective update: pkg lock contention, mismatched ABI after kernel/userland skew, pf rule drift, and stale shared-library references after upgrade. Triage with:

freebsd-version -kru
uname -K
pkg check -d
pfctl -sr

View all freebsd-12 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: FreeBSD VuXML. Useful manual pages on FreeBSD 12:

man pkg
man freebsd-update
man pfctl
man ipfw
man bectl
man periodic.conf

Other resources: the FreeBSD Handbook, the FreeBSD Security Advisories at security.freebsd.org, and the /usr/ports/UPDATING file for port-specific notes implicated in this advisory.