FreeBSD 14 — ceph — multiple vulnerabilities (3 CVEs) — patch and remediation guide

📖 ~4 min read  •  Source: FreeBSD VuXML

VuXML topic: ceph14 — HTTP header injection via CORS ExposeHeader tag

Related CVEs: CVE-2020-10753 CVE-2020-1759 CVE-2020-1760

Upstream summary: Red Hat bugzilla reports: A flaw was found in the Red Hat Ceph Storage RadosGW (Ceph Object Gateway). The vulnerability is related to the injection of HTTP headers via a CORS ExposeHeader tag. The newline character in the ExposeHeader tag in the CORS configuration file generates a header injection in the response when the CORS request is made.

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

ceph runs as a long-lived process on FreeBSD 14, started at boot and holding open sockets (visible in ss -lntp), so a flaw in it stays reachable for as long as the service is up. Exploitation usually means code execution in the daemon’s own security context, memory disclosure from a process holding credentials and cached client data, or a crash that takes the listener down. Workers started before the upgrade keep running the old binary until the service is restarted. ceph is a genuine long-running service, so restart steps do apply — but confirm the service name first with service -e | grep ceph, since it often differs from the package name (bind ships named).

Environment & Reproduction

Reproduction targets FreeBSD 14. Confirm release, installed package, and capture baseline state:

freebsd-version -kru
uname -a
pkg info ceph
pkg query "%n-%v" ceph
pkg audit -F
service -e

Exercise the workload that uses ceph while collecting:

tail -200 /var/log/messages
dmesg -a | tail -200
tail -200 /var/log/pkg.log

Root Cause Analysis

Root cause is tracked at FreeBSD VuXML. The FreeBSD ports security team shipped a corrective ceph port revision; hosts on an outdated build remain exposed. Correlate package logs with system logs and kernel state to isolate the change that triggered the failure mode:

tail -500 /var/log/pkg.log
tail -500 /var/log/messages
sysctl kern.lastpid
sysctl kern.osreldate     # numeric __FreeBSD_version, e.g. 1400097

Quick Triage

Run these checks on FreeBSD 14 to confirm the failure mode and current state of ceph:

pkg version -v ceph                # installed vs available version
pkg audit ceph                     # advisory match for this package
tail -100 /var/log/messages
dmesg -a | tail -100
kldstat                              # kernel module state (for kernel/driver pkgs)
pfctl -sr 2>/dev/null || ipfw list   # only if pf/ipfw is enabled
# ceph runs as the "ceph.target" service - restart it once the upgrade is in place:
# e.g. bind918→named, php83→php-fpm), check it:
service -e | grep -i ceph && service ceph.target status

Step-by-Step Diagnosis

1. List enabled services (only relevant if the package provides one).

   service -e

2. Follow live logs.

   tail -F /var/log/messages
   dmesg

3. Validate firewall rules (skip if neither pf nor ipfw is enabled).

   pfctl -sr -v 2>/dev/null || ipfw show

4. Check package integrity for ceph.

   pkg check -B ceph
   pkg check -d ceph    # verify shared-library deps

5. Reinstall ceph if integrity check fails.

   pkg install -fy ceph

6. Correlate findings with /var/log/pkg.log and FreeBSD VuXML to pin the commit that introduced the regression.

Solution – Primary Fix

Install the corrective ceph port revision referenced by FreeBSD VuXML:

sudo pkg update
sudo pkg upgrade ceph              # or: sudo pkg upgrade -y for the whole system
# ceph runs as the "ceph.target" service - restart it once the upgrade is in place:
sudo service ceph.target restart
pkg audit ceph                    # confirm no remaining advisory for this package

For ports-tree builders (FreeBSD 13.x and earlier used portsnap; on FreeBSD 14+ the ports tree is fetched with Git):

# FreeBSD 14+ (portsnap was removed):
sudo pkg install -y git-lite
sudo git clone --depth 1 https://git.FreeBSD.org/ports.git /usr/ports
# FreeBSD 13.x and earlier:
# sudo portsnap fetch update
cd /usr/ports/<category>/ceph
sudo make deinstall reinstall clean

Reboot only if the package ships a kernel module or replaces a shared library used by long-running daemons.

Solution – Alternative Approaches

If the primary fix is not viable, choose from these alternatives:

• Lock the package until the fix is vetted:

  sudo pkg lock ceph

• Downgrade to a known-good revision. pkg install pkgname-VERSION is not a real downgrade syntax — fetch a specific build instead:

  # 1. Discover available versions across configured repos:
  pkg search -e ceph
  pkg rquery -r FreeBSD-quarterly '%n-%v' ceph
  # 2. Install from a specific saved .pkg file:
  sudo pkg add -f /path/to/ceph-<older-version>.pkg
  # 3. Or switch the host repo to the quarterly branch (see snippet below) and:
  sudo pkg upgrade -fr FreeBSD-quarterly ceph

• Switch the pkg repository between quarterly and latest by editing /usr/local/etc/pkg/repos/FreeBSD.conf:

  FreeBSD: {
    url: "pkg+https://pkg.FreeBSD.org/${ABI}/quarterly",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/share/keys/pkg",
    enabled: yes
  }

• Isolate the affected service in a jail with stricter firewall rules:

  iocage create -n ceph-jail -r 14.1-RELEASE
  iocage set allow_raw_sockets=0 ceph-jail
  # or with Bastille:
  bastille create ceph-jail 14.1-RELEASE 10.0.0.10

• Replace the service with a vendored static build for the period between exposure detection and full rollout.

Verification & Acceptance Criteria

All of these should pass after the fix:

pkg info ceph                # shows the expected fixed version
pkg audit ceph               # no advisory for this package (exit code 0)
tail -50 /var/log/messages   # no new errors after upgrade
# If ceph ships a service, confirm it is running under its rc.d name:
# service ceph.target status

The conditions described in the advisory must no longer be reported for ceph across two consecutive runs.

Rollback Plan

Capture state before any change (only ZFS root has boot environments — UFS hosts skip bectl):

pkg query "%n-%v" > /root/pkg-pre.txt
# ZFS-on-root only:
sudo bectl create pre-ceph-patch

To revert if the upgrade is bad, reinstall the previously saved .pkg file:

sudo pkg add -f /var/cache/pkg/ceph-<previous-version>.pkg
# Or activate the pre-patch boot environment and reboot (ZFS-on-root only):
sudo bectl activate pre-ceph-patch
sudo shutdown -r now

For kernel/loader changes on a UFS host, boot the previous kernel from the loader prompt (press 3 at the menu, then boot kernel.old).

Prevention & Hardening

Prevent recurrence on FreeBSD 14 hosts running ceph:

• Enable the daily security pkg audit in /etc/periodic.conf:

  daily_status_security_pkgaudit_enable="YES"

• Subscribe to freebsd-security-notifications at lists.freebsd.org.

• Mirror through a local pkg repository managed by poudriere:

  poudriere jail -c -j 14amd64 -v 14.1-RELEASE
  poudriere ports -c -p default
  poudriere bulk -j 14amd64 -p default <category>/ceph

• Version-pin sensitive packages:

  sudo pkg lock ceph

• Take an automatic ZFS boot-environment snapshot before every upgrade (ZFS root only):

  sudo bectl create pre-upgrade-$(date +%Y%m%d)

• Monitor file integrity (create a baseline, verify against it later):

  # Create a baseline (use -c; target /usr/local/etc, /etc, /boot — NOT /):
  sudo mtree -c -K sha256digest -p /usr/local/etc > /var/db/usr-local-etc.mtree
  sudo mtree -c -K sha256digest -p /etc          > /var/db/etc.mtree
  # Verify later:
  sudo mtree -p /usr/local/etc < /var/db/usr-local-etc.mtree
  # Or with AIDE for a richer ruleset:
  sudo pkg install -y aide && sudo aide --init && sudo aide --check

• Harden jails with allow.* tunables in /etc/jail.conf:

  ceph_jail {
    allow.raw_sockets = 0;
    allow.sysvipc    = 0;
    allow.mount      = 0;
    allow.chflags    = 0;
  }

Related Errors & Cross-Refs

Issues that commonly surface alongside a ceph update: pkg lock contention, mismatched ABI after kernel/userland skew, pf rule drift, and stale shared-library references after upgrade. Triage with:

freebsd-version -kru
uname -K
pkg check -d
pfctl -sr

View all freebsd-14 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: FreeBSD VuXML. Useful manual pages on FreeBSD 14:

man pkg
man freebsd-update
man pfctl
man ipfw
man bectl
man periodic.conf

Other resources: the FreeBSD Handbook, the FreeBSD Security Advisories at security.freebsd.org, and the /usr/ports/UPDATING file for port-specific notes implicated in this advisory.