Amazon Linux 2023 — nvidia-container-toolkit — multiple vulnerabilities (7 CVEs) — patch and remediation guide

📖 ~4 min read  •  Source: Amazon Linux advisory ALAS2023NVIDIA-2025-125

Related CVEs: CVE-2025-23266 CVE-2025-23267 CVE-2025-23359 CVE-2024-0135 CVE-2024-0136 CVE-2024-0137 CVE-2024-0132

Upstream summary: NVIDIA Container Toolkit for all platforms contains a vulnerability in some hooks used to initialize the container, where an attacker could execute arbitrary code with elevated permissions. A successful exploit of this vulnerability might lead to escalation of privileges, data tampering, information disclosure, and denial of service. (CVE-2025-23266) NVIDIA Container Toolkit for all platforms contains a vulnerability in the update-ldcache hook, where an attacker could cause a

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

nvidia-container-toolkit runs when an image is pulled, unpacked, inspected or a container is started, so a defect here is normally reached through untrusted image content or a registry response rather than inbound traffic to a listening port. On Amazon Linux 2023 that shows up as escapes from the container’s namespaces, files written outside the intended root during buildah or podman build, mount and capability handling that grants more than was requested, and images produced before the update carrying the defect onward to every host that pulls them. nvidia-container-toolkit is container tooling — command-line binaries invoked per operation, though a node runtime such as cri-o does run as a unit — so restart steps apply to that runtime and to containers still running the old code, not to nvidia-container-toolkit itself.

Environment & Reproduction

Reproduction targets Amazon Linux 2023. Confirm release and the installed package:

cat /etc/system-release
cat /etc/os-release
rpm -q nvidia-container-toolkit
dnf info nvidia-container-toolkit | head -20

Exercise the workload that uses nvidia-container-toolkit while collecting:

sudo needs-restarting -u  # --useronly: restrict to the invoking user's processes (useful on shared/app hosts)
sudo lsof -n +L1  # link count < 1 = file deleted but still open/mmap'd; this is the ground truth after an RPM replaces a .so (package: lsof)
sudo lsof -n +L1 2>/dev/null | awk 'NR>1 {print $1, $2}' | sort -u  # condensed COMMAND + PID list of every process still running old code
sudo journalctl -xe --no-pager | tail -200
sudo tail -200 /var/log/dnf.log
sudo tail -200 /var/log/audit/audit.log
# For an evidence bundle bundle with sosreport (Amazon Linux ships it):
sudo sosreport --batch

For fleet-wide visibility, query Amazon Inspector and SSM at the same time:

aws inspector2 list-findings 
  --filter-criteria 'awsAccountId={comparison=EQUALS,value=<account-id>}' 
  --max-results 50
aws ssm describe-instance-patches --instance-id <i-xxxx> | head -40

Root Cause Analysis

Root cause is documented in Amazon Linux advisory ALAS2023NVIDIA-2025-125. The Amazon Linux Security Team shipped fixes in the corresponding nvidia-container-toolkit update for Amazon Linux 2023; running an outdated AMI or unpatched instance leaves the host exposed to the failure modes described in the advisory. Correlate dnf history with system logs:

sudo dnf history | head
sudo dnf history list nvidia-container-toolkit
sudo dnf history info <id>
sudo ausearch -m AVC,USER_AVC -ts today | tail -100
cat /proc/sys/kernel/tainted   # non-zero = tainted kernel / out-of-tree modules

Quick Triage

Run these on Amazon Linux 2023 to capture the current state of nvidia-container-toolkit:

rpm -q nvidia-container-toolkit                              # installed NVR
rpm -V nvidia-container-toolkit                              # verify shipped files
sudo dnf check-update --security
sudo dnf updateinfo list cves
systemctl --failed --no-pager
sudo firewall-cmd --list-all 2>/dev/null || sudo iptables -L -n
getenforce && sestatus
dnf repoquery -l nvidia-container-toolkit 2>/dev/null | grep -E '/lib/systemd/system/.*\.(service|socket|timer)$'  # same check WITHOUT installing (EL7/Amazon Linux 2: yum install yum-utils, then: repoquery -l nvidia-container-toolkit)
systemctl show --no-pager -p Type,RemainAfterExit <UNIT>  # Type=oneshot (often paired with a .timer) means a one-shot job, NOT a daemon -- do not 'restart' it, let the timer fire

Step-by-Step Diagnosis

1. List failed systemd units.

   systemctl --failed --no-pager

2. Tail the journal for nvidia-container-toolkit and the system bus.

   sudo journalctl -xe -f --no-pager

3. Inspect firewall / security-group posture from inside the instance.

   sudo firewall-cmd --list-all-zones --permanent 2>/dev/null || true
   sudo nft list ruleset 2>/dev/null | head -50
   ss -tulpen | head

4. Surface SELinux denials and author a local policy module if needed.

   sudo ausearch -m AVC,USER_AVC -ts today
   sudo ausearch -m AVC -ts today | audit2allow -a -M /tmp/local-fix
   sudo semodule -i /tmp/local-fix.pp

5. Verify nvidia-container-toolkit integrity and reinstall if anything is altered.

   sudo rpm -V nvidia-container-toolkit
   sudo dnf reinstall -y nvidia-container-toolkit

6. Correlate findings with /var/log/dnf.log, dnf history, Amazon Inspector findings, and Amazon Linux advisory ALAS2023NVIDIA-2025-125 to pin the change that introduced the regression.

Solution – Primary Fix

Apply the corrective dnf transaction referenced by Amazon Linux advisory ALAS2023NVIDIA-2025-125, then reload affected systemd units:

sudo dnf -y makecache
sudo dnf -y update --security              # apply ALL security errata (recommended)
# Or target a single package:
sudo dnf -y update nvidia-container-toolkit
sudo systemctl daemon-reload
# Unit name may differ from pkg name; check first:
readelf -p .comment /path/to/binary 2>/dev/null | grep -o 'GCC:.*'  # ELF .comment records the exact compiler that produced the file (package: binutils); absent if the build stripped it
annocheck /path/to/binary  # RHEL-native: reports the build-time compiler and hardening flags (package: annobin-annocheck on RHEL/Alma/Rocky 9-10, annobin on RHEL 8)
go version -m /path/to/binary  # prints the Go toolchain and module versions baked into a Go binary -- rebuild anything below the patched go version
rpm -q nvidia-container-toolkit                                # confirm new NVR

For kernel / glibc / systemd / openssl advisories a reboot is required (or Live Patching where available):

sudo needs-restarting -r                    # report whether reboot needed
sudo systemctl reboot                       # or: sudo shutdown -r now
# Amazon Linux Live Patching for the kernel (when enabled on the instance):
sudo dnf install -y kernel-livepatch
sudo dnf kernel-livepatch enable           # AL2 / AL2023
sudo dnf kernel-livepatch status

Roll the same change across an Auto Scaling group / fleet with AWS Systems Manager Patch Manager:

aws ssm send-command 
  --document-name AWS-RunPatchBaseline 
  --targets Key=tag:Patch,Values=yes 
  --parameters 'Operation=Install,RebootOption=RebootIfNeeded' 
  --comment 'Apply Amazon Linux security errata'
aws ssm list-command-invocations --details --max-results 5
# Confirm the patch landed across the fleet:
aws ssm describe-instance-patch-states-for-patch-group 
  --patch-group <patch-group>

For immutable infrastructure, rebuild the golden AMI in EC2 Image Builder so newly launched instances start patched (do not rely on in-place patching alone):

aws imagebuilder start-image-pipeline-execution 
  --image-pipeline-arn arn:aws:imagebuilder:<region>:<acct>:image-pipeline/<name>
aws imagebuilder list-images --owner Self --max-results 5
# Then update the launch template / ASG to the new AMI:
aws ec2 create-launch-template-version --launch-template-id <lt-id> 
  --source-version '$Latest' --launch-template-data 'ImageId=<new-ami-id>'

Solution – Alternative Approaches

If the primary patch is not viable, choose from these:

• Roll back the offending dnf transaction:

  sudo dnf history list | head
  sudo dnf history info <id>
  sudo dnf history undo <id>

• Version-lock the package so dnf cannot upgrade it:

  sudo dnf install -y python3-dnf-plugin-versionlock
  sudo dnf versionlock add nvidia-container-toolkit
  sudo dnf versionlock list
  sudo dnf versionlock delete nvidia-container-toolkit      # remove the lock

• Install an older NVR if a regression is suspected:

  dnf --showduplicates list nvidia-container-toolkit | tac | head
  sudo dnf install -y --allowerasing nvidia-container-toolkit-<older-NVR>

• Switch SELinux to permissive briefly to confirm policy is the cause, then re-enforce:

  sudo setenforce 0
  # reproduce, capture denials, author a custom module:
  sudo ausearch -m AVC -ts recent | audit2allow -a -M mylocal
  sudo semodule -i mylocal.pp
  sudo setenforce 1

• Take an EBS snapshot of the root volume before kernel / glibc upgrades for fast rollback:

  INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
  VOL_ID=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID 
    --query 'Reservations[].Instances[].BlockDeviceMappings[?DeviceName==`/dev/xvda`].Ebs.VolumeId' 
    --output text)
  aws ec2 create-snapshot --volume-id $VOL_ID 
    --description 'pre-patch-snapshot' --tag-specifications 
    'ResourceType=snapshot,Tags=[{Key=Name,Value=pre-patch}]'

• For immutable workloads, swap the ASG to a previous AMI version instead of patching in place:

  aws ec2 describe-launch-template-versions --launch-template-id <lt-id> --max-results 5
  aws autoscaling update-auto-scaling-group --auto-scaling-group-name <asg> 
    --launch-template LaunchTemplateId=<lt-id>,Version='<prev-version>'
  aws autoscaling start-instance-refresh --auto-scaling-group-name <asg>

• Where Kernel Live Patching is enabled, apply kernel fixes without reboot:

  sudo dnf kernel-livepatch status
  sudo dnf kernel-livepatch enable
  sudo dnf update -y --advisory=ALAS2023NVIDIA-2025-125 kernel

Verification & Acceptance Criteria

All of these should pass after the fix:

rpm -q nvidia-container-toolkit                                            # expected fixed NVR
sudo dnf updateinfo list cves --installed              # CVEs above no longer listed
sudo firewall-cmd --list-services 2>/dev/null || sudo iptables -L -n
getenforce
sudo needs-restarting -r
# Inspector should drop the finding within ~24h:
aws inspector2 list-findings 
  --filter-criteria 'vulnerabilityId={comparison=EQUALS,value=CVE-2025-23266}'

The conditions described in the advisory must no longer be reported for nvidia-container-toolkit across two consecutive runs.

Rollback Plan

Capture state before any change:

rpm -qa > /root/rpm-pre.txt
sudo dnf history list > /root/dnf-history-pre.txt
# Optional pre-patch EBS snapshot of the root volume (run from inside the instance):
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 create-snapshot --volume-id <vol-id> --description 'pre-patch'

To revert if the patch is bad:

sudo dnf history undo <id>
# Or downgrade just the package:
sudo dnf install -y --allowerasing nvidia-container-toolkit-<older-NVR>
sudo systemctl daemon-reload
# Or replace the instance from a snapshot/older AMI via ASG instance refresh:
aws autoscaling start-instance-refresh --auto-scaling-group-name <asg>
# Custom SELinux policy cleanup:
sudo semodule -r mylocal

Prevention & Hardening

Reduce the chance of this recurring on Amazon Linux 2023:

• Enable automatic security patching on each instance:

  sudo dnf install -y dnf-automatic
  sudo sed -i 's/^update_cmd.*/update_cmd = security/' /etc/dnf-automatic/dnf-automatic.conf 2>/dev/null || true
  sudo sed -i 's/^upgrade_type.*/upgrade_type = security/' /etc/dnf/automatic.conf 2>/dev/null || true
  sudo sed -i 's/^apply_updates.*/apply_updates = yes/' /etc/dnf/automatic.conf 2>/dev/null || true
  sudo systemctl enable --now dnf-automatic.timer

• Drive fleet-wide patching through AWS Systems Manager Patch Manager (preferred for any fleet bigger than a handful of instances):

  aws ssm send-command --document-name AWS-RunPatchBaseline 
    --targets Key=tag:Patch,Values=yes 
    --parameters 'Operation=Install,RebootOption=RebootIfNeeded'
  aws ssm describe-patch-baselines --filters Key=OWNER,Values=AWS
  aws ssm get-default-patch-baseline --operating-system AMAZON_LINUX_2023

• Enable Amazon Inspector for continuous CVE / package vulnerability scanning:

  aws inspector2 enable --resource-types EC2 ECR LAMBDA
  aws inspector2 list-findings 
    --filter-criteria 'severity={comparison=EQUALS,value=HIGH}' --max-results 20
  aws inspector2 batch-get-account-status

• Bake patched golden AMIs with EC2 Image Builder and roll them via ASG instance refresh instead of in-place patching for immutable infrastructure:

  aws imagebuilder list-image-pipelines
  aws imagebuilder start-image-pipeline-execution 
    --image-pipeline-arn arn:aws:imagebuilder:<region>:<acct>:image-pipeline/<name>
  aws autoscaling start-instance-refresh --auto-scaling-group-name <asg>

• Subscribe to alas.aws.amazon.com and watch AWS security bulletins for upstream changes.

• Version-lock sensitive packages so they cannot be auto-upgraded:

  sudo dnf install -y python3-dnf-plugin-versionlock
  sudo dnf versionlock add nvidia-container-toolkit

• Monitor file integrity with AIDE:

  sudo dnf install -y aide
  sudo aide --init && sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
  sudo aide --check

• Enable Kernel Live Patching so kernel CVEs can be remediated without reboot:

  sudo dnf install -y kernel-livepatch
  sudo dnf kernel-livepatch enable
  sudo dnf kernel-livepatch status

• Keep SELinux in enforcing mode and review custom modules in /etc/selinux/targeted/ after every package upgrade.

• Apply CIS Amazon Linux 2023 Benchmark hardening and remove unused packages.

Related Errors & Cross-Refs

Issues that commonly surface alongside a nvidia-container-toolkit update: dnf lock contention, systemd unit ordering cycles, SELinux AVC bursts, security-group / NACL drift, and kernel taint flags after out-of-tree modules. Useful triage:

sudo dnf check
systemd-analyze critical-chain
sudo ausearch -m AVC -ts today | tail
sudo firewall-cmd --get-active-zones 2>/dev/null || sudo iptables -L -n
cat /proc/sys/kernel/tainted
sudo needs-restarting -r
aws ssm describe-instance-patches --instance-id <i-xxxx> | tail -40

View all amazon-linux-2023 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

Primary reference: Amazon Linux advisory ALAS2023NVIDIA-2025-125. Manual pages useful on Amazon Linux 2023:

man dnf
man dnf.conf
man systemctl
man journalctl
man firewall-cmd
man semanage
man audit2allow
man sosreport

Other resources: alas.aws.amazon.com, SSM Patch Manager docs, Amazon Inspector docs, EC2 Image Builder docs, and per-package notes in /usr/share/doc/nvidia-container-toolkit/ for components implicated in this advisory.

View all Amazon Linux 2023 tutorials on the Tutorials Hub →