ModSecurity as a dynamic Nginx module extends Nginx with WAF capabilities, blocking SQL injection, XSS, command injection, and hundreds of other attack patterns defined by the OWASP Core Rule Set. Unlike the Apache version which uses the stable mod_security2, the Nginx connector uses the newer libmodsecurity3 C++ library with a ngx_http_modsecurity_module connector. On RHEL 9, the combination must be compiled from source or installed via a third-party repository since the RHEL AppStream does not include the Nginx ModSecurity module. This guide covers the complete installation of ModSecurity 3 with Nginx on RHEL 9 using the prebuilt packages from the official ModSecurity repository, configuring the OWASP Core Rule Set, and tuning for production.

Prerequisites

  • Nginx installed on RHEL 9
  • Development tools for compilation if using source build: dnf groupinstall "Development Tools"

Step 1 — Install ModSecurity 3 Library

# Install EPEL and required dependencies
dnf install -y epel-release
dnf install -y gcc make git libxml2-devel pcre2-devel curl-devel 
    yajl-devel GeoIP-devel lmdb-devel ssdeep-devel libmaxminddb-devel

# Clone and build libmodsecurity3
cd /usr/local/src
git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
cd ModSecurity
git submodule init
git submodule update

./build.sh
./configure
make -j$(nproc)
make install

Step 2 — Install the Nginx ModSecurity Connector

# Get your Nginx version
nginx -v 2>&1 | grep -oP '[0-9]+.[0-9]+.[0-9]+'
NGINX_VER=$(nginx -v 2>&1 | grep -oP '[0-9]+.[0-9]+.[0-9]+')

# Clone the Nginx ModSecurity connector
cd /usr/local/src
git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx

# Download matching Nginx source
curl -O https://nginx.org/download/nginx-${NGINX_VER}.tar.gz
tar xzf nginx-${NGINX_VER}.tar.gz
cd nginx-${NGINX_VER}

# Get the Nginx configure arguments from the running binary
nginx -V 2>&1 | grep "configure arguments"

# Build the dynamic module only (add --with-compat to existing args)
./configure --with-compat --add-dynamic-module=/usr/local/src/ModSecurity-nginx
make modules

# Install the module
cp objs/ngx_http_modsecurity_module.so /usr/lib64/nginx/modules/

Step 3 — Load the Module in Nginx

# Add to the top of /etc/nginx/nginx.conf
load_module modules/ngx_http_modsecurity_module.so;

Step 4 — Download OWASP Core Rule Set

mkdir -p /etc/nginx/modsecurity
cd /etc/nginx/modsecurity

# Get the OWASP CRS
git clone https://github.com/coreruleset/coreruleset
cp coreruleset/crs-setup.conf.example coreruleset/crs-setup.conf

# Create main modsecurity config
cp /usr/local/src/ModSecurity/modsecurity.conf-recommended modsecurity.conf

# Set to detection mode first
sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine DetectionOnly/' modsecurity.conf
vi /etc/nginx/modsecurity/main.conf
Include /etc/nginx/modsecurity/modsecurity.conf
Include /etc/nginx/modsecurity/coreruleset/crs-setup.conf
Include /etc/nginx/modsecurity/coreruleset/rules/*.conf

Step 5 — Enable ModSecurity in Nginx Server Block

vi /etc/nginx/conf.d/example.com.conf
server {
    listen 80;
    server_name example.com;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsecurity/main.conf;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
nginx -t && systemctl reload nginx

Step 6 — Test and Move to Enforcement

# Test detection
curl "http://example.com/?q=alert(1)"

# Check audit log
tail -f /var/log/modsec_audit.log

# After tuning false positives, enable enforcement
sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' 
    /etc/nginx/modsecurity/modsecurity.conf
nginx -t && systemctl reload nginx

Conclusion

ModSecurity 3 with Nginx on RHEL 9 and the OWASP CRS provides comprehensive WAF protection. The DetectionOnly → tuning → enforcement workflow ensures you do not block legitimate traffic. Keep the CRS updated with git pull in the coreruleset/ directory and reload Nginx after updates.

Next steps: How to Configure Nginx Rate Limiting and Connection Throttling on RHEL 9, How to Enable Brotli and Gzip Compression in Nginx on RHEL 9, and How to Harden Web Servers on RHEL 9.