How to Configure Nginx with ModSecurity WAF on RHEL 7

January 1, 2026
Linux
Comment off

How to Configure Nginx with ModSecurity WAF on RHEL 7

While ModSecurity is most commonly associated with Apache, it also runs as a dynamic module for Nginx through the ModSecurity-nginx connector. On RHEL 7, the official Nginx packages do not ship with ModSecurity built in, so you have two practical paths: compile Nginx and ModSecurity from source with the connector, or use a pre-built package from a third-party repository such as the nginx-mod-modsecurity package provided by some community repos. This tutorial covers the source-compilation approach, which gives you full control over version selection and module configuration, followed by integration with the OWASP CRS and practical testing and whitelisting workflows.

Prerequisites

RHEL 7 server with sudo access and at least 2 GB of free disk space for compilation
Development tools: sudo yum groupinstall -y "Development Tools"
EPEL repository enabled: sudo yum install -y epel-release
Required libraries: sudo yum install -y pcre-devel zlib-devel openssl-devel libxml2-devel curl-devel yajl-devel lmdb-devel ssdeep-devel lua-devel
Git installed: sudo yum install -y git
If Nginx is already installed from a repo, note its version with nginx -v — you must compile to the same version

Step 1: Install Build Dependencies

sudo yum install -y epel-release
sudo yum install -y 
    gcc gcc-c++ make automake autoconf libtool 
    pcre pcre-devel 
    zlib zlib-devel 
    openssl openssl-devel 
    libxml2 libxml2-devel 
    curl libcurl-devel 
    yajl yajl-devel 
    lmdb lmdb-devel 
    ssdeep ssdeep-devel 
    lua lua-devel 
    geoip geoip-devel 
    git wget

Step 2: Clone and Build libModSecurity (ModSecurity v3)

ModSecurity v3 (libmodsecurity) is the standalone C++ library that Nginx uses via the connector. It is separate from the mod_security2.so Apache module.

cd /usr/local/src

# Clone the ModSecurity source
sudo git clone --depth 1 -b v3/master 
    https://github.com/SpiderLabs/ModSecurity /usr/local/src/ModSecurity

cd /usr/local/src/ModSecurity
sudo git submodule init
sudo git submodule update

# Run the build scripts
sudo ./build.sh
sudo ./configure
sudo make -j$(nproc)
sudo make install

# The library is installed to /usr/local/modsecurity/
ls /usr/local/modsecurity/lib/

This step takes 5–10 minutes depending on your CPU. The output library libmodsecurity.so.3 is installed under /usr/local/modsecurity/lib/.

Step 3: Clone the ModSecurity-Nginx Connector

sudo git clone --depth 1 
    https://github.com/SpiderLabs/ModSecurity-nginx /usr/local/src/ModSecurity-nginx

Step 4: Download Nginx Source and Compile with the Connector

Determine the Nginx version you need, then download matching source. If you are adding ModSecurity to an existing Nginx installation, you must use the exact same version as what is currently installed.

# Get the Nginx version currently installed (or choose latest stable)
NGINX_VERSION="1.24.0"

cd /usr/local/src
sudo wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz"
sudo tar -xzf nginx-${NGINX_VERSION}.tar.gz
cd nginx-${NGINX_VERSION}

# Configure Nginx with the ModSecurity connector as a dynamic module
# --add-dynamic-module builds a .so that can be loaded with load_module
sudo ./configure 
    --prefix=/etc/nginx 
    --sbin-path=/usr/sbin/nginx 
    --modules-path=/usr/lib64/nginx/modules 
    --conf-path=/etc/nginx/nginx.conf 
    --error-log-path=/var/log/nginx/error.log 
    --http-log-path=/var/log/nginx/access.log 
    --pid-path=/var/run/nginx.pid 
    --with-http_ssl_module 
    --with-http_v2_module 
    --with-http_gzip_static_module 
    --with-http_stub_status_module 
    --with-pcre 
    --add-dynamic-module=/usr/local/src/ModSecurity-nginx

sudo make -j$(nproc)
sudo make install

After compilation, the dynamic module will be available at /usr/lib64/nginx/modules/ngx_http_modsecurity_module.so.

Step 5: Configure ModSecurity for Nginx

Copy the recommended configuration template from the libModSecurity source and create the ModSecurity config directory:

sudo mkdir -p /etc/nginx/modsecurity

# Copy the recommended base configuration
sudo cp /usr/local/src/ModSecurity/modsecurity.conf-recommended 
        /etc/nginx/modsecurity/modsecurity.conf

# Copy the unicode mapping table
sudo cp /usr/local/src/ModSecurity/unicode.mapping 
        /etc/nginx/modsecurity/unicode.mapping

Edit the base configuration to enable the rule engine:

sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' 
    /etc/nginx/modsecurity/modsecurity.conf

# Also enable response body inspection
sudo sed -i 's/SecResponseBodyAccess Off/SecResponseBodyAccess On/' 
    /etc/nginx/modsecurity/modsecurity.conf

# Set the audit log path
sudo sed -i 's|SecAuditLog /var/log/modsec_audit.log|SecAuditLog /var/log/nginx/modsec_audit.log|' 
    /etc/nginx/modsecurity/modsecurity.conf

Step 6: Download and Configure the OWASP CRS

CRS_VERSION="3.3.5"
curl -L "https://github.com/coreruleset/coreruleset/archive/v${CRS_VERSION}.tar.gz" 
     -o /tmp/crs.tar.gz

sudo tar -xzf /tmp/crs.tar.gz -C /etc/nginx/modsecurity/

sudo mv /etc/nginx/modsecurity/coreruleset-${CRS_VERSION} 
        /etc/nginx/modsecurity/owasp-crs

sudo cp /etc/nginx/modsecurity/owasp-crs/crs-setup.conf.example 
        /etc/nginx/modsecurity/owasp-crs/crs-setup.conf

# Create a main rules include file
sudo tee /etc/nginx/modsecurity/main.conf << 'EOF'
Include /etc/nginx/modsecurity/modsecurity.conf
Include /etc/nginx/modsecurity/owasp-crs/crs-setup.conf
Include /etc/nginx/modsecurity/owasp-crs/rules/*.conf
EOF

Step 7: Load ModSecurity in nginx.conf

Edit /etc/nginx/nginx.conf to load the dynamic module and enable ModSecurity for your server block:

# At the very top of nginx.conf, before the 'events' block:
load_module modules/ngx_http_modsecurity_module.so;

# Inside the http block:
http {
    # ... other directives ...

    server {
        listen 80;
        server_name example.com;

        # Enable ModSecurity for this server block
        modsecurity on;
        modsecurity_rules_file /etc/nginx/modsecurity/main.conf;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }

        # Access and error logs
        access_log /var/log/nginx/access.log;
        error_log  /var/log/nginx/error.log warn;
    }
}

Test the configuration and restart Nginx:

sudo nginx -t
# Expected: nginx: configuration file /etc/nginx/nginx.conf test is successful

sudo systemctl restart nginx
sudo systemctl status nginx

Step 8: Test WAF Rules with Simulated Attacks

# SQL injection test
curl -v "http://localhost/?id=1%27%20OR%20%271%27%3D%271"
# Expected: HTTP/1.1 403 Forbidden

# XSS test
curl -v "http://localhost/?name=<script>alert(document.cookie)</script>"
# Expected: HTTP/1.1 403 Forbidden

# Path traversal test
curl -v "http://localhost/?file=../../../../etc/passwd"
# Expected: HTTP/1.1 403 Forbidden

# Check audit log for the blocked requests
sudo tail -100 /var/log/nginx/modsec_audit.log | grep "id|msg|severity"

Step 9: Configuring Whitelist Exceptions

Create an exceptions file to suppress rules that trigger false positives for your specific application:

sudo tee /etc/nginx/modsecurity/exclusions.conf << 'EOF'
# Whitelist by rule ID — disables the rule completely
SecRuleRemoveById 920350

# Whitelist by rule ID for a specific URI
SecRule REQUEST_URI "@beginsWith /api/v1/content" 
    "id:9001, 
     phase:1, 
     pass, 
     nolog, 
     ctl:ruleRemoveById=941100"

# Whitelist a specific IP address from all inspection
SecRule REMOTE_ADDR "@ipMatch 192.168.1.50" 
    "id:9002, 
     phase:1, 
     allow, 
     log, 
     msg:'Whitelisted internal IP'"
EOF

Add the include to /etc/nginx/modsecurity/main.conf before the CRS rules, then reload Nginx.

Step 10: Logging and Monitoring

# Watch the ModSecurity audit log in real time
sudo tail -f /var/log/nginx/modsec_audit.log

# Count blocked requests by rule ID
sudo grep "id "" /var/log/nginx/modsec_audit.log | 
    grep -oP 'id "K[0-9]+' | 
    sort | uniq -c | sort -rn | head -20

# Check Nginx error log for ModSecurity messages
sudo grep -i modsecurity /var/log/nginx/error.log | tail -20

Running ModSecurity with Nginx on RHEL 7 requires more initial effort than the Apache integration due to the need to compile from source, but the end result is a high-performance, standards-compliant WAF that inspects every request and response flowing through your Nginx server. Pairing it with the OWASP CRS gives you immediate protection against the most common web attack categories. The key to a stable production deployment is a careful tuning phase: deploy in DetectionOnly mode first, identify false positives in the audit log, write targeted exclusion rules, and then switch to SecRuleEngine On for active blocking.

How to Configure Nginx with ModSecurity WAF on RHEL 7