ModSecurity is not limited to Apache — it can also protect Nginx deployments through a dedicated dynamic module called ngx_http_modsecurity_module. The module embeds the ModSecurity v3 library directly into Nginx’s request processing pipeline, enabling the same OWASP Core Rule Set that Apache users rely on. On RHEL 8, Nginx is available from the AppStream module stream, and the ModSecurity Nginx connector can be compiled against it or sourced from a third-party package. This tutorial walks through installing Nginx, building or enabling the ModSecurity module, integrating the OWASP CRS, and switching between detection and prevention modes.

Prerequisites

  • RHEL 8 server with root or sudo access
  • Basic familiarity with Nginx configuration syntax
  • Development tools (gcc, make, pcre-devel, openssl-devel) if compiling from source
  • Internet access to download the ModSecurity source or a third-party repository
  • Ports 80 and 443 open in firewalld

Step 1 — Install Nginx from AppStream

RHEL 8 AppStream provides Nginx as a module stream. Install the latest stable stream.

sudo dnf module enable nginx:mainline -y
sudo dnf install -y nginx
sudo systemctl enable --now nginx
sudo systemctl status nginx

Confirm Nginx is running before proceeding. Note the exact Nginx version — you will need it when building the ModSecurity connector.

nginx -v

Step 2 — Install ModSecurity v3 and the Nginx Connector

Install the required build dependencies and the ModSecurity v3 library, then clone and compile the Nginx connector. If a pre-built package is available for your environment, use it to avoid the build step.

sudo dnf install -y gcc make git pcre-devel openssl-devel 
    libxml2-devel curl-devel geoip-devel lmdb-devel 
    libmaxminddb-devel yajl-devel

# Install ModSecurity v3 library
sudo dnf install -y libmodsecurity3 libmodsecurity3-devel

# Clone and build the Nginx connector
cd /usr/local/src
sudo git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git

# Download the same Nginx version source to compile the dynamic module
NGINX_VER=$(nginx -v 2>&1 | grep -oP '[d.]+')
sudo curl -O http://nginx.org/download/nginx-${NGINX_VER}.tar.gz
sudo tar xzf nginx-${NGINX_VER}.tar.gz
cd nginx-${NGINX_VER}
sudo ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
sudo make modules
sudo cp objs/ngx_http_modsecurity_module.so /usr/lib64/nginx/modules/

Step 3 — Load the Module in nginx.conf

Add the load_module directive at the top of /etc/nginx/nginx.conf (outside any block context), then add the ModSecurity directives to the relevant server or location block.

sudo nano /etc/nginx/nginx.conf

At the very top of the file, before any events or http block:

load_module modules/ngx_http_modsecurity_module.so;

Inside the http block, add the path to the ModSecurity configuration:

modsecurity on;
modsecurity_rules_file /etc/nginx/modsecurity/main.conf;

Step 4 — Configure ModSecurity with OWASP CRS

Create the ModSecurity configuration directory, copy the recommended base configuration, and install the OWASP CRS.

sudo mkdir -p /etc/nginx/modsecurity

# Copy the base ModSecurity configuration
sudo cp /usr/share/doc/libmodsecurity3/modsecurity.conf-recommended 
    /etc/nginx/modsecurity/modsecurity.conf

# Start in detection mode initially
sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine DetectionOnly/' 
    /etc/nginx/modsecurity/modsecurity.conf

# Download the OWASP CRS
cd /etc/nginx/modsecurity
sudo git clone --depth 1 
    https://github.com/coreruleset/coreruleset.git owasp-crs
sudo cp owasp-crs/crs-setup.conf.example owasp-crs/crs-setup.conf

# Create the main.conf that includes everything
sudo tee /etc/nginx/modsecurity/main.conf <<'EOF'
Include /etc/nginx/modsecurity/modsecurity.conf
Include /etc/nginx/modsecurity/owasp-crs/crs-setup.conf
Include /etc/nginx/modsecurity/owasp-crs/rules/*.conf
EOF

Step 5 — Switch Between Detection and Prevention Mode

Detection mode logs attacks without blocking. Prevention mode blocks and returns HTTP 403. Edit modsecurity.conf to change mode.

# Detection only (log but do not block):
sudo sed -i 's/^SecRuleEngine.*/SecRuleEngine DetectionOnly/' 
    /etc/nginx/modsecurity/modsecurity.conf

# Prevention mode (block malicious requests):
sudo sed -i 's/^SecRuleEngine.*/SecRuleEngine On/' 
    /etc/nginx/modsecurity/modsecurity.conf

sudo nginx -t && sudo systemctl reload nginx

Test that blocked requests are handled correctly:

curl -I "http://localhost/?q=alert(1)"
# Expect: HTTP/1.1 403 Forbidden in prevention mode

sudo tail -f /var/log/nginx/error.log | grep -i modsec

Conclusion

You have installed Nginx on RHEL 8, compiled the ModSecurity v3 Nginx connector, loaded the module dynamically, integrated the OWASP Core Rule Set, and tested both detection and prevention modes. Beginning with detection mode is the recommended approach — it allows you to baseline normal traffic and tune false positives before enabling blocking. The error log provides actionable rule IDs that you can suppress using SecRuleRemoveById in a local exclusions file.

Next steps: How to Configure Nginx Rate Limiting and Connection Throttling on RHEL 8, How to Set Up ModSecurity WAF with Apache on RHEL 8, and How to Enable Brotli and Gzip Compression in Nginx on RHEL 8.