How to Use Prometheus Blackbox Exporter for Endpoint Monitoring on RHEL 7

How to Use Prometheus Blackbox Exporter for Endpoint Monitoring on RHEL 7

The Prometheus Blackbox Exporter probes external endpoints over HTTP, HTTPS, TCP, ICMP, and DNS from the perspective of an outside caller, making it an essential complement to node-level metrics collected by Node Exporter. While Node Exporter tells you about CPU load and memory on your server, Blackbox Exporter answers the question: “Can users actually reach my services?” This guide covers a complete deployment of Blackbox Exporter on Red Hat Enterprise Linux 7, including binary installation, systemd service configuration, Prometheus scrape job setup with relabeling, alert rules for endpoint outages and slow responses, and an example probe URL list.

Prerequisites

• RHEL 7 server with a running Prometheus instance (v2.x)
• A non-root user with sudo privileges
• Internet access or a local mirror to download the Blackbox Exporter binary
• Firewall access to port 9115 (Blackbox Exporter default)
• Basic familiarity with YAML and Prometheus configuration

Step 1: Download the Blackbox Exporter Binary

Download the latest stable release of Blackbox Exporter from the official Prometheus GitHub releases page. At the time of writing, version 0.24.0 is the current stable release. Adjust the version number as needed.

# Create a directory for the download
cd /tmp

# Download the Blackbox Exporter tarball (adjust version as needed)
curl -LO https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz

# Verify the checksum (compare with the sha256sums file on the release page)
sha256sum blackbox_exporter-0.24.0.linux-amd64.tar.gz

# Extract the archive
tar xzf blackbox_exporter-0.24.0.linux-amd64.tar.gz

# Move the binary to /usr/local/bin
sudo mv blackbox_exporter-0.24.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/blackbox_exporter

# Verify the binary runs
blackbox_exporter --version

Step 2: Create a Dedicated System User

Always run exporters under a non-privileged system account to limit the blast radius if the process is compromised. ICMP probing requires the binary to have the CAP_NET_RAW capability — granted below without running as root.

# Create the system user
sudo useradd -r -s /sbin/nologin -d /var/lib/blackbox_exporter blackbox_exporter

# Create the configuration directory
sudo mkdir -p /etc/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter

# Grant CAP_NET_RAW so ICMP probing works without root
sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter

Step 3: Create the Blackbox Exporter Configuration File

The configuration file defines modules — named probe configurations that Prometheus will reference in its scrape jobs. Create /etc/blackbox_exporter/blackbox.yml with modules for HTTP 2xx success checks, ICMP pings, and TCP connection probes.

sudo vi /etc/blackbox_exporter/blackbox.yml

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []          # defaults to 2xx
      method: GET
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: false
      preferred_ip_protocol: "ip4"

  http_2xx_insecure:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: []
      method: GET
      follow_redirects: true
      tls_config:
        insecure_skip_verify: true    # for self-signed certs in dev

  icmp:
    prober: icmp
    timeout: 10s
    icmp:
      preferred_ip_protocol: "ip4"

  tcp_connect:
    prober: tcp
    timeout: 10s
    tcp:
      preferred_ip_protocol: "ip4"

  dns_lookup:
    prober: dns
    timeout: 10s
    dns:
      transport_protocol: "udp"
      preferred_ip_protocol: "ip4"
      query_name: "example.com"
      query_type: "A"

# Set correct ownership
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml

# Validate the configuration file syntax
blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml --dry-run 2>&1 || echo "Config OK (no --dry-run flag; check logs)"

Step 4: Create the systemd Service Unit

Create a systemd service file so Blackbox Exporter starts automatically on boot and restarts on failure.

sudo vi /etc/systemd/system/blackbox_exporter.service

[Unit]
Description=Prometheus Blackbox Exporter
Documentation=https://github.com/prometheus/blackbox_exporter
After=network.target

[Service]
Type=simple
User=blackbox_exporter
Group=blackbox_exporter
ExecStart=/usr/local/bin/blackbox_exporter 
    --config.file=/etc/blackbox_exporter/blackbox.yml 
    --web.listen-address=:9115 
    --log.level=info
Restart=on-failure
RestartSec=5s
NoNewPrivileges=yes
ProtectHome=yes
ProtectSystem=full
ReadWritePaths=/var/lib/blackbox_exporter

[Install]
WantedBy=multi-user.target

# Reload systemd and start the service
sudo systemctl daemon-reload
sudo systemctl enable blackbox_exporter
sudo systemctl start blackbox_exporter

# Check status
sudo systemctl status blackbox_exporter

# Confirm it is listening on port 9115
ss -tlnp | grep 9115

Step 5: Open the Firewall Port

# Allow Prometheus to scrape Blackbox Exporter on port 9115
sudo firewall-cmd --zone=public --add-port=9115/tcp --permanent
sudo firewall-cmd --reload

# Quick sanity check: probe an HTTP endpoint manually via curl
curl "http://localhost:9115/probe?target=https://example.com&module=http_2xx"

The response is a Prometheus text-format metrics page. Look for probe_success 1 to confirm the probe succeeded.

Step 6: Configure Prometheus Scrape Jobs with Relabeling

The standard approach for Blackbox Exporter is to maintain a list of target URLs in a file (or inline) and use Prometheus relabeling to pass the target as a URL parameter to the exporter. Add the following jobs to your /etc/prometheus/prometheus.yml.

scrape_configs:

  # HTTP endpoint probes
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
          - https://shop.example.com
          - https://docs.example.com
    relabel_configs:
      # Move the target URL into the 'target' query parameter
      - source_labels: [__address__]
        target_label: __param_target
      # Set the instance label to the original target URL for readability
      - source_labels: [__param_target]
        target_label: instance
      # Replace the scrape address with the Blackbox Exporter address
      - target_label: __address__
        replacement: 127.0.0.1:9115

  # ICMP / ping probes
  - job_name: 'blackbox_icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 192.168.1.1
          - 192.168.1.10
          - 10.0.0.1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

  # TCP port probes
  - job_name: 'blackbox_tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - db-server.example.com:5432
          - smtp.example.com:25
          - redis.example.com:6379
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

# Validate the Prometheus configuration
promtool check config /etc/prometheus/prometheus.yml

# Reload Prometheus (no restart needed)
sudo systemctl reload prometheus

Step 7: Write Alert Rules for Endpoint Down and Slow Response

Create a dedicated alert rules file for Blackbox Exporter metrics. The two most important alerts are probe_success equalling zero (endpoint is down) and probe_duration_seconds exceeding a threshold (endpoint is slow).

sudo vi /etc/prometheus/rules/blackbox.yml

groups:
  - name: blackbox_exporter
    interval: 60s
    rules:

      # Alert when any probed endpoint is unreachable
      - alert: EndpointDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Endpoint {{ $labels.instance }} is DOWN"
          description: >
            The probe for {{ $labels.instance }} (job={{ $labels.job }})
            has been failing for more than 2 minutes.
            probe_success = {{ $value }}

      # Alert when HTTP probe latency exceeds 3 seconds
      - alert: EndpointSlowResponse
        expr: probe_duration_seconds{job="blackbox_http"} > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Slow response from {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} probe duration is {{ $value | humanizeDuration }},
            which exceeds the 3-second threshold.

      # Alert when SSL certificate expires within 14 days
      - alert: SSLCertificateExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() 
            The SSL certificate for {{ $labels.instance }} expires in
            {{ $value | humanizeDuration }}.

# Reference the rules file in prometheus.yml
# Add under the rule_files: section:
#   - /etc/prometheus/rules/blackbox.yml

# Validate rule syntax
promtool check rules /etc/prometheus/rules/blackbox.yml

# Reload Prometheus
sudo systemctl reload prometheus

Step 8: Verify Probes in the Prometheus UI

Open the Prometheus web UI (typically http://your-server:9090) and run these queries to confirm Blackbox data is being collected correctly.

# All endpoints and their current probe success status
probe_success

# Average HTTP probe duration per instance over the last 5 minutes
avg_over_time(probe_duration_seconds[5m])

# SSL certificate expiry time in days
(probe_ssl_earliest_cert_expiry - time()) / 86400

# All currently DOWN endpoints
probe_success == 0

Conclusion

You have deployed the Prometheus Blackbox Exporter on RHEL 7 with a systemd service, configured HTTP, ICMP, and TCP probe modules in blackbox.yml, and integrated them into Prometheus using relabeling so each target URL maps cleanly to a scrape parameter. Alert rules now fire when an endpoint is unreachable, responding too slowly, or serving a certificate that is about to expire. This external probing approach complements server-side exporters by simulating real user connectivity, giving you full visibility into both the health of your infrastructure and the experience of the users who depend on it.