How to Configure Prometheus AlertManager on RHEL 9

Prometheus AlertManager is a standalone component of the Prometheus monitoring stack responsible for deduplicating, grouping, routing, and dispatching alerts to notification channels such as Slack, email, and PagerDuty. While Prometheus evaluates alerting rules and fires alerts, it is AlertManager that decides who gets notified, when, and how — preventing alert storms and on-call fatigue through intelligent grouping and inhibition logic. Running AlertManager as a dedicated systemd service on RHEL 9 ensures it restarts automatically on failure and integrates cleanly with your existing monitoring infrastructure. This guide covers installing AlertManager from binary, writing a production-ready configuration, and connecting it to a running Prometheus instance.

Prerequisites

• RHEL 9 server with sudo/root access
• Prometheus already installed and running (AlertManager is an add-on, not a replacement)
• A Slack incoming webhook URL, or SMTP credentials, or a PagerDuty integration key for notifications
• Port 9093 open if AlertManager needs to be reachable from other hosts

Step 1 — Download and Install AlertManager

# Find the latest release at https://github.com/prometheus/alertmanager/releases
export AM_VERSION=0.27.0

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz

# Extract and install the binaries
tar xzf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
cd alertmanager-${AM_VERSION}.linux-amd64

cp alertmanager /usr/local/bin/
cp amtool       /usr/local/bin/
chmod +x /usr/local/bin/alertmanager /usr/local/bin/amtool

# Create the AlertManager user and directories
useradd --no-create-home --shell /bin/false alertmanager
mkdir -p /etc/alertmanager /var/lib/alertmanager
chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Step 2 — Write the alertmanager.yml Configuration

# /etc/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  # Global SMTP settings (used by all email receivers unless overridden)
  smtp_smarthost: smtp.example.com:587
  smtp_from: [email protected]
  smtp_auth_username: [email protected]
  smtp_auth_password: YourSMTPPassword

route:
  # Group alerts by alertname and cluster to reduce noise
  group_by: ['alertname', 'cluster', 'service']
  group_wait:      30s   # wait 30s before sending first alert in a group
  group_interval:  5m    # wait 5m before sending updates on a group
  repeat_interval: 3h    # re-notify after 3h if still firing
  receiver: slack-critical

  routes:
    # Route database alerts to the DBA team
    - match:
        team: database
      receiver: email-dba

    # Route low-severity alerts to Slack only
    - match:
        severity: warning
      receiver: slack-warnings

    # Route critical production alerts to PagerDuty
    - match:
        severity: critical
      receiver: pagerduty-oncall

inhibit_rules:
  # Suppress warning alerts when a critical alert is already firing
  # for the same alertname + instance
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'instance']

receivers:
  - name: slack-critical
    slack_configs:
      - api_url: https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxx
        channel: '#alerts-critical'
        send_resolved: true
        title: '{{ template "slack.default.title" . }}'
        text:  '{{ template "slack.default.text" . }}'

  - name: slack-warnings
    slack_configs:
      - api_url: https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxx
        channel: '#alerts-warnings'
        send_resolved: true

  - name: email-dba
    email_configs:
      - to: [email protected]
        send_resolved: true

  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key: YOUR_PAGERDUTY_INTEGRATION_KEY
        send_resolved: true

Step 3 — Create and Enable the systemd Service

# /etc/systemd/system/alertmanager.service

[Unit]
Description=Prometheus AlertManager
After=network-online.target
Wants=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager 
  --web.listen-address=:9093 
  --cluster.advertise-address=0.0.0.0:9094
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

# Write the unit file and start the service
cat > /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Prometheus AlertManager
After=network-online.target
Wants=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager 
  --web.listen-address=:9093 
  --cluster.advertise-address=0.0.0.0:9094
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now alertmanager

# Check status and open the firewall port
systemctl status alertmanager
firewall-cmd --permanent --add-port=9093/tcp
firewall-cmd --reload

Step 4 — Connect AlertManager to Prometheus

# Add the alerting block to /etc/prometheus/prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

# Also load alerting rules from a separate file
rule_files:
  - /etc/prometheus/rules/*.yml

# Reload Prometheus to apply the change
systemctl reload prometheus
# or: curl -X POST http://localhost:9090/-/reload

# Example alert rule — save to /etc/prometheus/rules/node.yml

groups:
  - name: node-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf "%.1f" }}% for more than 5 minutes."

Step 5 — Test Alerts with amtool

# Configure amtool to know where AlertManager is
cat > ~/.config/amtool/config.yml << 'EOF'
alertmanager.url: http://localhost:9093
EOF

# Check the current AlertManager config is valid
amtool check-config /etc/alertmanager/alertmanager.yml

# List currently active alerts
amtool alert

# Send a test alert manually
amtool alert add alertname="TestAlert" severity="warning" instance="test-host" 
  --annotation=summary="This is a test alert"

# Create a silence to suppress a known maintenance window
amtool silence add alertname="HighCPUUsage" instance="db01.example.com" 
  --duration=2h 
  --comment="Planned maintenance window"

# List active silences
amtool silence query

Conclusion

AlertManager transforms a noisy stream of Prometheus alerts into actionable, deduplicated notifications delivered to the right person at the right time. Grouping related alerts reduces pager fatigue, inhibition rules prevent redundant notifications when a root-cause alert is already firing, and silences allow planned maintenance windows without disabling alerting globally. Running AlertManager as a systemd service ensures high availability and clean log management via journald. For production environments, deploy AlertManager in a high-availability cluster mode by running three instances and configuring each with the same --cluster.peer flags, and store sensitive credentials such as Slack webhook URLs and SMTP passwords in a secrets manager rather than in plain-text YAML.

Next steps: How to Install Prometheus on RHEL 9, How to Install and Configure Grafana on RHEL 9, and How to Set Up a Local Container Registry with Harbor on RHEL 9.