Prometheus Alertmanager handles routing, deduplication, and delivery of alerts fired by Prometheus to external notification channels. On RHEL 9, integrating Alertmanager with PagerDuty and Slack allows teams to receive critical pages and contextual chat notifications from a single alert pipeline. This tutorial walks through downloading the Alertmanager binary, configuring a complete route tree, setting up both receivers, and validating alerts using the amtool command-line utility. By the end you will have a production-ready Alertmanager instance routing critical alerts to PagerDuty and warning-level alerts to Slack with inhibition rules to prevent notification floods.

Prerequisites

  • RHEL 9 server with Prometheus already installed and running
  • PagerDuty account with a Prometheus integration key (Events API v2)
  • Slack workspace with an incoming webhook URL for your target channel
  • A non-root user with sudo privileges
  • Firewall access on port 9093 for the Alertmanager web UI

Step 1 — Download and Install Alertmanager

Download the latest Alertmanager release from the Prometheus project, extract it, and place the binaries on your system path.

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo mv alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Step 2 — Create the Alertmanager Configuration File

Create /etc/alertmanager/alertmanager.yml with a complete route tree, PagerDuty receiver for critical alerts, Slack receiver for warnings, and inhibition rules to suppress child alerts when a parent fires.

sudo tee /etc/alertmanager/alertmanager.yml > /dev/null <<'EOF'
global:
  resolve_timeout: 5m
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-warnings'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: false
    - match:
        severity: warning
      receiver: 'slack-warnings'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - integration_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY'
        severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}warning{{ end }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          resolved: '{{ .Alerts.Resolved | len }}'
          instance: '{{ .CommonLabels.instance }}'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#alerts'
        send_resolved: true
        title: '{{ if eq .Status "firing" }}:fire: FIRING{{ else }}:white_check_mark: RESOLVED{{ end }} — {{ .CommonLabels.alertname }}'
        text: |
          *Severity:* {{ .CommonLabels.severity | title }}
          *Summary:* {{ .CommonAnnotations.summary }}
          *Description:* {{ .CommonAnnotations.description }}
          *Instances:* {{ range .Alerts }}{{ .Labels.instance }} {{ end }}
EOF

Step 3 — Create the Systemd Service

Create a systemd unit file so Alertmanager starts on boot and is managed with standard service commands.

sudo tee /etc/systemd/system/alertmanager.service > /dev/null <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager 
  --web.listen-address=:9093 
  --cluster.listen-address=""
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
sudo systemctl status alertmanager

Step 4 — Open the Firewall and Connect Prometheus

Allow traffic on port 9093 and update your Prometheus configuration to send alerts to Alertmanager.

sudo firewall-cmd --permanent --add-port=9093/tcp
sudo firewall-cmd --reload

# Add or update the alerting block in /etc/prometheus/prometheus.yml
# Find the alerting: section and set:
#
# alerting:
#   alertmanagers:
#     - static_configs:
#         - targets:
#             - localhost:9093
#
sudo systemctl reload prometheus

Step 5 — Configure amtool and Test Alert Delivery

Configure amtool with the Alertmanager URL and send a test alert to verify routing to each receiver.

mkdir -p ~/.config/amtool
cat > ~/.config/amtool/config.yml <<'EOF'
alertmanager.url: http://localhost:9093
EOF

# Fire a test critical alert (routes to PagerDuty)
amtool alert add alertname="TestCritical" severity="critical" 
  instance="web01:9100" 
  --annotation=summary="Test critical alert from amtool" 
  --annotation=description="Verify PagerDuty integration is working"

# Fire a test warning alert (routes to Slack)
amtool alert add alertname="TestWarning" severity="warning" 
  instance="web01:9100" 
  --annotation=summary="Test warning alert from amtool" 
  --annotation=description="Verify Slack integration is working"

# List active alerts
amtool alert
# Check silence status
amtool silence query

Step 6 — Validate the Configuration with amtool

Use amtool check-config to validate the YAML syntax and test route matching logic before deploying changes.

# Validate config file syntax
amtool check-config /etc/alertmanager/alertmanager.yml

# Test which receiver a given set of labels routes to
amtool config routes test severity=critical alertname=InstanceDown
amtool config routes test severity=warning alertname=HighMemoryUsage

# Show the full route tree
amtool config routes show

# Reload config without restart (sends SIGHUP)
sudo systemctl reload alertmanager

Conclusion

You now have Alertmanager running on RHEL 9 with severity-based routing that escalates critical alerts to PagerDuty and sends warning alerts to Slack. The inhibition rule prevents redundant notifications when a critical condition already covers the same service, and the group timing settings avoid alert fatigue during incident spikes. Replace the placeholder integration key and webhook URL with your real credentials and adjust the repeat_interval and group_interval values to match your team’s on-call policy.

Next steps: How to Write Prometheus Alerting Rules for Node Exporter on RHEL 9, How to Set Up VictoriaMetrics as a Prometheus Alternative on RHEL 9, and How to Configure Grafana Notification Channels for Multi-Team Alerting.