Prometheus generates alerts, but it is Alertmanager that handles routing, deduplication, grouping, and delivery to your notification channels. Without a properly configured Alertmanager, even the most thorough alerting rules in Prometheus are useless — they fire into the void. Integrating Alertmanager with both PagerDuty and Slack gives you a two-tier notification strategy: high-severity incidents page the on-call engineer immediately via PagerDuty, while informational or warning-level alerts appear in a Slack channel for team visibility. This tutorial walks through the full Alertmanager setup on RHEL 7, including configuration, routing trees, inhibit rules, silence management, and testing with amtool.

Prerequisites

  • RHEL 7 with a running Prometheus instance (port 9090)
  • A PagerDuty account with an Events API v2 integration key
  • A Slack workspace with an Incoming Webhook URL configured
  • Root or sudo access on the server
  • Ports 9093 (Alertmanager) open in firewalld if accessed remotely

Step 1: Download and Install Alertmanager

Alertmanager is distributed as a standalone binary. Download the latest stable release from the Prometheus GitHub releases page.

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
cd alertmanager-0.27.0.linux-amd64

Copy the binaries to a system-wide location and set up the directory structure:

cp alertmanager amtool /usr/local/bin/
chmod 755 /usr/local/bin/alertmanager /usr/local/bin/amtool

useradd --no-create-home --shell /sbin/nologin alertmanager

mkdir -p /etc/alertmanager
mkdir -p /var/lib/alertmanager/data
chown -R alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Step 2: Create the Main Configuration File

The Alertmanager configuration lives in /etc/alertmanager/alertmanager.yml. This file defines receivers (where to send alerts), a routing tree (which alerts go where), and optional inhibit rules. Create a comprehensive configuration:

cat > /etc/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m
  smtp_smarthost: 'localhost:25'
  smtp_from: '[email protected]'

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 1h
      continue: true
    - match:
        severity: critical
      receiver: 'slack-critical'
    - match:
        severity: warning
      receiver: 'slack-warning'
    - match:
        alertname: Watchdog
      receiver: 'null'

receivers:
  - name: 'null'

  - name: 'default-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXX'
        channel: '#alerts-general'
        send_resolved: true
        title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}'
        text: >-
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Description:* {{ .Annotations.description }}
          *Severity:* {{ .Labels.severity }}
          *Started:* {{ .StartsAt | since }}
          {{ end }}

  - name: 'slack-warning'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXX'
        channel: '#alerts-warning'
        send_resolved: true
        color: '{{ if eq .Status "firing" }}warning{{ else }}good{{ end }}'
        title: 'WARNING: {{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

  - name: 'slack-critical'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXX'
        channel: '#alerts-critical'
        send_resolved: true
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
        title: ':fire: CRITICAL: {{ .CommonLabels.alertname }}'
        text: >-
          {{ range .Alerts }}
          *Summary:* {{ .Annotations.summary }}
          *Description:* {{ .Annotations.description }}
          *Runbook:* {{ .Annotations.runbook_url }}
          {{ end }}

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY_HERE'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          resolved: '{{ .Alerts.Resolved | len }}'
          num_firing: '{{ .Alerts.Firing | len }}'
          alertname: '{{ .CommonLabels.alertname }}'
          cluster: '{{ .CommonLabels.cluster }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'Node.*'
    equal: ['instance']
EOF

Validate the configuration file syntax before proceeding:

amtool check-config /etc/alertmanager/alertmanager.yml

Step 3: Create the Systemd Service Unit

Create a systemd service file so Alertmanager starts automatically and is managed like any other service on RHEL 7:

cat > /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager for Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager/data 
  --web.listen-address=0.0.0.0:9093 
  --web.external-url=http://YOUR_SERVER_IP:9093 
  --cluster.listen-address=""
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

Enable and start the service:

systemctl daemon-reload
systemctl enable alertmanager
systemctl start alertmanager
systemctl status alertmanager

Open the firewall port if Alertmanager needs to be reached remotely:

firewall-cmd --permanent --add-port=9093/tcp
firewall-cmd --reload

Step 4: Connect Prometheus to Alertmanager

Prometheus must know where to send firing alerts. Edit /etc/prometheus/prometheus.yml and add or update the alerting section:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093
      timeout: 10s

rule_files:
  - "/etc/prometheus/rules/*.yml"

Reload Prometheus to apply the change without restarting:

curl -X POST http://localhost:9090/-/reload

Step 5: Configure the amtool CLI Client

The amtool binary is the command-line interface for Alertmanager. Configure it to talk to your local instance:

mkdir -p ~/.config/amtool
cat > ~/.config/amtool/config.yml << 'EOF'
alertmanager.url: http://localhost:9093
author: admin
comment_required: false
output: extended
EOF

Verify connectivity and list current alerts:

amtool alert
amtool config routes show
amtool config routes test --verify-receivers=pagerduty-critical severity=critical alertname=HighCPU

Step 6: Send a Test Alert

Use amtool to inject a synthetic alert directly into Alertmanager and confirm it routes correctly to your configured receivers:

amtool alert add 
  alertname="TestCritical" 
  severity="critical" 
  instance="test-host-01" 
  --annotation=summary="Test critical alert from amtool" 
  --annotation=description="This is a synthetic test to verify PagerDuty integration" 
  --end=$(date -d "+5 minutes" --rfc-3339=seconds | sed 's/ /T/')

List active alerts to confirm it was received:

amtool alert query alertname="TestCritical"

Check your Slack channel and PagerDuty incidents dashboard to verify delivery. Once confirmed, expire the test alert by setting its end time to now.

Step 7: Create and Manage Silences

Silences suppress notifications during planned maintenance windows without stopping alert evaluation in Prometheus:

# Silence all alerts for a specific instance for 2 hours
amtool silence add 
  instance="web-server-01" 
  --comment="Scheduled maintenance window" 
  --duration=2h

# Silence a specific alert by name
amtool silence add 
  alertname="DiskSpaceLow" 
  severity="warning" 
  --comment="Known issue, ticket INFRA-4521" 
  --duration=24h

# List all active silences
amtool silence

# Expire a silence by its ID
amtool silence expire SILENCE_ID_HERE

You can also create silences through the Alertmanager web UI at http://YOUR_SERVER:9093/#/silences.

Step 8: Understanding Inhibit Rules

The inhibit rules defined in the configuration prevent alert noise. When a critical alert fires for the same alertname, cluster, and service as an existing warning alert, the warning is suppressed. The second rule suppresses all Node.* alerts when NodeDown is firing for the same instance, preventing a flood of secondary alerts when a host goes completely offline.

Review which alerts are currently inhibited:

amtool alert query --inhibited

With Alertmanager configured, your Prometheus ecosystem now has a production-grade notification pipeline. Critical pages go directly to the on-call engineer via PagerDuty with a short group_wait of 10 seconds, while warnings aggregate in Slack with a 30-second wait to reduce noise from transient spikes. The inhibit rules prevent alert storms during major incidents, and the amtool CLI gives you full operational control over silences and routing directly from the command line. Revisit repeat_interval values as you tune the system — setting them too low causes fatigue, and too high risks missing ongoing incidents.