How to Configure Prometheus AlertManager on RHEL 7

How to Configure Prometheus AlertManager on RHEL 7

Prometheus collects and stores metrics, but on its own it cannot notify your team when something goes wrong. Alertmanager is the dedicated component that handles alert routing, deduplication, grouping, and delivery to receivers such as email, Slack, or PagerDuty. On RHEL 7, setting up Alertmanager alongside Prometheus gives you a production-grade alerting pipeline without relying on external SaaS platforms. This tutorial walks through every step: downloading the binary, writing the configuration file, wiring up multiple notification channels, configuring inhibition rules, silencing alerts through the web UI, connecting Prometheus to Alertmanager, firing a test alert, and using the amtool command-line interface to manage the system from the terminal.

Prerequisites

• RHEL 7 server with sudo or root access
• Prometheus already installed and running (version 2.x recommended)
• Firewall access to ports 9093 (Alertmanager) and 9090 (Prometheus)
• SMTP credentials if you plan to use email notifications
• Slack webhook URL or PagerDuty integration key for those receivers
• Basic familiarity with YAML syntax and systemctl

Step 1: Download and Install the Alertmanager Binary

Alertmanager is distributed as a pre-compiled Go binary. Download the latest stable release from the Prometheus GitHub releases page. At the time of writing, version 0.26.0 is current.

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvf alertmanager-0.26.0.linux-amd64.tar.gz
cd alertmanager-0.26.0.linux-amd64

Move the binaries to a system-wide location and create the necessary directories:

sudo mv alertmanager /usr/local/bin/
sudo mv amtool /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Verify the binary is accessible:

alertmanager --version
# alertmanager, version 0.26.0

Step 2: Write the Alertmanager Configuration File

The main configuration file lives at /etc/alertmanager/alertmanager.yml. It is structured into four top-level sections: global, route, receivers, and inhibit_rules. Create it now:

sudo tee /etc/alertmanager/alertmanager.yml <<'EOF'
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'supersecretpassword'
  smtp_require_tls: true
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'email-ops'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'email-ops'
    email_configs:
      - to: '[email protected]'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX'
        channel: '#alerts-warnings'
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'
        send_resolved: true

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'your-pagerduty-integration-key'
        severity: '{{ if eq .GroupLabels.severity "critical" }}critical{{ else }}warning{{ end }}'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'job', 'instance']
EOF
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml

Understanding the Configuration Sections

The global block sets defaults shared across all receivers, such as SMTP credentials and the resolve_timeout (how long after an alert stops firing before it is declared resolved). The route block defines a decision tree: every incoming alert starts at the root route and is matched against child routes top-to-bottom. group_by clusters related alerts into a single notification; group_wait delays the first notification to collect more alerts in the same group; repeat_interval controls how often a notification re-fires for an ongoing alert. The inhibit_rules section suppresses lower-severity alerts when a higher-severity alert with the same labels is already firing — preventing alert storms.

Step 3: Create the systemd Service Unit

sudo tee /etc/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager 
  --web.listen-address=0.0.0.0:9093 
  --cluster.listen-address=""
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

Step 4: Connect Prometheus to Alertmanager

Edit /etc/prometheus/prometheus.yml to add the alerting and optionally rule_files sections:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'localhost:9093'

rule_files:
  - '/etc/prometheus/rules/*.yml'

Create a sample alerting rule to test the pipeline:

sudo mkdir -p /etc/prometheus/rules
sudo tee /etc/prometheus/rules/test_alerts.yml <<'EOF'
groups:
  - name: test
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} has been down for more than 1 minute."
      - alert: HighCPULoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU load on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 5 minutes."
EOF
sudo systemctl reload prometheus

Step 5: Open Firewall Ports

sudo firewall-cmd --permanent --add-port=9093/tcp
sudo firewall-cmd --reload

Access the Alertmanager web UI at http://<server-ip>:9093. The interface lists active alerts grouped by label set, and provides a form to create silences.

Step 6: Creating Silences via the Web UI and amtool

A silence suppresses notifications for matching alerts during a defined time window without removing the underlying alert rule. In the web UI, click Silences → New Silence, fill in the matcher (e.g., alertname="InstanceDown"), set the duration, add a comment, and click Create.

The same operation is available from the terminal using amtool:

# Configure amtool to know where Alertmanager is
sudo tee /etc/amtool/config.yml <<'EOF'
alertmanager.url: http://localhost:9093
EOF

# List current alerts
amtool alert query

# Create a silence for 2 hours
amtool silence add alertname="InstanceDown" 
  --duration=2h 
  --author="sysadmin" 
  --comment="Planned maintenance window"

# List all silences
amtool silence query

# Expire a silence by ID
amtool silence expire <silence-id>

Step 7: Firing a Test Alert Manually

You can send a synthetic alert directly to Alertmanager’s API to verify your receiver configuration without waiting for a real incident:

curl -s -X POST http://localhost:9093/api/v2/alerts 
  -H 'Content-Type: application/json' 
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "job": "test-job",
      "instance": "localhost:9999"
    },
    "annotations": {
      "summary": "This is a synthetic test alert",
      "description": "Fired manually via curl to verify Alertmanager routing."
    },
    "generatorURL": "http://prometheus.example.com/graph"
  }]'

Within 30 seconds (the group_wait value), you should receive a Slack message on #alerts-warnings because the label severity: warning matches that route.

Step 8: Validating the Configuration

Alertmanager ships with a built-in configuration check. Run it before reloading the service after any change:

amtool check-config /etc/alertmanager/alertmanager.yml
# Checking '/etc/alertmanager/alertmanager.yml'  SUCCESS
# Found:
#  - global config
#  - route
#  - 0 inhibit rules
#  - 3 receivers
#  - 0 templates

# Reload Alertmanager after a config change without restart
sudo systemctl kill -s SIGHUP alertmanager
# Or via API:
curl -s -X POST http://localhost:9093/-/reload

Configuring Prometheus Alertmanager on RHEL 7 gives your monitoring stack a robust notification backbone. By separating alert detection (Prometheus rules) from alert delivery (Alertmanager routing), you gain fine-grained control over who gets notified, when, and through which channel. The inhibition rules prevent alert storms during cascading failures, silences let you suppress noise during maintenance, and the amtool CLI makes it possible to manage everything from scripts or CI pipelines. From here you can explore templating with Go’s text/template to craft richer notification messages, and HA clustering with multiple Alertmanager instances for high-availability deployments.