Grafana 10 ships with a completely redesigned unified alerting engine that replaces the older panel-level alerts, offering multi-dimensional alert rules, flexible notification policies, and first-class support for external notification channels including Slack, email, and PagerDuty. Rather than binding an alert to a single dashboard panel, unified alerting lets you define alert rules independently against any data source and then route firing alerts through a hierarchical policy tree that maps labels to contact points. In this tutorial you will configure Grafana’s SMTP settings for email delivery, add a Slack webhook and a PagerDuty integration as contact points, build a notification policy, and create an alert rule from a Prometheus query — all on a RHEL 8 server. You will also learn how to test alert delivery and create silences to suppress noise during maintenance windows.

Prerequisites

  • Grafana 10.x installed and running on RHEL 8 (port 3000)
  • At least one Prometheus data source configured in Grafana with metrics flowing in
  • An SMTP relay or Gmail app-password available for email notifications
  • A Slack workspace where you can create an incoming webhook URL
  • A PagerDuty account with a Services integration key (Prometheus or generic Events API v2)
  • The Grafana configuration file at /etc/grafana/grafana.ini

Step 1 — Configure SMTP for Email Notifications in grafana.ini

Edit the Grafana main configuration file to enable outgoing email. Restart the service after saving so the changes take effect before you test any contact points.

sudo vi /etc/grafana/grafana.ini

# Locate the [smtp] section and set the following values:
[smtp]
enabled = true
host = smtp.gmail.com:587
user = [email protected]
# Store the password in an environment variable or use a secrets manager.
# For a quick test you can paste it here, but rotate it after testing.
password = your-app-password-here
from_address = [email protected]
from_name = Grafana Alerts
startTLS_policy = MandatoryStartTLS

sudo systemctl restart grafana-server
sudo systemctl status grafana-server

Step 2 — Create Contact Points (Slack, Email, PagerDuty)

In the Grafana UI navigate to Alerting → Contact points → Add contact point. Create one contact point for each notification channel as described below.

# ---- Slack contact point ----
# Name: slack-ops
# Integration: Slack
# Webhook URL: https://hooks.slack.com/services/T000/B000/xxxxxxxxxxxx
# Channel: #ops-alerts
# Mention users: @oncall  (optional)

# ---- Email contact point ----
# Name: email-ops
# Integration: Email
# Addresses: [email protected], [email protected]
# (Grafana uses the SMTP config from grafana.ini set in Step 1)

# ---- PagerDuty contact point ----
# Name: pagerduty-critical
# Integration: PagerDuty
# Integration Key: 
# Severity: critical

# After saving each contact point, click "Test" to send a test notification.
# Confirm delivery before proceeding to notification policies.

Step 3 — Configure Notification Policies

Notification policies form a routing tree. The root policy catches all alerts that do not match a more specific rule. Navigate to Alerting → Notification policies and configure the tree to route by label values.

# Root policy (default receiver)
# Contact point: email-ops
# Group by: [alertname, env]
# Group wait: 30s
# Group interval: 5m
# Repeat interval: 4h

# Add a nested policy for critical alerts → PagerDuty:
# Matching labels:  severity = critical
# Contact point:    pagerduty-critical
# Continue matching subsequent sibling policies: false

# Add a nested policy for warning alerts → Slack:
# Matching labels:  severity = warning
# Contact point:    slack-ops
# Continue matching: false

# Save the policy tree.
# All alerts that do NOT match severity=critical or severity=warning
# fall through to the root policy and are sent by email.

Step 4 — Create an Alert Rule from a Dashboard Panel

Open a dashboard that contains a panel backed by a Prometheus query — for example the Node Exporter Full dashboard from the previous tutorial. Click the panel title, choose Edit, then click Alert → Create alert rule from this panel. Configure the rule as shown below and save it to the default alert rule folder.

# Example alert rule — high CPU usage

# Rule name: High CPU Usage
# Folder:    Infrastructure
# Group:     node-exporter

# Query A (Prometheus):
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Condition:
# WHEN  last()  OF  A  IS ABOVE  85

# Pending period: 5m
# (The alert must remain firing for 5 minutes before notifications are sent,
#  reducing false positives from short CPU spikes.)

# Labels:
#   severity = warning
#   env      = production
# (These labels are matched by the notification policy created in Step 3.)

# Annotations:
#   summary:     "High CPU usage on {{ $labels.instance }}"
#   description: "CPU usage is {{ $values.A }}% — above the 85% threshold."

# Save the rule.

Step 5 — Test Alerts and Create a Silence

You can force an alert to fire for testing without waiting for a real threshold breach by temporarily lowering the threshold in the rule definition. Once you have confirmed that notifications arrive at the correct contact points, restore the original threshold. During planned maintenance windows use silences to prevent alert noise.

# --- Test the alert ---
# 1. Edit the alert rule created in Step 4.
# 2. Lower the threshold to 0 (fires immediately).
# 3. Wait ~1 minute, then check Alerting → Alert rules — state should be "Firing".
# 4. Check Slack / email / PagerDuty for the test notification.
# 5. Restore the threshold to 85 and save.

# --- Create a silence (UI or API) ---
# Navigate to Alerting → Silences → Add silence
# Matchers:  severity = warning, env = production
# Start:     now
# End:       now + 2h   (covers the maintenance window)
# Comment:   "Scheduled maintenance — patching prod nodes"
# Click Save.

# Via the Grafana HTTP API (useful for automation):
curl -s -X POST http://admin:admin@localhost:3000/api/alertmanager/grafana/api/v2/silences 
  -H "Content-Type: application/json" 
  -d '{
    "matchers": [{"name":"severity","value":"warning","isRegex":false}],
    "startsAt": "2026-05-17T22:00:00Z",
    "endsAt":   "2026-05-18T00:00:00Z",
    "comment":  "Scheduled maintenance"
  }'

Conclusion

You have configured Grafana 10 unified alerting on RHEL 8 with three contact points covering email, Slack, and PagerDuty, a label-based notification policy tree that routes alerts by severity, and a CPU usage alert rule derived directly from a Prometheus query. The pending period and label-based routing together ensure that only sustained, actionable alerts reach the right on-call teams, while silences provide a clean mechanism to suppress noise during planned maintenance. As your monitoring stack grows you can add further alert rules for memory, disk, and custom application metrics using the same pattern.

Next steps: How to Configure Prometheus Node Exporter on RHEL 8, How to Set Up Loki and Promtail for Log Aggregation on RHEL 8, and How to Install InfluxDB and Telegraf on RHEL 8.