How to Set Up Grafana Alerting and Notification Channels on RHEL 7

Grafana’s alerting system transforms your monitoring dashboards from a passive display into an active early-warning network. When a metric crosses a threshold — CPU saturation, disk exhaustion, an elevated error rate — Grafana fires an alert and routes a notification to whichever channels you have configured: email, Slack, PagerDuty, or dozens of other integrations. This guide covers both the legacy panel-based alert system and Grafana’s newer Unified Alerting engine introduced in version 8, walking through SMTP configuration for email notifications, Slack webhook setup, PagerDuty integration, and the creation of practical alert rules backed by Prometheus queries on a RHEL 7 installation.

Prerequisites

Grafana 8.x or 9.x installed and running on RHEL 7 (systemctl status grafana-server)
A Prometheus data source already connected to Grafana
Root or sudo access to edit Grafana’s configuration file
An SMTP relay or mail server for email notifications
A Slack workspace with permission to create incoming webhooks (for Slack channel)

Step 1: Understand Legacy vs Unified Alerting

Grafana shipped two alerting systems during the version 7–9 transition period.

Legacy panel alerts (Grafana <= 7.x): Alerts are defined directly on individual dashboard panels. Each panel can have one alert rule. Notification channels are configured globally and selected per alert. This system is simple but not well suited to managing hundreds of rules.
Unified Alerting (Grafana >= 8.x, default from 9.x): Rules are managed centrally under Alerting in the left sidebar, independently of dashboards. Supports multi-dimensional alerts, silence management, and contact point routing via a notification policy tree. This is the system covered in detail in this guide.

If you are on Grafana 8.x and need to enable Unified Alerting (it may be disabled by default on upgrades), add the following to /etc/grafana/grafana.ini:

[unified_alerting]
enabled = true

[alerting]
enabled = false

sudo systemctl restart grafana-server

Step 2: Configure SMTP for Email Notifications

Email notifications require an outbound SMTP relay. Edit /etc/grafana/grafana.ini and fill in the [smtp] section:

sudo vi /etc/grafana/grafana.ini

[smtp]
enabled = true
host = smtp.example.com:587
user = [email protected]
# Password can also be set via the GF_SMTP_PASSWORD environment variable
password = YourSMTPPassword
cert_file =
key_file =
skip_verify = false
from_address = [email protected]
from_name = Grafana Alerts
ehlo_identity = grafana.example.com
startTLS_policy = MandatoryStartTLS

[emails]
welcome_email_on_sign_up = false

sudo systemctl restart grafana-server

Test the SMTP configuration from the Grafana UI at Alerting → Contact points → Add contact point, select Email, enter a test recipient address, and click Test.

Step 3: Create an Email Contact Point

In Grafana, navigate to Alerting → Contact points and click New contact point.

Name: ops-email
Type: Email
Addresses: [email protected];[email protected]
Enable Single email to send one email per alert group rather than one per firing alert instance

Click Save contact point.

Step 4: Create a Slack Notification Channel

In your Slack workspace, go to Apps → Incoming Webhooks and create a new webhook for the channel you want to alert in (e.g., #ops-alerts). Copy the webhook URL — it looks like https://hooks.slack.com/services/T000000/B000000/xxxx.

In Grafana, create a new contact point:

Name: slack-ops
Type: Slack
Webhook URL: paste the Slack webhook URL
Channel: #ops-alerts (overrides the webhook default if needed)
Username: Grafana
Icon emoji: :bell:
Title: {{ template "slack.default.title" . }}
Text: {{ template "slack.default.text" . }}

Click Test to send a test message to the Slack channel, then Save contact point.

Step 5: Configure PagerDuty Integration

PagerDuty is the standard on-call escalation platform for production environments. In your PagerDuty account, create a new Service with an Events API v2 integration and copy the Integration Key (also called the routing key).

In Grafana, create a new contact point:

Name: pagerduty-critical
Type: PagerDuty
Integration Key: paste the PagerDuty integration key
Severity: critical
Class: server
Component: leave blank or enter a service name

Click Save contact point. PagerDuty incidents will be automatically resolved when the alert state returns to OK.

Step 6: Set Up a Notification Policy

Notification policies control which contact points receive which alerts based on label matchers. Navigate to Alerting → Notification policies.

The root policy is the catch-all. Edit it to use the ops-email contact point as the default. Then add specific child policies:

Matcher: severity = critical → contact point: pagerduty-critical, group wait: 30s, group interval: 5m
Matcher: team = devops → contact point: slack-ops, group wait: 1m, group interval: 10m

Step 7: Create an Alert Rule on a Prometheus CPU Query

Navigate to Alerting → Alert rules → New alert rule and build a CPU alert:

Name: HighCPUUsage
Data source: Prometheus
Query A:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Condition: IS ABOVE 90
Evaluate every: 1m, For: 5m (alert only fires if condition is true for a sustained 5 minutes)

Scroll down to Labels and add:

severity = warning
team = devops

Under Annotations, add human-readable context:

Summary: High CPU usage on {{ $labels.instance }}
Description: CPU usage is {{ $values.A }}% on {{ $labels.instance }}

Click Save rule and exit.

Step 8: Alert States Explained

Grafana Unified Alerting has four possible alert instance states:

OK — The query returned a value and the condition was not met. No notification is sent (unless a resolve notification is configured).
Alerting — The condition has been met for the full For duration. Notifications are sent to the matched contact points.
No Data — The query returned no data. Can be configured to treat as OK, Alerting, or No Data depending on your needs.
Error — The query failed to execute (for example, the data source is unreachable). Configurable to treat as OK or Alerting.

Step 9: Notification Templates

Grafana supports Go template syntax for customising notification message bodies. Navigate to Alerting → Contact points → Notification templates and create a custom template:

{{ define "custom.email.subject" }}
[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }} on {{ .GroupLabels.instance }}
{{ end }}

{{ define "custom.email.body" }}
Alert: {{ .GroupLabels.alertname }}
Status: {{ .Status }}
Severity: {{ .CommonLabels.severity }}
Instance: {{ .CommonLabels.instance }}

{{ range .Alerts }}
  State: {{ .Status }}
  Summary: {{ .Annotations.summary }}
  Description: {{ .Annotations.description }}
  Started: {{ .StartsAt }}
{{ end }}
{{ end }}

Reference this template in the Email contact point by setting Subject to {{ template "custom.email.subject" . }} and Body to {{ template "custom.email.body" . }}.

Step 10: Creating and Managing Silences

During planned maintenance windows, use silences to suppress notifications without modifying alert rules. Navigate to Alerting → Silences → New silence.

Matchers: instance = db01.example.com (suppress all alerts for a specific host)
Duration: Set a start and end time for the maintenance window
Comment: describe the reason (e.g., “Scheduled disk replacement Saturday 02:00–06:00 UTC”)

Silences can also be created via the Grafana API for automated integration with your change management system:

curl -X POST http://localhost:3000/api/alertmanager/grafana/api/v2/silences 
  -H "Content-Type: application/json" 
  -u admin:admin 
  -d '{
    "matchers": [{"name": "instance", "value": "db01.example.com", "isRegex": false}],
    "startsAt": "2026-05-17T02:00:00Z",
    "endsAt": "2026-05-17T06:00:00Z",
    "comment": "Scheduled maintenance",
    "createdBy": "automation"
  }'

Conclusion

Grafana’s Unified Alerting system provides a powerful, centralised way to manage alert rules across all your Prometheus (and other) data sources on RHEL 7. Email notifications through SMTP ensure broad reach for non-urgent alerts, while Slack webhooks deliver real-time messages to your team channel. PagerDuty integration handles critical escalations that require an on-call response, including automatic incident resolution when the alert clears. Custom notification templates let you tailor the content of every notification to match your team’s conventions. With silences and a well-designed notification policy tree, you have full control over who gets notified, when, and at what severity level — building an alerting system that informs rather than fatigues your operations team.

How to Set Up Grafana Alerting and Notification Channels on RHEL 7