Grafana Unified Alerting, introduced in Grafana 9 and enabled by default in Grafana 10+, provides a centralised alerting engine that works across all data sources and replaces the older per-panel Legacy Alerting system. On RHEL 9, configuring Grafana alerting correctly means setting up contact points for notification delivery, building a notification policy routing tree that maps alert labels to the right recipients, and writing alert rules that fire when Prometheus queries breach defined thresholds. This tutorial walks through the complete workflow from enabling Unified Alerting in grafana.ini through to creating a live alert rule and verifying delivery to Slack and email.

Prerequisites

  • Grafana 9.x or 10.x installed on RHEL 9 and running (systemctl status grafana-server)
  • A Prometheus data source already configured in Grafana with Node Exporter metrics available
  • A Slack workspace with an incoming webhook URL, or an SMTP server for email alerts
  • Root or sudo access to edit /etc/grafana/grafana.ini

Step 1 — Enable Unified Alerting in grafana.ini

Grafana 9 ships with Unified Alerting enabled but Legacy Alerting may still be active if you upgraded from an older version. Verify and explicitly enable Unified Alerting, then configure SMTP for email contact points.

vi /etc/grafana/grafana.ini

# Under the [alerting] section, disable legacy alerting:
[alerting]
enabled = false

# Under the [unified_alerting] section, enable unified alerting:
[unified_alerting]
enabled = true

# Configure SMTP for email notifications under [smtp]:
[smtp]
enabled = true
host = smtp.example.com:587
user = [email protected]
password = YourSMTPPassword
from_address = [email protected]
from_name = Grafana Alerts
startTLS_policy = MandatoryStartTLS

systemctl restart grafana-server
systemctl status grafana-server

Step 2 — Create a Contact Point

Contact points define where alert notifications are delivered. In the Grafana web UI, navigate to Alerting → Contact Points → Add contact point. Create both a Slack contact point and an email contact point as shown in the configuration reference below.

# Slack contact point settings (enter in Grafana UI):
# Name: slack-ops
# Type: Slack
# Webhook URL: https://hooks.slack.com/services/T.../B.../...
# Channel: #ops-alerts
# Title: [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
# Text: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}

# Email contact point settings (enter in Grafana UI):
# Name: email-ops
# Type: Email
# Addresses: [email protected]
# Subject: [Grafana] {{ .CommonLabels.alertname }} is {{ .Status | toUpper }}

# After saving, use "Test" button to verify delivery before creating alert rules.
# A test notification should appear in your Slack channel within seconds.

Step 3 — Create a Notification Policy

Notification policies form a routing tree that matches alert labels to contact points. Navigate to Alerting → Notification Policies to configure the default route and any specific label-based routes.

# Default policy (root route) settings in Grafana UI:
# Default contact point: email-ops
# Group by: alertname, instance
# Group wait: 30s
# Group interval: 5m
# Repeat interval: 4h

# Add a nested policy for critical severity to also notify Slack:
# Matching labels: severity = critical
# Contact point: slack-ops
# Continue matching subsequent sibling nodes: enabled

# This means critical alerts go to BOTH email-ops (default) and slack-ops (specific route).
# Warning-level alerts only go to email-ops via the default route.

Step 4 — Create an Alert Rule

Navigate to Alerting → Alert Rules → New alert rule. Define a rule based on a PromQL expression that evaluates server load and fires when the 1-minute load average exceeds 2 for more than 5 minutes.

# Alert rule configuration (entered in Grafana UI):
# Rule name: HighSystemLoad
# Data source: Prometheus
# PromQL expression:
node_load1{instance="rhel9-server:9100"} > 2

# Evaluation settings:
# Evaluation group: system-health (create new, interval: 1m)
# Pending period: 5m   (alert must be firing for 5m before notifications send)

# Labels:
# severity = critical
# team = ops

# Annotations:
# summary = High system load on {{ $labels.instance }}
# description = Load average is {{ $value | printf "%.2f" }} — threshold is 2.0

# After saving, the rule appears under Alert Rules with state: Normal.
# Trigger it manually: stress --cpu 4 --timeout 120 (requires stress package)

Step 5 — View Alert History, Silences, and Dashboard Panels

Grafana records a full history of alert state transitions and allows you to silence noisy alerts during maintenance windows. You can also annotate dashboard panels with alerting state indicators.

# Install the stress tool to test your alert rule
dnf install -y stress

# Generate artificial CPU load to trigger the alert
stress --cpu 4 --timeout 360 &

# In Grafana UI — monitor alert state:
# Alerting → Alert Rules → HighSystemLoad → state changes: Normal → Pending → Firing

# To create a silence (suppress notifications during maintenance):
# Alerting → Silences → Add Silence
# Matchers: alertname = HighSystemLoad
# Duration: 2h
# Comment: Planned maintenance window

# To add alert state to a dashboard panel:
# Edit panel → Alert tab → Link existing alert rule: HighSystemLoad
# The panel heartbeat line turns red when the alert is firing.

Conclusion

Grafana Unified Alerting is now fully configured on your RHEL 9 server with a Slack contact point for critical alerts, an email contact point as the default route, and a live alert rule watching system load via Node Exporter metrics. The notification policy routing tree ensures the right teams are notified at the right severity, while the pending period prevents flapping alerts from flooding your channels. For more sophisticated setups, explore Grafana’s alert templating with Go template syntax, multi-dimensional alerts using label-based grouping, and integrating with PagerDuty or OpsGenie contact point types for on-call escalation.

Next steps: How to Install InfluxDB and Telegraf on RHEL 9, How to Configure Prometheus Node Exporter on RHEL 9, and How to Set Up Loki and Promtail for Log Aggregation on RHEL 9.