Grafana Loki is a horizontally scalable, highly available log aggregation system designed to be cost-effective by indexing only metadata labels rather than full log content. Promtail is its companion agent that tails log files and ships them to Loki. Compared to the ELK stack, Loki is significantly lighter on resources, making it an excellent choice for RHEL 9 servers where you want log aggregation without the overhead of Elasticsearch. This tutorial covers deploying both Loki and Promtail as systemd services and querying logs through Grafana using LogQL.

Prerequisites

  • RHEL 9 server with at least 2 GB RAM
  • Root or sudo access
  • Grafana already installed and running (see related tutorials)
  • Port 3100 open in firewalld for Loki’s HTTP API
  • wget or curl available for downloading binaries

Step 1 — Download Loki and Promtail Binaries

Loki and Promtail are distributed as single static binaries from the GitHub releases page. Download the latest 2.x versions, make them executable, and place them in /usr/local/bin.

LOKI_VERSION="2.9.8"

# Download and install Loki
curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip"
unzip loki-linux-amd64.zip
mv loki-linux-amd64 /usr/local/bin/loki
chmod +x /usr/local/bin/loki

# Download and install Promtail
curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail
chmod +x /usr/local/bin/promtail

# Verify
loki --version
promtail --version

Step 2 — Create Configuration Files

Loki needs a configuration file defining storage paths and retention settings. Promtail needs a configuration file listing the log targets it should tail and the labels to attach to each stream.

mkdir -p /etc/loki /etc/promtail /var/lib/loki

# Loki configuration
cat > /etc/loki/loki-local-config.yaml < /etc/promtail/promtail-local-config.yaml << 'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: __hostname__
          __path__: /var/log/messages

  - job_name: secure
    static_configs:
      - targets:
          - localhost
        labels:
          job: secure
          host: __hostname__
          __path__: /var/log/secure
EOF

mkdir -p /var/lib/promtail

Step 3 — Create systemd Service Units

Running Loki and Promtail as systemd services ensures they start automatically on boot and are managed consistently with the rest of your system.

# Loki service
cat > /etc/systemd/system/loki.service < /etc/systemd/system/promtail.service << 'EOF'
[Unit]
Description=Promtail Log Shipper
After=network.target loki.service

[Service]
Type=simple
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-local-config.yaml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now loki promtail
systemctl status loki promtail

Step 4 — Open Firewall and Add Loki as a Grafana Data Source

Open port 3100 so Grafana can reach the Loki API, then add Loki as a data source in Grafana’s web interface.

firewall-cmd --permanent --add-port=3100/tcp
firewall-cmd --reload

# Verify Loki is responding
curl -s http://localhost:3100/ready
# Expected output: ready

# Verify Promtail is pushing logs
curl -s http://localhost:9080/metrics | grep promtail_sent_entries_total

In Grafana, go to Connections → Data Sources → Add data source, choose Loki, and set the URL to http://localhost:3100. Click Save & Test to confirm connectivity.

Step 5 — Query Logs with LogQL

Open the Grafana Explore view, select the Loki data source, and use LogQL to query your log streams. LogQL uses label selectors followed by optional filter expressions.

# Show all syslog entries
{job="syslog"}

# Filter for error-level messages
{job="syslog"} |= "error"

# Filter secure log for SSH authentication failures
{job="secure"} |= "Failed password"

# Count SSH failures per minute (metric query)
rate({job="secure"} |= "Failed password" [1m])

# Parse structured fields with logfmt parser
{job="syslog"} | logfmt | level="error"

Conclusion

Loki and Promtail are now running as managed systemd services on your RHEL 9 server, collecting logs from /var/log/messages and /var/log/secure and making them queryable through Grafana’s Explore interface. The Loki approach—indexing only labels rather than full text—means storage costs and memory usage remain low even as log volume grows, making it a practical alternative to full ELK deployments for smaller environments. For production use, consider configuring log rotation, setting a meaningful retention_period in loki-local-config.yaml, and adding additional Promtail scrape targets for application logs.

Next steps: How to Configure Prometheus Node Exporter on RHEL 9, How to Set Up Grafana Alerting and Notification Channels on RHEL 9, and How to Install InfluxDB and Telegraf on RHEL 9.