How to Set Up Loki and Promtail for Log Aggregation on RHEL 7

How to Set Up Loki and Promtail for Log Aggregation on RHEL 7

Loki is a horizontally scalable, highly available log aggregation system developed by Grafana Labs. Unlike traditional log management solutions that index the full text of every log line, Loki indexes only the metadata labels attached to each log stream. This makes it dramatically cheaper to operate at scale while still enabling fast log retrieval through Grafana’s Explore interface. Promtail is the companion agent that tails log files on each host and ships them to Loki. Together they form a lightweight, cost-effective alternative to an ELK stack. This tutorial covers downloading and configuring both components on Red Hat Enterprise Linux 7, creating systemd services for each, and connecting them to a Grafana instance for log exploration.

Prerequisites

• RHEL 7 with root or sudo access
• Grafana already installed and running (port 3000)
• Internet access to download binaries from GitHub releases
• At least 2 GB RAM; 4 GB recommended for moderate log volumes
• curl, tar, and unzip utilities installed

Step 1: Create System Users for Loki and Promtail

Running Loki and Promtail as dedicated non-root users limits the blast radius of any security issue. Create the users and the directories they need before downloading the binaries.

sudo useradd --system --no-create-home --shell /sbin/nologin loki
sudo useradd --system --no-create-home --shell /sbin/nologin promtail

sudo mkdir -p /etc/loki /etc/promtail /var/lib/loki /var/log/loki
sudo chown loki:loki /var/lib/loki /var/log/loki
sudo chown promtail:promtail /etc/promtail

Add the promtail user to the adm group so it can read standard system log files in /var/log:

sudo usermod -aG adm promtail

Step 2: Download Loki and Promtail Binaries

Download the pre-built Linux AMD64 binaries directly from the Grafana Labs GitHub release page. At the time of writing, version 2.9.x is the current stable release for RHEL 7 compatibility.

LOKI_VERSION="2.9.4"
cd /tmp

curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip"
curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/promtail-linux-amd64.zip"

unzip loki-linux-amd64.zip
unzip promtail-linux-amd64.zip

sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

sudo chmod 755 /usr/local/bin/loki /usr/local/bin/promtail

# Verify both binaries run
loki --version
promtail --version

Step 3: Configure Loki

Create the Loki configuration file. This defines where Loki stores its data, what schema it uses for indexing, and which address it listens on for incoming Promtail pushes.

sudo tee /etc/loki/loki-config.yaml <<'EOF'
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2022-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
EOF

sudo chown loki:loki /etc/loki/loki-config.yaml

Step 4: Create the Loki systemd Service

Create a systemd unit file so that Loki starts automatically on boot and is managed by systemctl.

sudo tee /etc/systemd/system/loki.service <<'EOF'
[Unit]
Description=Loki Log Aggregation System
Documentation=https://grafana.com/docs/loki/latest/
After=network.target

[Service]
User=loki
Group=loki
Type=simple
ExecStart=/usr/local/bin/loki -config.file /etc/loki/loki-config.yaml
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=loki

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki
sudo systemctl status loki

Verify Loki is listening on port 3100:

curl -s http://localhost:3100/ready
# Expected output: ready

Step 5: Configure Promtail

Promtail reads log files and pushes them to Loki with labels. The configuration below tails /var/log/messages, /var/log/secure, and all files in /var/log/*.log, tagging each stream with the hostname and job name.

sudo tee /etc/promtail/promtail-config.yaml <<'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/promtail-positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: __HOSTNAME__
          __path__: /var/log/{messages,secure,cron,maillog}

  - job_name: app_logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: applogs
          host: __HOSTNAME__
          __path__: /var/log/*.log
EOF

# Replace __HOSTNAME__ with the actual hostname
HOSTNAME=$(hostname -f)
sudo sed -i "s/__HOSTNAME__/${HOSTNAME}/g" /etc/promtail/promtail-config.yaml

sudo chown promtail:promtail /etc/promtail/promtail-config.yaml

Step 6: Create the Promtail systemd Service

sudo tee /etc/systemd/system/promtail.service <<'EOF'
[Unit]
Description=Promtail Log Shipper for Loki
Documentation=https://grafana.com/docs/loki/latest/clients/promtail/
After=network.target loki.service

[Service]
User=promtail
Group=promtail
Type=simple
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-config.yaml
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=promtail

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail
sudo systemctl status promtail

Check Promtail’s internal metrics and confirm it is tailing files:

curl -s http://localhost:9080/metrics | grep promtail_files_active

Step 7: Open Firewall Ports

If Promtail on remote hosts needs to push logs to this Loki server, open port 3100. If Loki is only used locally (Promtail on the same box), this is optional.

sudo firewall-cmd --permanent --add-port=3100/tcp
sudo firewall-cmd --reload

Step 8: Add Loki as a Data Source in Grafana

Log in to Grafana (port 3000). Navigate to Configuration → Data Sources → Add data source. Search for and select Loki. Set the URL to http://localhost:3100 (or the Loki server IP if Grafana is on a different host). Click Save & Test; you should see a green “Data source connected and labels found” message.

Step 9: Exploring Logs in Grafana Explore

Switch to the Explore view in Grafana and select the Loki data source from the dropdown. Use LogQL queries to filter log streams. A few examples:

# All logs from the system job
{job="varlogs"}

# Filter for SSH authentication failures
{job="varlogs", host="web01.example.com"} |= "Failed password"

# Count error lines per minute over the last hour
sum(rate({job="applogs"} |= "ERROR" [1m])) by (host)

# Show only lines containing a specific pattern
{job="varlogs"} |~ "(?i)error|critical|warn"

Use the time picker to restrict the query window and the line limit controls to manage result volume. You can convert any Explore query into a dashboard panel by clicking the “Add to dashboard” button at the top of the Explore view.

Conclusion

You have deployed a complete Loki and Promtail log aggregation stack on RHEL 7. Loki runs as a dedicated system service, storing indexed log chunks on the local filesystem with a boltdb-shipper index. Promtail tails all standard RHEL system log files and forwards them with descriptive labels to Loki’s push API. Grafana’s Explore interface allows you to query logs with LogQL, mixing label filters and text searches to zero in on specific events quickly. This lightweight stack uses a fraction of the resources required by a comparable ELK deployment while integrating seamlessly with the Grafana dashboards you may already use for Prometheus metrics, giving you correlated metrics and logs in a single interface.