Fluentd is a unified logging layer that collects logs from multiple sources, parses and transforms them, and forwards the output to destinations such as Elasticsearch, S3, or another Fluentd aggregator. On RHEL 8, the Treasure Data package td-agent provides a supported, RPM-based distribution of Fluentd that integrates with systemd and works alongside your existing application stack. This tutorial covers installing td-agent from the official Treasure Data repository, configuring it to tail Nginx access logs and forward them to Elasticsearch, and exploring Fluent Bit as a lightweight sidecar alternative for resource-constrained hosts.

Prerequisites

  • RHEL 8 server with sudo access
  • Nginx installed and writing logs to /var/log/nginx/access.log
  • An Elasticsearch instance reachable from the server (optional for full pipeline testing)
  • Outbound internet access to reach the Treasure Data and Fluent Bit package repositories

Step 1 — Add the Treasure Data Repository and Install td-agent

Treasure Data provides a convenience script that adds the correct RPM repository for your OS version and then installs td-agent with dnf.

# Add the Treasure Data RPM repository
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sudo sh

# Install td-agent
sudo dnf install -y td-agent

# Verify installation
td-agent --version

# Enable and start the service
sudo systemctl enable --now td-agent
sudo systemctl status td-agent

Step 2 — Install Required td-agent Plugins

The Elasticsearch output plugin and the parser plugin for Nginx logs are not bundled by default. Install them with td-agent-gem.

# Install the Elasticsearch output plugin
sudo td-agent-gem install fluent-plugin-elasticsearch

# Install the Nginx parser helper (optional but recommended)
sudo td-agent-gem install fluent-plugin-nginx-parser

# List installed plugins
sudo td-agent-gem list | grep fluent-plugin

# Allow td-agent to read Nginx log files
sudo usermod -aG adm td-agent
sudo systemctl restart td-agent

Step 3 — Configure td-agent to Tail Nginx Logs

Replace the default configuration in /etc/td-agent/td-agent.conf with a pipeline that tails the Nginx access log, parses each line into structured fields, and forwards the result to Elasticsearch.

sudo tee /etc/td-agent/td-agent.conf > /dev/null <<'EOF'
# ── Source: tail Nginx access log ──────────────────────────────────────────

  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag nginx.access
  read_from_head true
  
    @type nginx
  


# ── Filter: add hostname field ──────────────────────────────────────────────

  @type record_transformer
  
    hostname "#{Socket.gethostname}"
    environment production
  


# ── Match: forward to Elasticsearch ────────────────────────────────────────

  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  logstash_prefix nginx
  include_timestamp true
  
    flush_interval 5s
    retry_max_times 5
    overflow_action block
  


# ── Match: also forward raw records to another Fluentd aggregator ──────────
# Uncomment to enable forwarding to a central log server
# 
#   @type forward
#   
#     host log-aggregator.example.com
#     port 24224
#   
# 
EOF

sudo systemctl restart td-agent
sudo systemctl status td-agent

Step 4 — Test the Pipeline and Inspect Logs

Generate some Nginx traffic and verify that td-agent is parsing and forwarding records correctly by inspecting the td-agent log file.

# Generate test traffic against Nginx
for i in {1..10}; do curl -s http://localhost/ > /dev/null; done

# Watch td-agent process the new log lines
sudo tail -f /var/log/td-agent/td-agent.log

# Check for errors in the td-agent log
sudo grep -i "error|warn" /var/log/td-agent/td-agent.log | tail -20

# Run a quick syntax check on the config file
sudo td-agent --dry-run -c /etc/td-agent/td-agent.conf

Step 5 — Install Fluent Bit as a Lightweight Alternative

Fluent Bit is a sub-project of Fluentd optimised for low memory footprint, making it ideal for edge nodes or containers. Install it on RHEL 8 from the official Fluent Bit repository.

# Add the Fluent Bit repository
sudo tee /etc/yum.repos.d/fluent-bit.repo > /dev/null < /dev/null <<'EOF'
[SERVICE]
    Flush        5
    Log_Level    info

[INPUT]
    Name   tail
    Path   /var/log/nginx/access.log
    Tag    nginx.access
    Parser nginx

[OUTPUT]
    Name   forward
    Match  *
    Host   localhost
    Port   24224
EOF

sudo systemctl enable --now fluent-bit
sudo systemctl status fluent-bit

Step 6 — Manage and Rotate Logs

Configure logrotate to rotate td-agent’s own log files and ensure the service reloads gracefully after rotation to avoid losing buffered records.

# td-agent ships a logrotate file; verify it is in place
cat /etc/logrotate.d/td-agent

# Force a manual log rotation for testing
sudo logrotate -f /etc/logrotate.d/td-agent

# Send SIGUSR1 to td-agent to trigger a graceful log reopen
sudo kill -USR1 $(sudo systemctl show -p MainPID td-agent | cut -d= -f2)

# Check buffer directory size (important if Elasticsearch is unavailable)
sudo du -sh /var/log/td-agent/buffer/ 2>/dev/null || echo "No buffer directory found"

# Reload td-agent config without full restart
sudo systemctl reload td-agent

Conclusion

You have installed Fluentd via the td-agent package on RHEL 8, configured a full log pipeline that parses Nginx access logs and ships them to Elasticsearch, and explored Fluent Bit as a resource-efficient alternative for forwarding roles. The buffering and retry settings in the Elasticsearch output plugin protect against data loss during downstream outages, while logrotate integration keeps disk usage under control. With this foundation you can extend the pipeline to cover application logs, system journals, and custom parsers for any log format your stack produces.

Next steps: Centralising Kubernetes Pod Logs with Fluentd DaemonSet, Building an EFK Stack on RHEL 8, and Parsing Custom Application Logs with Fluentd Regex Filters.