Table of Contents
Introduction
Managing web server logs is essential for maintaining performance, troubleshooting issues, and understanding user behavior. Nginx generates valuable access and error logs. This tutorial will guide you through installing Fluent Bit on a Droplet, configuring it to collect Nginx logs, and sending them to the cloud provider's Managed OpenSearch for analysis.
Prerequisites
Before you start, ensure you have the following in place:
- A cloud servers/s with Nginx webserver installed.
- A Managed OpenSeach Cluster.
Step 1 - Installing Fluent Bit
Fluent Bit is an open-source and lightweight log processor and forwarder. It is designed to collect data and logs from various sources, process or transform them, and then forward them to different destinations.
FluentBit can be installed on multiple Platforms like Ubuntu, Debian, RedHat, CentOS by running the following command on your Droplet terminal:
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
Step 2 - Configuring Fluent Bit to Send Logs to OpenSearch
By default, Fluent Bit configuration files are located in /etc/fluent-bit/. To forward logs to OpenSearch, you'll need to modify the fluent-bit.conf file.
FluentBit Inputs
Fluent Bit provides a range of input plugins to gather log and event data from various sources. For our use case of collecting logs from log files, we will use the *tail* input plugin. This plugin is specifically designed to read data from files, handle log rotation, and stream new entries as they are written to the log files.
Update the fluent-bit.conf file as follows:
[INPUT]
name tail
Tag nginx.access
path /var/log/nginx/access.log
parser nginx
[INPUT]
Name tail
Tag nginx.error
path /var/log/nginx/error.log
For more information on Fluent Bit inputs, refer to this link: https://docs.fluentbit.io/manual/pipeline/inputs.
fluentbit already provides a default parser for nginx access logs.
FluentBit Outputs
Like input plugins, fluentbit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to OpenSearch, let's make use of the opensearch output plugin.
[OUTPUT]
Name opensearch
Match nginx.access
Host <OpenSearch_Host>
port 25060
HTTP_User doadmin
HTTP_Passwd <OpenSearch_Password>
Index fbit-nginx-access
tls On
Suppress_Type_Name On
[OUTPUT]
Name opensearch
Match nginx.error
Host <OpenSearch_Host>
port 25060
HTTP_User doadmin
HTTP_Passwd <OpenSearch_Password>
Index fbit-nginx-error
tls On
Suppress_Type_Name On
Replace the <OpenSearch_Host> with your OpenSearch server’s hostname and <OpenSearch_Password> with your OpenSearch password.
Once the configurations are set, start fluent bit service by running:
systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service
Troubleshooting
You can verify that Logstash can connect to OpenSearch by testing connectivity:
curl -u your_username:your_password -X GET "https://your-opensearch-server:25060/_cat/indices?v"Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials.
Ensure that data is properly indexed in OpenSearch:
curl -u your_username:your_password -X GET "http://your-opensearch-server:25060/<your-index-name>/_search?pretty"Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials, and your-index-name with the index name.
Ensure firewall rules and network settings allow traffic between Logstash and OpenSearch on port.
By default, logs are written to the system log.
sudo journalctl -u fluent-bitEnsure the configuration files are syntactically correct.
/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-runConclusion
In this tutorial, we've walked through the process of installing and configuring Fluent Bit to collect Nginx logs and forward them to the cloud provider's Managed OpenSearch for analysis. By following these steps, you should now have a streamlined log management system in place, allowing you to effectively monitor and analyze your web server logs.
Installation: We installed Fluent Bit on a Droplet using a simple curl command suitable for various platforms.
Configuration: We updated the fluent-bit.conf file to collect Nginx access and error logs using the tail input plugin and send them to OpenSearch using the opensearch output plugin.
Service Management: We enabled and started the Fluent Bit service to ensure it's running smoothly.
Troubleshooting: We covered essential troubleshooting steps, including verifying connectivity, checking data ingestion, and reviewing Fluent Bit logs.
With Fluent Bit successfully configured, you'll be able to leverage OpenSearch's powerful search and visualization capabilities to gain insights from your Nginx logs.
