Table of Contents
This tutorial will build on a basic LEMP stack (Linux, Nginx, MySQL, PHP). Monit will be incorporated to monitor all services in the stack and alert the root user of any adverse conditions. An external Monit server can also be used for remote monitoring of a web application.
URL: https://www.progressiverobot.com/lemp-stack-monitoring-with-monit-on-ubuntu-14-04/
Introduction
Monit is a free and open source service monitoring application which can perform various event-based actions. Monit can send email notifications, restart a service or application, or take other responsive actions.
This tutorial will build on a basic LEMP stack (Linux, Nginx, MySQL, PHP). Monit will be incorporated to monitor all services in the stack and alert the root user of any adverse conditions.
An optional external Monit server can also be used for remote monitoring of a web application or other services.
Prerequisites
- Before we begin, you will first need to set up a Ubuntu 14.04 Droplet
- You will need a standard user account which has sudo privileges
- This tutorial adds Monit to an existing LEMP stack. For a tutorial on how to create an initial LEMP stack, see How To Install Linux, nginx, MySQL, PHP (LEMP) stack on Ubuntu 14.04
- Optional: If you want to monitor a remote website, DNS, or mail server, you should have that server set up with a publicly accessible domain or IP address (more in Step 6)
Step 1 — Configure Email Delivery for Monit Notifications
Part of system monitoring typically involves email notifications for alerts. As such, proper email delivery must be in place in order for Monit to send email notifications. A typical Monit alert email will look similar to this:
From: monit@example.com
To: root@yourserver.com
Resource limit matched Service example.com
Date: Mon, 22 Dec 2014 03:04:06
Action: alert
Host: example.com
Description: cpu user usage of 79.8% matches resource limit [cpu user usage>70.0%]
Your faithful employee,
Monit
This tutorial will set up Monit to send you an email every time an alert is triggered.
Note: Monit's notifications will likely go to your spam folder by default. Reverse DNS (known as a PTR record) must be properly configured to ensure the highest chance mail will be delivered successfully. Your Droplet's hostname must match its fully qualified domain name (FQDN), so, for example, they could both be <^>hostname.example.com<^>. To edit the PTR record of a cloud servers, access the the cloud provider Control Panel. Navigate to Settings and choose the Rename tab. Enter the new hostname and click Rename.
This guide assumes you do not have a preexisting mail transfer agent (MTA), so we will install Postfix. A local installation of Postfix allows the system to send notification emails to an external mail provider such as Gmail or Yahoo.
To begin installing Postfix as your MTA, first update the system's repository source list.
sudo apt-get update
Then install the Postfix and GNU Mailutils packages from Ubuntu's repositories.
sudo apt-get install postfix mailutils
Near the end of the installation, you will be prompted to select a server configuration type as shown in the screenshot below. Choose Internet Site.
When prompted for the System Mail Name, use the fully qualified domain name (FQDN) of your Droplet. Note: The System Mail Name can also be changed later in /etc/mailname.
Next, open the file <^>/etc/aliases<^> for editing. This guide will use Nano but you can use whichever text editor you prefer.
sudo nano /etc/aliases
Here we will add a personal email address where we'll receive Monit's notification emails. These mail notifications will come from our LEMP server's root user.
postmaster: root
root: <^>myemail@gmail.com<^>
It is also possible to add multiple destinations if desired:
root: <^>username<^>, <^>itstaff@mycompany.com<^>, <^>otherperson@other.com<^>
Save your changes and exit Nano. Then run the following to update the aliases file:
sudo newaliases
A test message can be sent from your Droplet to check mail delivery. Please check spam folders if the test message is not first seen in your inbox.
echo test | mail -s "test message from my VPS" root
Step 2 — Install and Configure Monit
Monit is also available in the Ubuntu package repositories. For a brief reference guide on Monit, please see this tutorial.
Monit can be installed on your LEMP server with:
sudo apt-get install monit
On Ubuntu 14.04, the Monit configuration files are located in <^>/etc/monit/<^> and the main Monit configuration file is /etc/monit/monitrc.
To open <^>monitrc<^> in Nano for editing:
sudo nano /etc/monit/monitrc
Uncomment the following lines and change them to match what's shown below:
set mailserver <^>localhost<^> #Use localhost for email alert delivery.
set mail-format {
from: monit@$HOST
subject: monit alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
Your faithful employee,
Monit
}
set alert <^>root@localhost not on { instance, action }<^> #Set email address to receive alerts. This guide uses root mail.
Still in the <^>monitrc<^> file, now uncomment the following lines and change <^>example.com<^> to match your server's domain or IP address.
check system <^>example.com<^>
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 75% then alert
if swap usage > 25% then alert
if cpu usage (user) > 70% then alert
if cpu usage (system) > 30% then alert
if cpu usage (wait) > 20% then alert
We'll also add this entry at the end of the file:
check filesystem rootfs with path / #Alert if low on disk space.
if space usage > 90% then alert
Save your changes and exit Nano.
Step 3 — Configure Service Monitoring for LEMP Services in Monit
On Ubuntu 14.04, Monit configurations can be specified directly in the /etc/monit/monitrc file or via individual files in /etc/monit/conf.d/. In this tutorial, individual files will be created under the /etc/monit/conf.d/ directory.
First we will provide Monit with the means to manage a service. For the sake of simplicity in this tutorial, we will place all process monitoring into a single file located at /etc/monit/conf.d/lemp-services. Using the following entries, Monit will watch Nginx, MySQL and PHP-FPM, and restart these services if they are abnormally stopped for any reason.
We can create the working file with Nano:
sudo nano /etc/monit/conf.d/lemp-services
Add the following entries for the services in our LEMP stack:
check process nginx with pidfile /var/run/nginx.pid
group www-data
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
check process mysql with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
check process php5-fpm with pidfile /var/run/php5-fpm.pid
start program = "/etc/init.d/php5-fpm start"
stop program = "/etc/init.d/php5-fpm stop"
Then save your changes.
Step 4 — Add Actions for Restarting Unhealthy LEMP Services
Now that Monit is able to manage select services, actions can be added to restart services as desired. For example, Monit has the ability to monitor TCP connections. If the server is no longer proving HTTP connections, Monit can restart PHP-FPM or Nginx to automatically resolve the issue.
To build on our existing configuration, we'll now further edit /etc/monit/conf.d/lemp-services. The additions we'll make below are shown <^>in red<^> and we'll tell Monit to restart Nginx and PHP-FPM if HTTP connections are no longer available. Additionally, we will have Monit restart MySQL if the socket is unavailable.
Note: Make sure to use your Droplet's domain or IP address where you see <^>example.com<^> in the first and third entries.
check process nginx with pidfile /var/run/nginx.pid
group www-data
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
<^>if failed host example.com port 80 protocol http then restart<^>
<^>if 5 restarts within 5 cycles then timeout<^>
check process mysql with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
<^>if failed unixsocket /var/run/mysqld/mysqld.sock then restart<^>
<^>if 5 restarts within 5 cycles then timeout<^>
check process php5-fpm with pidfile /var/run/php5-fpm.pid
start program = "/etc/init.d/php5-fpm start"
stop program = "/etc/init.d/php5-fpm stop"
<^>if failed host example.com port 80 protocol http then restart<^>
<^>if 5 restarts within 5 cycles then timeout<^>
Save your changes and close Nano. Then restart Monit to apply the configuration changes you've made so far.
sudo service monit restart
Step 5 (Optional) — Monitor Logs for Errors and Keywords
Monit can also monitor logs for specific keywords and then perform an action or send an alert. This is helpful in cases where a web application is having trouble or when a team requires notification of a particular traceback or event from logs.
Below is an Nginx log example with a timeout error which Monit can monitor for, and alert of:
2014/12/22 11:03:54 [error] 21913#0: *202571 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2600:3c01::f03c:91ff:fe6e:5a91, server: example.com, request: "GET /wp-admin/admin-ajax.php?action=wordfence_doScan&isFork=1&cronKey=40cb51ccsdfsf322fs35 HTTP/1.0", upstream: "fastcgi://unix:/var/run/example.com.sock", host: "example.com"
Building on our existing configuration, open your LEMP services configuration file again in Nano.
sudo nano /etc/monit/conf.d/lemp-services
Add the following entry. This will send a notification when any timeouts occur from Nginx communicating with PHP-FPM.
check file nginx-error with path /var/log/nginx/error.log
if match "^timed out" then alert
Save your change and close Nano. Then restart Monit for the change to take effect:
sudo service monit restart
Step 6 (Optional) — Use Monit to Monitor a Remote Website and Other Services
In addition to using Monit locally, Monit can watch a variety of external services and connections. In this example, we'll use the local instance of Monit we've already set up, and add some new monitoring configurations for external services.
It is preferable to have an external Monit system in an entirely different data center for out-of-band purposes. If a web application is based in New York, having a small external Monit server in San Francisco would be ideal.
Below are examples of external Monit checks which can be implemented on a second host running Monit. These examples would be placed in the external server's /etc/monit/conf.d/lemp-external file to remotely check our LEMP stack at <^>remote-example.com<^>.
Use Nano to create this configuration file:
sudo nano /etc/monit/conf.d/lemp-external
Monitor ICMP response and HTTP & HTTPS connectivity:
# ICMP check
check host <^>remote-example.com<^> with address <^>remote-example.com<^>
if failed icmp type echo
for 5 times within 5 cycles
then alert
# HTTP check
if failed
port 80 protocol http
for 5 times within 5 cycles
then alert
# HTTPS check
if failed
port 443 type tcpSSL protocol http
for 5 times within 5 cycles
then alert
Monitor DNS:
check host <^>ns1.example.com<^> with address <^>ns1.example.com<^>
if failed port 53 type udp protocol dns then alert
Monitor SMTP:
check host <^>smtp.example.com<^> with address <^>smtp.example.com<^>
if failed port 25 type tcp protocol smtp then alert
Monitor Healthcheck URL of Web Application
For web applications, Monit can also perform a particular request on a healthcheck URL. Below is an example of a site <^>remote-example.com<^> with a healthcheck URL of: https://remote-example.com/healthcheck.
check host <^>remote-example.com<^> with address <^>remote-example.com<^>
if failed
port 443 type tcpSSL protocol http
request "/healthcheck"
for 5 times within 5 cycles
then alert
Step 7 — Manage Monit From Command Line
Monit provides a command line utility as well. From there, simple commands can be used to check the overall Monit status and complete useful tasks such as temporarily starting or stopping monitoring.
In order to run Monit status checks from the command line, the Monit web service must be enabled. To do this, open /etc/monit/monitrc for editing in Nano.
sudo nano /etc/monit/monitrc
Uncomment the following lines to enable the web service locally:
set httpd port 2812 and
use address localhost
allow localhost
Save your changes and exit Nano. Then restart Monit:
sudo service monit restart
Now it is possible to check Monit's status from the command line.
Below are commands to temporarily disable and enable monitoring:
sudo monit unmonitor all
sudo monit monitor all
Step 8 — View Reports
Let's take a look at the reports for all the checks we've been setting up.
sudo monit status
Now you'll see output for everything you've configured Monit to check, including local LEMP services, and any external checks:
sudo monit status
The Monit daemon 5.6 uptime: 0m
System 'example.com'
status Running
monitoring status Monitored
load average [0.00] [0.01] [0.05]
cpu 0.5%us 0.4%sy 0.0%wa
memory usage 115132 kB [22.9%]
swap usage 0 kB [0.0%]
data collected Mon, 22 Dec 2014 16:50:42
Filesystem 'rootfs'
status Accessible
monitoring status Monitored
permission 755
uid 0
gid 0
filesystem flags 0x1000
block size 4096 B
blocks total 5127839 [20030.6 MB]
blocks free for non superuser 4315564 [16857.7 MB] [84.2%]
blocks free total 4581803 [17897.7 MB] [89.4%]
inodes total 1310720
inodes free 1184340 [90.4%]
data collected Mon, 22 Dec 2014 16:50:42
Process 'nginx'
status Running
monitoring status Monitored
pid 14373
parent pid 1
uptime 28m
children 4
memory kilobytes 1364
memory kilobytes total 9228
memory percent 0.2%
memory percent total 1.8%
cpu percent 0.0%
cpu percent total 0.0%
port response time 0.018s to example.com:80 [HTTP via TCP]
data collected Mon, 22 Dec 2014 16:50:42
Process 'mysql'
status Running
monitoring status Monitored
pid 12882
parent pid 1
uptime 32m
children 0
memory kilobytes 44464
memory kilobytes total 44464
memory percent 8.8%
memory percent total 8.8%
cpu percent 0.0%
cpu percent total 0.0%
unix socket response time 0.000s to /var/run/mysqld/mysqld.sock [DEFAULT]
data collected Mon, 22 Dec 2014 16:50:42
Process 'php5-fpm'
status Running
monitoring status Monitored
pid 17033
parent pid 1
uptime 0m
children 2
memory kilobytes 13836
memory kilobytes total 22772
memory percent 2.7%
memory percent total 4.5%
cpu percent 0.0%
cpu percent total 0.0%
port response time 0.018s to example.com:80 [HTTP via TCP]
data collected Mon, 22 Dec 2014 16:50:42
File 'nginx-error'
status Accessible
monitoring status Monitored
permission 644
uid 0
gid 0
timestamp Mon, 22 Dec 2014 16:18:21
size 0 B
data collected Mon, 22 Dec 2014 16:50:42
Remote Host 'example.com'
status Online with all services
monitoring status Monitored
icmp response time 0.021s [Echo Request]
port response time 0.107s to example.com:443 [HTTP via TCPSSL]
port response time 0.062s to example.com:80 [HTTP via TCP]
data collected Mon, 22 Dec 2014 16:50:42
Use this data to check the health of your services and see useful statistics.
Troubleshooting
If any problems arise, first check Monit's logs located at /var/log/monit.log. This will give you more information as to the nature of the problem.
Example error log entries:
[UTC Dec 22 13:59:54] error : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 14:10:16] error : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 15:24:19] error : 'example.com' failed protocol test [HTTP] at INET[example.com:80] via TCP -- HTTP: Error receiving data -- Resource temporarily unavailable
[UTC Dec 22 15:57:15] error : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 17:00:57] error : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 17:49:00] error : 'example.com' failed, cannot open a connection to INET[example.com:443/API] via TCPSSL
Conclusion
Having completed this guide, you should now have Monit configured for monitoring a LEMP stack on Ubuntu 14.04. Monit is quite extensible and can be easily customized or expanded for monitoring all kinds of services for small and large networks.
Below are some additional links for Monit:
