Damia Blog

Mail service monitoring is not as easy as web monitoring, even outgoing smtp is more difficult to monitorize. We can summarize some service problems and how to monitorize them:

If you have your cloud infraestructure on Amazon AWS for audit and control purposes you may want monitor when the firewall rules of any of your security groups have changed.

With this script you will get notified when any of the security groups are modified.


# AWS credentials
export EC2_KEYPAIR=GIJDUYE75JRHFJEJEBHFHEJE88E8ZGGG # name only, not the file name
export EC2_URL=https://ec2.eu-west-1.amazonaws.com
export EC2_PRIVATE_KEY=$HOME/.certs/pk-GIJDUYE75JRHFJEJEBHFHEJE88E8ZGGG.pem
export EC2_CERT=$HOME/.certs/cert-GIJDUYE75JRHFJEJEBHFHEJE88E8ZGGG.pem
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
# aws commands
ec2-describe-group > /home/ubuntu/.audi/group.txt
diff /home/ubuntu/.audi/group.txt /home/ubuntu/.audi/group-old.txt> /home/ubuntu/.audi/diff.txt
if test `cat /home/ubuntu/.audi/diff.txt|wc -l` != 0 ; then
cat /home/ubuntu/.audi/diff.txt|mail -s FW_AMAZON alert@bpmalert.com ;
/usr/local/bin/msg2mobile FW_AMAZON `cat diff.txt |grep PERMISSION|awk '{print($4)}'|head -1`;
fi
mv /home/ubuntu/.audi/group.txt /home/ubuntu/.audi/group-old.txt


All the magic is done with the ec2-describe-group command, storing the current state and watching for differences.

On this post We are going to explain a tip used to count log entries in order to compare load of a specific entries. Let’s explain it:

We have a web server log, for example:

179.158.24.43 - - [02/Nov/2013:17:46:51 +0100] "GET /blog/wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1" 304 214 "http://blog.dom.net/blog/" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/10.0.12 Iceweasel/10.0.12"
179.158.24.43 - - [02/Nov/2013:17:46:51 +0100] "GET /blog/wp-content/themes/the-bootstrap/js/bootstrap.min.js?ver=2.0.3 HTTP/1.1" 304 212 "http://blog.damia.net/damianetblog/" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/10.0.12 Iceweasel/10.0.12"
179.158.24.43 - - [02/Nov/2013:17:46:51 +0100] "GET /blog/wp-content/themes/the-bootstrap/js/the-bootstrap.min.js?ver=2.0.1 HTTP/1.1" 304 212 "http://blog.dom.net/damianetblog/" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/10.0.12 Iceweasel/10.0.12"
179.158.24.43 - - [02/Nov/2013:17:46:58 +0100] "GET /blog/index.php/2011/1/manire/ HTTP/1.1" 200 4916 "http://blog.dom.net/blog/" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/10.0.12 Iceweasel/10.0.12"


Let’s image we want to count how many entries have happened in order to compare the load of the server, we need then to compare on the same interval of time, the solution found is to watch one minute ago.

Let’s made with this script:

tail -1000 /var/log/apache2/access.log |grep -i ` date --date='1 minute ago' +%d/%b/%Y:%H:%M|cut -c 1-17`|wc -l


This script will tell how many entries had the log one minute ago. We can grep it for look for a specific request, and see how many per minute We have.

Today society requires services available 24×7, IT management requires knowledge of the state of the services offered. This necessity have created a lot of tools, products and services. The main approach for this is the external web monitoring.

External web monitoring approach a tries to simulate the actions done by the clients, that is if the clients visits webpages, the monitor software will simulate as a client a request to the webserver, there are some pitfalls on that approach, let’s review them:

Where is located the monitor? Is it representative of the location of clients? Could it detect WAN Network problems? Normally this is solved with monitors distributed across continents being representative of Internet world availability

-Which is the url being monitored? It is very difficult to find a always representative URL, if you simply test a static web page the monitor will miss when the database is not working, the disk is full, or the application server is failing, even is difficult for the monitor to know if the answer of the page is right or not. If the monitor detects the page is failing that is true, but there could be a lot of times that the web is failing and the monitors didn’t notice it.

The first approach:


#wget http://www.bpmalert.com/index.html


A better approach


#wget http://www.bpmalert.com/register.php?user=test&data=test

but the response from the webserver could be “200 OK” but the webpage could say “Database error”.

Even accessing a simple url doesn’t emulate today web complexities, pages are not a simple url, it have a lot of components, and there are things like cookies, session id, ajax, that are not emulated.

-Simulating user flow: As we explain if we limit the monitoring to only a url, you dind’t see the full flow of a web process, for example a real customer will register, receive a mail, click a link and then make a purchase. There are very complex tools that uses real browsers, with a recorded user session and so on, but is impossible to simulate a real user, only a real person could confirm that the web is working ok.

There are some tools that emulates user behaviour but are more used to load tests than for availability monitoring, some of these tools are:

• Jmeter
• Grinder
• Funkload
• Gatling

As a conclusion the external monitoring of webservers are a necessary but they are imperfect, and must be completed with other business monitors in order to know that everything is working ok.

Normaly when you are troubleshooting an Apache server you are checking parameters that don’t have monitors for them.

With the “observate” aproach the best place too look at is:


# tail -f /var/log/apache/access.log


Things to check:

Let’s do in a basic way with:

# tail -f /var/log/apache/access.log|egrep '404|500|503|504'

But there is an advaced tool that read the logs and give you insights regarding the response codes, this tool is
Apachetop, it allows you to overlook, after that for more details you have the real logs, tail and grep as your friends.

On next post we will descrive other tools and techniques to check and test and monitor apache web servers, but we wanted to start with this “observate aproach”.