Installing a centralized syslog

Remember that I told you about this guide? So here it is. Basically, it explains step by step how to install Adiscon LogAnalyzer in an Ubuntu box. Let me know if something's not clear enough:


The Prep:

Before we dive into setting up any of this we need to do a little prep work. If you are going to be looking at these logs in a web browser then it might be good if the time stamps you are seeing reflect the appropriate timezone.



You first need to select your timezone so that your system knows where it is located. Look for your timezone in /usr/share/zoneinfo, then copy it to /etc/localtime. Please avoid the /usr/share/zoneinfo/Etc/GMT* timezones as their names do not indicate the expected zones. For instance, GMT-8 is in fact GMT+8.

Suppose you are in the eastern timezone:

~# cp /usr/share/zoneinfo/EST5EDT /etc/localtime

Using your favorite text editor, create the file /etc/cron.daily/ntpdate and insert the following:

ntpdate ntp.ubuntu.com

Save the file then

~# chmod 755 /etc/cron.daily/ntpdate

Now let's run it to get our time corrected:

~# /etc/cron.daily/ntpdate

Lastly, verify...

~# date

You should see your current time here.


The Logging:

Now that we have our clocks synchronized let's move on to the meat and potatoes of this article. There are some prerequisite software packages that need to be setup here: Apache, MySQL, PHP... aka the LAMP stack. I am not going to go into detail on these but I do to note a couple of things:

On Ubuntu, just run tasksel and pick LAMP from there.



After installing the LAMP stack, don’t forget to manually restart Apache. I would suggest also setting up phpMyAdmin to simplify managing MySQL.

Ubuntu now comes with rsyslog as its defacto logger but we need to add a little bit of additional functionality to it. Namely, we need to add MySQL output support and add in the Reliable Event Logging Protocol (relp):

~# apt-get install rsyslog-mysql rsyslog-relp
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
dbconfig-common librelp0
The following NEW packages will be installed:
dbconfig-common librelp0 rsyslog-mysql rsyslog-relp
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 677kB of archives.
After this operation, 2,335kB of additional disk space will be used.
Do you want to continue [Y/n]? y

During this install’s process you will be prompted to make the tables that are needed in MySQL:



Do this. You will then be asked for your MySQL root password followed by being asked to create a password for rsyslog to use. This is the password that rsyslog will use in its config files.

Now we need to make a couple of tweeks to the config files of rsyslog. Ubuntu takes advantage of the fact that rsyslog can use multiple config files that are merged into one “config.” You have /etc/rsyslog.conf but you also have a directory named /etc/rsyslog.d/ that contains additional configs. In there you will now see one named mysql.conf that contains the needed info to dump our logs into the database. To turn on accepting remote logs though we still have to uncoment a couple of lines in /etc/rsyslog.conf

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Now, let's apply our changes:

~# service rsyslog restart

It is a little rough to read if your terminal is not wide enough, but you can verify that the logs are going to the database with

~# mysql -p -e "SELECT * FROM Syslog.SystemEvents;"

RELP: Reliable Event Logging Protocol

At this point we have configured our servers to accept messages over TCP & UDP on the standard port of 514, which is what is needed to be compatible with the majority of network devices that can shoot logs to a central server. Now, let's take it a step farther and add support to the config files for a third protocol: RELP, which is implemented via librelp. This is supposed to be a much more reliable method of sending syslog data over the network than even TCP. More info about the unreliability of tcp and the creation of RELP can be found on the developer’s blog. Rsyslog has native support for it via the imrelp & omrelp modules.

Use your favorite editor and create /etc/rsyslog.d/relp.conf and enter the following in it:

$ModLoad imrelp
$InputRELPServerRun 20514

Adding reliability to your logging systems

One of the many helpful articles at rsyslog.com explains how to log massive amounts of syslog data to a database by setting up buffering on your server so that messages are less likely to be lost if a system or link is under very high load. To do this you first need to make a working directory that it can store queue files in if needed.

~ # mkdir -p /var/rsyslog/work

Now we need to add the following to /etc/rsyslog.conf or /etc/rsyslog.d/mysql.conf

# Buffering stuff:
$WorkDirectory /var/rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure

And now we need to restart rsyslog:

~ # service rsyslog restart

The Viewing:

To view the info that we are now dumping into MySQL via the web we need to setup LogAnalyzer. Step one of this is to download the software from http://loganalyzer.adiscon.com/downloads. As of this writing, the newest version is v3.2.1.

~ # wget http://download.adiscon.com/loganalyzer/loganalyzer-3.2.1.tar.gz
~ # tar -xzf loganalyzer-*.*.*.tar.gz
~ # cd loganalyzer-3.0.1
~/loganalyzer-3.0.1# mkdir /var/www/logs
~/loganalyzer-3.0.1# cp -R src/* /var/www/logs/
~/loganalyzer-3.0.1# cp contrib/* /var/www/logs/
~/loganalyzer-3.0.1# cd /var/www/logs/
/var/www/logs# chmod +x configure.sh secure.sh
/var/www/logs# ./configure.sh

To enable the authentication part of LogAnalyzer we need to make an empty database for users to be stored in and grant privileges on it.

/var/www/logs# mysql -p
mysql> create database LogAnalyzerUsers;
mysql> show databases;
mysql> grant all on LogAnalyzerUsers.* to LAUser@'localhost' identified by 'password';
mysql> quit

Now, go to http://yourlogserver/logs/ and you will be pointed to the installation script which will guide you through the process of setting up LogAnalyzer.

Basic Configuration

You can set several basic options here.

• Number of syslog messages per page = 50 (default)
• This is the number of syslog messages displayed on each page. You can increase the value (makes LogAnalyzer slower) or decrease the value (makes it faster).
• Message character limit for the main view = 80 (default)
• Set the number of characters per message which will be shown in the last column of the main view.
• Full messages can be reviewed by hovering the mouse over it.
• Many folks prefer to use a setting of "0", which means complete messages will be displayed
• Show message details popup (default yes) = yes (default). Note that many people find the popups intrusive and prefer to disable them. Use "no" in this case.
• During the setup you will also be prompted to enable the user database. Do so and enter in the information that is requested.
• A couple of pages later you will be prompted for the main (admin) user.
• The defaults on Step 7 demonstrate that it is possible to use this without the database backend. We need to change this to match our setup though.
• Name the source something logical seeing as it is going to be the compiled logs from all your servers.
• Source Type = MYSQL Native
• Select View = Syslog Fields
• Table type = MonitorWare
• Database Host = localhost
• Database Name = Syslog
• Database Tablename = SystemEvents
• Database User = rsyslog
• Enable Row Counting = no

Once you finish up, log into your new site and have a look at what has been being logged on your server so far.