Splunk is a powerful log database that can be used for the analysis of any sort of log data through its easy to use search engine. Security logs, Syslog, Web server logs, and Windows logs are just the beginning.
One of the great features of Splunk is that you can feed pretty much any log into it and start searching. Here at hackertarget.com we usually focus on Open Source Security projects; Splunk is not open source, it is commercial however it does have a Free option that allows up to 500mb of data to be added into the system per day. For larger volume than 500mb per day, the licensing costs start to add up. Splunk installation under Ubuntu is so easy, you can fire up an instance to do ad-hoc analysis of static log files.
Open Source Splunk Alternative
If you are interested in a purely Open Source log search engine, take a look at ELSA - Enterprise Log Search and Archive. This is a relatively new project that is making good progress. It has been included on the latest Security Onion release.
Another Open Source log management option is Graylog2. I am yet to test or explore this alternative to Splunk but I have read some good reviews and it looks promising.
Download Splunk for Ubuntu
Splunk runs on a wide range of computing platforms including Windows, Linux, FreeBSD, OSX, Solaris, AIX and even HPUX.
http://www.splunk.com/download?r=header
We are after the Linux download option, specifically the .deb file as Ubuntu uses the Debian based .deb package format for binary installs. It is a matter of selecting either 32bit or 64bit and then downloading the .deb file.
Not sure whether your Ubuntu is 32bit or 64bit? The easiest way to check this is to use a Unix command uname -a in a terminal window. Bring up a terminal window and type in that command. x64 indicates 64bit while i686 i386 indicates a 32 bit install.
The 32 bit package is about 35.5mb, you will need to signup for a Splunk account to begin the download. It is worth creating an account you will remember as this same acocunt will be used to download additional plugins (apps) from the Splunk site.
Once the download is complete you can install it with the following dpkg command as seen in the output below:
fred@x-wing1:~$ sudo dpkg -i Downloads/splunk-5.0.1-143156-linux-2.6-intel.deb
[sudo] password for fred:
Selecting previously unselected package splunk.
(Reading database ... 239507 files and directories currently installed.)
Unpacking splunk (from .../splunk-5.0.1-143156-linux-2.6-intel.deb) ...
Setting up splunk (5.0.1-143156) ...
----------------------------------------------------------------------
Splunk has been installed in:
/opt/splunk
To start Splunk, run the command:
/opt/splunk/bin/splunk start
To use the Splunk Web interface, point your browser at:
http://x-wing1:8000
Complete documentation is at http://docs.splunk.com/Documentation/Splunk
----------------------------------------------------------------------
Yes it is that easy, no dependencies or mucking around. Now its time to start the Splunk server.
sudo /opt/splunk/bin/splunk start
After some initial setup, you should see:
The Splunk web interface is at http://x-wing1:8000
Login and change your password. You now have your very own Splunk server, just like the cool kids. 🙂
Feed Splunk Data and Search!
Start getting data in the system and then you can search on that data. Data can be input from simple files for some one off analysis, it can read known log files or can listen on a port similar to a syslog server. It is very flexible, for example running it on a TCP port you could even use netcat to pipe a file over the network into Splunk server, or have a syslog server forward some of its logs to the Splunk instance. This would leave you with your existing syslog infrastructure intact for archival purposes but you also have the Splunk instance for easy analysis.
Now you are up to the point where it depends on your network and requirements, so think about how you are going to use it, feed it some data and start searching for stuff. The stuff could be configuration issues, errors, utilization trends or security events. If you want to do some easy testing, just grab a web server log file or other log and feed it in directly with the a file or directory option.
This video is a good introduction to performing Splunk log searches and pulling relevant information from your data.
While I suspect most people will find value from the first day, as you explore the capabilities of the Splunk search engine you will find stuff - its a rabbit hole for systems administrators.