At AppliedTrust we really like our logs. We've written about them plenty in the past, including articles on how to get the most out of your centralized logging and on the difficulties of event correlation. Logs allow us to do everything from analyzing incidents to spotting misconfigurations in software. Logs can also help us trend important messages, giving us a better understanding of what's actually going on in our network.
Combing through logs is not fun work. Luckily, there are a few tools out on the market to help make this tedious work a lot less time consuming. Splunk is probably the most popular log analysis product on the market. While I love Splunk and think it's a great tool, its price puts it out of reach for a lot of organizations. That got me thinking about what other opensource software is out there that can provide a similar experience.
My search led me to a couple of products that use some of the industry's latest buzzwords: NoSQL, ElasticSearch, etc. Great -- I've been wanting to learn and play with some products to get my hands dirty. I installed both Graylog and logstash on some test machines to give them a spin.
Graylog2 - Graylog recently switched to using ElasticSearch for message storage. ElasticSearch allows you to scale your logging infrastructure when you want to add more logs. The other neat thing about using ElasticSearch is that it allows you to build graphs of longer periods of time without having to wait for ages to crunch through all of that data. The Graylog2-web-interface has a neat Splunk-like interface that allows a system administrator to create searches that dig through the data easily.
logstash - Logstash is like the Swiss Army knife of event collection and forwarding. Logstash is capable of taking in logs of many different forms. Aside from your normal syslog type message inputs, logstash is capable of receiving Twitter events from the streaming API. The outputs are pretty amazing as well. Logstash can dump your events in ElasticSearch or MongoDB, as a couple of examples. You can even configure XMPP outputs for posting events to humans over instant message. Be sure to use a good filter on there to only send the most important messages, or you'll be lynched by the nearest sysadmin receiving those messages.
