Loggly is a fairly new cloud base log management service. You can think about it as lightweight and hosted version of Splunk. I find it pretty interesting, especially because it provides pay-as-go licensing model which I believe suits better to SaaS business than Splunk’s traditional licenses. I also love the idea that Somebody Else is responsible for keeping the service up and running..
Loggly provides few options for getting the logs into their system. You can use either syslog (tcp or udp) or HTTP. They also provide (or at least link to) some Ruby scripts that can take care of monitoring existing log files. In our case we needed to collect the logs from multiple Linux servers running JBoss. One issue that made things a little bit more complicated was that for certain reasons we only wanted to make changes to the JBoss configuration. The provided Ruby scripts did not feel a good solution as they required ruby & rubygems to be installed (and our servers did not have those).
A logical option was to start investigating how to get logs directly from JBoss to Loggly. JBoss 5.1.0 uses Log4j by default. There were few options. Loggly provide links to various projects that provide log4j appenders that work with Loggly HTTP API. Instead of using those, I decided to tryout syslog. Log4j comes with syslog appender, but there is one major drawback associated with it. The standard appender does not support TCP, only UDP. And sending the logs as UDP packets over the internets did not feel good solution. In my use case it is acceptable to lose some logging for example when server is shutting down, but otherwise I want to make sure all messages end up in Loggly.
Luckily I found http://www.syslog4j.org/ – a pure-Java syslog implementation that has bee around for some time. And even better – it comes with built-in Log4J appender. Syslog4j is very comprehensive solution. It provides support for TCP and UDP among many other fine things and does not require any third party jars for basic operations.
Getting JBoss to work with syslog4j turned out to be quite a challenge although it is actually really simple when you know what you should do. The main problem is that if something goes wrong, you probably don’t get any errors. The logging just does not work. There are two options for doing this. You can either set the configuration inside war/ear or you can use the main JBoss configuration, which affects all applications. In my case I wanted to use the JBoss wide configuration as I don’t want to be modifying the wars when they are deployed to production env where Loggly is used.
Steps on Loggly:
- Create a new syslog TCP input. Make a note of the server name (probably logs.loggly.com) and the port number (which varies) – you will need these when configuring JBoss.
- Make sure the input is in discovery mode. Discovery mode means Loggly will accept inputs from all IP addresses. The input will automatically stay in discovery mode for about 15 mins after it has been created. After that it will only accept traffic from hosts is already knows about.
Steps on JBoss 5.1.0:
- Download syslog4j binary jar from the website e.g. http://www.syslog4j.org/downloads/syslog4j-0.9.46-bin.jar and place it to $JBOSS_HOME/lib directory.
- Modify the $JBOSS_HOME/bin/run.sh script. Locate line that says:
JBOSS_BOOT_CLASSPATH="$runjar"
Modify it to:
JBOSS_BOOT_CLASSPATH="$JBOSS_HOME/lib/syslog4j-0.9.46-bin.jar:$JBOSS_HOME/lib/log4j-boot.jar:$runjar"
NOTE: This seems to be the important thing to get right. I believe the order of jars above can be significant. - Modify the log4j configuration. This is located in $JBOSS_HOME/server//conf/jboss-log4j.xml
- Add new entry for Syslog4j appender that logs into Loggly into jboss-log4j.xml
<appender name="LOGGLY" class="org.productivity.java.syslog4j.impl.log4j.Syslog4jAppender"> <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/> <param name="Protocol" value="tcp"/> <param name="Facility" value="user"/> <param name="Host" value="<YOUR LOGGLY HOST, probably logs.loggly.com"/> <param name="Port" value="<YOUR LOGGLY PORT>"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%c (%t) %m%n"/> </layout> </appender>The conversion pattern is also special. You can’t pass timestamps to Loggly (at least not right now). The messages the current time automatically, so there is no need to add them to log messages. Syslog4j appender also includes the priority so that is not needed either.
- Add references to the newly added appender. Locate this section at the end of the file and add reference to Loggly.
<priority value="${jboss.server.log.threshold}"/> <appender-ref ref="LOGGLY" /> <appender-ref ref="CONSOLE"/> <appender-ref ref="FILE"/>
After making these changes and restarting JBoss you should see some activity on your Loggly account.
Two things I haven’t yet investigated are the performance of this combination and how errors are handled.
The calls to Log4J are blocking by default. This means the application returns only after the message has been written to log file (or sent to syslog, in this case). This could slow down the application if log writing is taking time. Log4J comes with Async appender that partly solves this problem. Async appender maintains internal queue for log messages and a separate thread that takes care of calling the actual appender. This means the calls from application can return quickly, even if writing the log message takes time. If the internal queue becomes full, the calls to async appender either block or it starts discarding the messages.
A (not yet tested) example is below. If you try this, remember to change the reference at the end of Log4J configuration to ASYNC_LOGGLY (instead of LOGGLY).
<appender name="ASYNC_LOGGLY" class="org.apache.log4j.AsyncAppender">
<errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
<!-- How many messages can be kept in the queue -->
<param name="BufferSize" value="100000"/>
<!-- If buffer becomes full, drop log messages-->
<param name="Blocking" value="false" />
<appender-ref ref="LOGGLY"/>
</appender>
