On my current project, our Grails app is running on multiple tomcat instances, which share session state using Terracotta. This gives us a number of benefits, including redundancy and fairly easy zero-downtime deployments. However, any time I need to watch the logs, I have to open multiple screens to get a full view of what’s going on with the app. This isn’t terrible, but over time it’s become more and more annoying to me. I’ve used the free version of Splunk to aggregate logs from these servers, but the results weren’t real-time enough for me. Specifically, I was looking for the same type of responsiveness as tailing the logs on the server directly.

Then, last month at the Strange Loop conference during Kevin Weil’s NoSQL At Twitter session, I was introduced to Scribe, the open source log aggregation server released by Facebook. This seemed to fit what I was looking for. After a little searching I found some great articles by Nathan Milford on installing Scribe and using Digg’s log4j appender for it.

Below are the steps I took to get my Grails app configured to use Scribe via the log4j appender. This is all based on a few of Nathan’s separate blog posts, but pulled together into one place and altered specifically for a Grails app.

1. Installing and Running the Scribe server

To start with, I built and installed Scribe from source rpms as described here.

Next I tried to start Scribe, but ran into problems because the file path didn’t exist as described in the default config file /etc/scribed/default.conf. I created the path and restarted the Scribe server to get things running so I could test it out.

$ mkdir /tmp/scribetest

$ /etc/rc.d/init.d/scribed restart
Stopping scribed:
Starting scribed: 

$ /etc/rc.d/init.d/scribed status
ALIVE

I expect to change that path to something more meaningful than /tmp/scribetest. But to get it running, I really didn’t care what it was for now.

Step 1: Done.

2. Installing the Scribe log4j appender

I pulled the log4j appender from github as described here and copied it into my Grails lib directory.

$ yum -y install git

$ git clone http://github.com/lenn0x/Scribe-log4j-Appender.git

$ cp Scribe-log4j-Appender/scribelog4j.jar my-project/lib

Our project already includes the Thrift client because we’re using Cassandra, otherwise I would have also needed the thrift jar from the Scribe-log4j-Appender/lib directory. Grails already uses log4j and slf4j, so you won’t need those jars.

Next, I added the ScribeAppender to the Grails log4j config in Config.groovy

log4j = {

	appenders {
		console name:'stdout', layout:pattern(conversionPattern: '[%d{yyyy-MM-dd hh:mm:ss.SSS}] %c{2} %m%n')
		appender new org.apache.log4j.scribe.ScribeAppender(name:"scribe", scribe_category:"Core", layout:pattern(conversionPattern: '[%d{yyyy-MM-dd hh:mm:ss.SSS}] %c{2} %m%n'))
	}

	root {
	    debug 'stdout', 'scribe'
	    additivity = true
	}

	// ...
}

For some reason I had to explicitly add ‘org.apache.lucene’ and ‘org.compass’ (which come from the Grails Searchable Plugin) to the log4j config. I don’t fully understand it, but for some reason I’m getting debug-level logs for almost everything (including Compass/Lucene) to both stdout and scribe appenders … but only when the ScribeAppender is configured. If anyone has thoughts on why that might be, feel free to let me know.

With the appender jar file in the lib directory and configured in Config.groovy, I generated a war file and deployed to tomcat. While tomcat was starting, I could see that the scribe logs were already updating.

$ ls /tmp/scribetest/Core
$ tail -f /tmp/scribetest/Core/Core_current

So, I now have a local scribe server running and my Grails app is logging to it.

Step 2: Done.

3. Setting up and writing to a central Scribe server

As expected, setting up a parent/central Scribe server was pretty straightforward. I followed the same steps as above and then modified /etc/scribed/default.conf on the client/tomcat server as shown in the scribe example. The main config change was to update the <primary> section to send log info to the central server, rather than to the local filesystem:

type=network
remote_host=central.scribe.server
remote_port=1463

Step 3: Done.

Next steps

All in all, this came together pretty well for me.

I plan to spend some time to learn more about the Scribe config options, since my current configuration is basically just set to defaults right now. Next up is moving this from dev to our QA servers and then work on what the configuration should be.