I just released the CircuitBreaker Plugin for Grails.

I recently read Michael Nygard’s book Release It, which is based on his experiences from the trenches in tracking down the causes of large system failures. He relates a number of his experiences and then discusses the patterns and antipatterns that he’s distilled after seeing the same causes of failure over and over. It changed how I think about certain aspects of designing and developing inter-related systems. I highly recommend the book. Though it doesn’t get into implementation details, it’s one of the most interesting (and applicable) technical books I’ve read in a while.

The two things that stick with me the most are both from the section on stability. The first is the effective use of timeouts – especially when integrating with other systems. This can be done in Grails/Groovy/Java/etc by using HttpClient from Apache commons.

The second thing that I can’t stop thinking about is the CircuitBreaker Pattern. Like an electrical circuit breaker in a house, the idea is to recognize a problem and fail first, thereby controlling the overall failure mode. When integrating your app with an external system, this means you stop calling it when it continues to fail over and over and over again.

From Release It!:

…the circuit breaker exists to allow one subsystem (an electrical circuit) to fail (excessive current draw, possibly from a short-circuit) without destroying the entire system (the house). Furthermore, once the danger has passed, the circuit breaker can be reset to restore full function to the system.

You can apply the same technique to software by wrapping dangerous operations with a component that can circumvent calls when the system is not healthy. This differs from retries, in that circuit breakers exist to prevent operations rather than reexecute them.

Whether these are calls to an external system, or even internal operations that could fail repeatedly and need time to recover, the Circuit Breaker pattern guards against cascading failures, so that failures in one area don’t bring down the entire application.

This led me down the path of looking into AOP in Grails because I wanted to reduce the impact on my application codebase. And for the very same reason, creating a Grails plugin seemed like the obvious way to implement it. Many thanks to Ken DeLong because his post Circuit Breaker in Java from last year was very helpful in getting the groundwork set for the plugin. In addition to the tests included in the plugin, I’ve created an example application to show how to use the plugin.

Check out the plugin and let me know what you think.