The changes in this release consist mainly of exposing the internals of each Circuit Breaker to developers, ops, or anyone that might care.

At Tuesday night’s GUM meeting, I presented my experiences with AOP in Grails and we discussed the Circuit Breaker Plugin for most of the meeting. As part of that discussion, I talked about things that I would like to add to the plugin in the near future.

The main thing on my list was to expose the information about each Circuit Breaker via JMX. To make this happen, I first submitted a patch for the JMX Grails Plugin to allow any Spring bean to be exposed, rather than just Grails Services. Many thanks to Ken Sipe for committing my patch and releasing a new version of the plugin the same evening I submitted the patch!

All you have to do is install the JMX plugin and you can now view the current state of each Circuit Breaker via the JMX Console. In addition to viewing the current breaker state, failure threshold, current failure count, and open-state timeout, you can also manually trip or reset the breaker.

As part of the discussion around JMX on Tuesday, Brian mentioned that often he would prefer to use something quicker and more lightweight than JMX to get a snapshot of the current application state. Whether for easy scripting purposes or just to provide developers with easier access to the data, he thought a simple controller would be more beneficial than JMX integration. That’s now in the plugin as well.

Run grails install-circuit-breaker-controller after the plugin is installed to add CircuitBreakerController.groovy to your project. The controller displays the current state of each Circuit Breaker that has been configured in the application.

Hopefully more updates will come in the not-too-distant future, but before I get too deep I want to make sure I know what would be useful to add. Until I use it more, or get feedback from others, it’s hard to say what would actually add value (and not just clutter). My short list of next potential updates are:

• Additional metrics to expose via JMX and the Controller
  • e.g. number of requests, average response time, number of good responses, number of errors, current number of concurrent requests, concurrent request high-water mark, etc
• Allow configuration of timeout & failureThreshold by exception type. From Release It!:
  

  For example, you may choose to have a lower threshold for “timeout calling remote system” failures than “connection refused” errors.

In addition to my own usage, I definitely appreciate any feedback that you might have if you have a need to use the plugin.

I just released the CircuitBreaker Plugin for Grails.

I recently read Michael Nygard’s book Release It, which is based on his experiences from the trenches in tracking down the causes of large system failures. He relates a number of his experiences and then discusses the patterns and antipatterns that he’s distilled after seeing the same causes of failure over and over. It changed how I think about certain aspects of designing and developing inter-related systems. I highly recommend the book. Though it doesn’t get into implementation details, it’s one of the most interesting (and applicable) technical books I’ve read in a while.

The two things that stick with me the most are both from the section on stability. The first is the effective use of timeouts - especially when integrating with other systems. This can be done in Grails/Groovy/Java/etc by using HttpClient from Apache commons.

The second thing that I can’t stop thinking about is the CircuitBreaker Pattern. Like an electrical circuit breaker in a house, the idea is to recognize a problem and fail first, thereby controlling the overall failure mode. When integrating your app with an external system, this means you stop calling it when it continues to fail over and over and over again.

From Release It!:

…the circuit breaker exists to allow one subsystem (an electrical circuit) to fail (excessive current draw, possibly from a short-circuit) without destroying the entire system (the house). Furthermore, once the danger has passed, the circuit breaker can be reset to restore full function to the system.

You can apply the same technique to software by wrapping dangerous operations with a component that can circumvent calls when the system is not healthy. This differs from retries, in that circuit breakers exist to prevent operations rather than reexecute them.

Whether these are calls to an external system, or even internal operations that could fail repeatedly and need time to recover, the Circuit Breaker pattern guards against cascading failures, so that failures in one area don’t bring down the entire application.

This led me down the path of looking into AOP in Grails because I wanted to reduce the impact on my application codebase. And for the very same reason, creating a Grails plugin seemed like the obvious way to implement it. Many thanks to Ken DeLong because his post Circuit Breaker in Java from last year was very helpful in getting the groundwork set for the plugin. In addition to the tests included in the plugin, I’ve created an example application to show how to use the plugin.

Check out the plugin and let me know what you think.

Success! After playing around with a few things, I got AOP working in Grails this weekend and I’ve checked in the changes to my example app. I’ve also added a test that more clearly shows aop in action.

I made a few changes inspired by Graeme’s comment on my earlier post on AOP in Grails. Graeme pointed me to BeanBuilderTests and both testSpringAOPSupport() and testUseSpringNamespaceAsMethod() were very helpful.

I also took a deeper look at the Spring docs on AOP - specifically around advice and advice params.

Looking at the changes I made, the first change clearly make some sense in retrospect. The second is a little confusing to me and I’m guessing that I’m doing something wrong.

1. In my Aspects, I switched from using the aopalliance classes to aspectj classes. This meant instead of using org.aopalliance.intercept.MethodInvocation & MethodInterceptor classes, I used to org.aspectj.lang.ProceedingJoinPoint. That makes sense and I was just wrong before (not sure where I saw examples of this).
2. The second set of changes I made were in the aop bean definition in resources.groovy.
   1. First, I needed to make sure that I was proxying the target class (proxy-target-class=true).
   2. Second, I had to define my execution point without explicit parameters. As in Service.method(..) vs. Service.method(Integer) && args(i) (which I used with 'arg-names':'i').

The last changes are what confuse me. I tried a number of different things, but could not get explicit parameters to work. Apparently, I need read through the Spring docs on advice params again.

One new wrinkle is that when running the app, my around advice seems to be invoked twice. I wrote the test (inspired by BeanBuilderTests), but I don’t see the same problem when executing the test. Maybe it still has something to do with the pointcut execution declaration. My quest continues….

I’m trying to get AOP working in Grails 1.1 and running into some problems. I’m sure I’m just doing something simple wrong, due to my lack of experience with Spring AOP. My problem is that I can’t seem to get ‘around advice’ working on a Grails Service. I get no error, but my aspect is not invoked.

To simplify everything, I created a Hello World app that does almost nothing: my controller calls my service and both print log messages. The service implements an interface as described in a couple posts to the grails user email list (the interface is defined in src/groovy). And my aspect just logs before and after the method invocation (the aspect is defined in grails-app/utils).

I’ve tried a number of different ways of configuring the aspect in resources.groovy (in grails-app/conf/spring), but nothing seems to change (except I do get errors on startup if I define the aspect incorrectly in resources.groovy). I’ve even tried defining my Service and interface in the default package and in their own package.

Again, since this is my first attempt to use Spring AOP, I figure that I must be missing something simple. I uploaded my hello world app to google code if anyone is interested in downloading and running the app.

Anyone with ideas where to look? The relevant bits of my app are included below as well.

Read the rest of this entry »

Before RSS most of us had a blogroll of sites we visited daily, weekly, etc. With RSS we could let those sites come to us. Now, more and more, those sites are more stagnant and the real action is in the micro-post spaces such as Twitter. Sure people like us, software developers and designers may still use RSS but for the masses, RSS has not caught on, nor will it ever.

As you may have noticed, things are quite around these parts recently but that isn’t due to a lack of things to say. We are busy at work getting Lean-to ready to come out of beta and have even started our second web application in between sweet* client gigs.

If you really want to hear what we are doing here are a couple ways to do that these days:

Follow Refactr on Twitter. Follow Lean-to on Twitter. Hell, you can even follow Ben, Jesse, and Scott from the Refactr team with varying degrees of satisfaction.

* And not just sweet because they are paying cash money, but also because we are really lucky to be able to work with some really cool companies doing really cool things that we hope we can tell everybody about soon!

Today is a big day for everyone interested in Groovy and Grails. Following quickly on the release of Groovy 1.6, today Grails 1.1 has been released. This is a major milestone for the framework and I can’t wait to get my hands dirty trying out the new features.

Happy? Sad? Disgusted?

Nat Pryce wrote a program to find out how developers feel about their projects. He extracts code comments and feeds them into Synesketch. Nat has even posted results from a number of different open source (and a couple nameless closed source) projects.

I find it really interesting. I wonder if it truly correlates to the health of the project or the morale of the project team.

Our labor of love, the simpler project management tool, Lean-to got some new features this week. Scott, Jesse, and I as well as the rest of our growing team are excited to announce several new additions (and some important subtractions) to our favorite web-based software product:

• You can now attach files (images, pdfs, docs) to stories and bugs

• Stories and Bugs now have unique numbers that can be used for reference
• There are now more ways you can view your stories/bugs: From the Plan or Backlog views click “View only this iteration”.
• Burn up report now can show backlog stories and bugs or just those on plan
• You can now search within a project (plan and backlog) for stories and bugs

We also fixed a number of things that annoyed us (and probably you):

• Iteration boxes on backlog now allow you to click through to just see items in that iteration
• The UI has been modified to emphasize the project name and options and clean up interface as well as to demphasized the Lean-to branding.
• We cleaned up several workflow and interface issues
• We cleaned up some character issues in form fields

While these are the highlights there is even more to explore. And, shhhhh, don’t tell anyone but we have more on the way very soon!

Please let us know your feedback and share your stories of how you are using Lean-to. Oh, and by the way, we left one bug in there. Let us know if you can find it.

Martin Fowler recently started considering the usefulness of tracking interest on a system’s technical debt. I think this is an interesting idea. But as he notes, it’s hard to estimate an objective and accurate measure of “interest” in this way - and it may end up only having limited value in the end anyway.

I’ve used the idea of technical debt for years and I like the concept. But there have been times where it has been difficult to tell exactly when is the right time to tackle the debt and settle up. On these teams, I’ve even had discussions about the impact that different technical debt items have had on new development, but usually that’s as far as it goes. That’s why I like the idea of tracking “interest” payments on technical debt.

At the very least, tracking this data seems like a good ongoing reminder of just how costly that debt actually is to the application and to the team.

I have been a big fan of the PNG image format for several years, or I should really say I am a fan of the alpha-transparency* you can achieve with the 24 and 32-bit versions of the format (you get the more GIF-like transparency with the 8-bit PNGs). Not only that but the use of PNG files as backgrounds for divs and such allow me to better cut up and position elements using CSS in semantic ways that allows for more flexibility in the future as well as each component is self-contained.

The thing is, aside from all the benefits, I could only use PNGs sparingly because I would feel guilty about the large file sizes that result. Until now that is! I am not sure when it was implemented but using Adobe Fireworks you can export 8-bit PNG files with alpha transparency! This is a huge file-size savings and in most cases there isn’t a noticeable degradation in image quality.

* Alpha-transparency is the ability to show multiple levels of transparency/opacity instead of the GIF format’s simple transparent or not, resulting in much better looking transparent transitions and less “choppiness” around the edges.