Daniel Harrison's Personal Blog

Personal blog for daniel harrison

CruiseControl Custom Listener October 16, 2007

Filed under: development — danielharrison @ 12:06 am

The first three points of the joel test, if you’re not familiar with it, basically ensures that your software is always in a known state. You always have a build you can throw to testers, internal users or sales and it means fixing a critical bug is about the bug, and not the build. If you’ve ever been on a project where you have a weekly or monthly integration task where builds and releases ‘come together’ you’ll understand the value of release early and often.

One of the tools I introduced where I work was cruisecontrol. As a continuous integration tool, it means our software is always ready to go. Basically the way we have it setup, clicking build can build all our java and c# source, run unit tests, version and package into all the various installers and publish to the releases directory for testing. The problem at the moment is the last mile. Our UI tests for the winforms components are run in Mercury, sorry now, HP Quick Test Pro which lives out of the continuous integration tool and are run manually. This is a pretty obvious sore point so I’ve been working with the testing team to close the circle.

The last step in our build automation plan is to automate the UI testing into our nightly and release builds. We basically have build machines for each product, and each stream for each product and then another beefy test box which runs all the Mercury tests. This distributed environment means that we needed the test cruisecontrol to be able to watch the releases share to pick up new builds. Because we have multiple products publishing into this directory the filesystem modificationset listener isn’t specific enough as it won’t pick up a single product suite. So to resolve this, today I wrote a custom modification set listener. All up the process was pretty straight forward, and in the end including unit tests and the automated install script for building and configuring build machine, it took about an hour and a half.

So basically it’s the filesystem listener but now takes a base bath and then a regex to watch for new directories being created in that base directory. Whenever we get a new root directory we know that we’ve had a published release and we should run the UI tests. The most confusing thing when writing the new plugin is that technically it appears that the all the modification system classes implement a sourcecontrol interface, which at least semantically, seemed misleading.

All up I think it’s an extremely solid product and the cost of getting the source, figuring it out and getting a new plugin running was simple enough. The only downside I think is that plugins, despite being relatively simple are a bit of a pain to install and run. My current thought was that if we could supply a new plugin or customise via a scripting language it would be considerably simpler rather than deploying a new jar into cruisecontrol and the associated configuration in the config.


4 Responses to “CruiseControl Custom Listener”

  1. Thanks for information.
    many interesting things

  2. Awesome post! A couple questions for ya if you have the time..
    1. What happens once your custom filesystem modificationset detects that a new build has been published? And how do you respond to a notification? (ie, once you know that a new build has been published, how do you then call your remote UI tests? Do you have code in your plugin that calls an external script to do this?)

    2. What happens when you have a build failure? My guess is that the filesystem modificationset will think there’s a new build available and then start the UI test execution process. How do you tell the filesystem modificationset plugin not to execute your FVT tests if the build has failed?

    3. How do you report the results of your UI test cases and do these results make their way back to CruiseControl?

    – Thank you kindly

  3. danielh Says:

    Thanks Sargon,
    For 1) Yes, we have a nant build script that runs an external script (vbs) (invokes HP winrunner), basically runs installers and runs regression tests. Results are published from winrunner and managed via quality center. If winrunner fails the script fails and fails the cruisecontrol project which of course emails everyone.
    For 2) The ui tests are on a separate cruise control instance (the one that has the custom modification listener) (separate vm in ESX server), the build, packaging and unit tests happens on a different build machine (it has much more fine grained component projects and an overarching ‘publish’ and ‘nightly-build’). So a publish is really a complete rebuild after doing a fresh get and cleaning (Installshield leaves cruft around) all it then does is (assuming it’s successful) copy the built product to a shared build drive on our data store. So if the build fails any earlier due to say unit tests failing it won’t get to the installers and the ‘publish’. So in that case it won’t kick off the ui test on that instance as nothing makes it onto the shared build publish location. So the ui test machine basically watches for a new directory coming in the form of \\share\builds\product\major.minor.patch.build. The whole reason I had to write the custom listener is that the filesystem watcher would be pointed at say \\share\builds\product\ but because often you end up with small file changes when people are installing, using, would trigger too much, so basically we needed a way to ensure that only fresh builds were being picked up.
    For 3) Results don’t really make their way to cruise control (apart from a fundamental fail, and standard nant execution output), they could, we’ve got another custom tool that we’ve added in as a separate log consumer, but the results are reported to the testing team who back it up with manual testing. So basically build and unit tests failures get reported to entire dev team, ui CC gets reported to test team + lead dev and ui scripting/fails is managed and reported via the testing management chain. Eg a unit test failing in the dev build wouldn’t trigger an incident report, but a regression ui test would get flagged and logged as a bug.

    Hope this answers your questions, cheers.

  4. Thanks for the quick response. At first glance of your reply, I was a bit confused as to why you had two cruise control instances and which cruisecontrol instance had the custom modification listener. I reread the reply and it makes sense. Thanks for the details. I’ll definitely implement something similar for our group upon build completion. We have an fvt framework (it sits on top of HTTPUnit) and I’ll have a custom modification listener that waits for a newly published build and then spawns off the test execution script. I’ll muck around with seeing what I can do with results and see if I can customize the CC dashboard to include these results.
    Thank you again! very very helpful

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s