Continuous Integration – Single Code Line

A common practice in SCM is to create multiple branches (code lines) from a stable baseline, allow teams to work in isolation on these feature branches until they meet some quality gate. The feature branch can then be merged into the baseline to form a release. I find this approach abhorrent in almost all cases. My three main objections are:

1. Multiple active code lines force a conservative approach to design improvement (refactoring)

While there is more than one active code line most teams will defer any widespread design improvement, as any widespread change will be difficult to merge. This means that emergent design and refactoring do not occur, and the software will build further inconsistency and duplication. This effect must not be underestimated – effectively it’s another source of fear, preventing the teams from moving forward.

2. Deferring integration of code lines usually leads to high risk late in delivery

The longer an isolated code line lives, the more pain and risk incurred when merging. This risk can be largely mitigated if the teams are disciplined in regularly merging changes into the feature branches from baseline. However most teams I’ve observed aren’t very disciplined in this regard, and this risk becomes a real issue.

3. Multiple active code lines works against collective code ownership

Teams working in isolation on a separate code line share their work with other teams as late as possible. This leads to code ownership problems, and inconsistency. The code introduced by an isolated team is often quite clearly different to the rest of the codebase, and is disowned by other developers working on other branches.

Other issues with multiple code lines:

  • complexity can cause significant errors that may not be caught by automated or manual testing, risking production stability.
  • it is very difficult to consistently spread good technical practices (automated testing, coding standard)
  • it works against the CI principle of production-ready increments – isolated branches are often used as excuses to leave the software in a broken state for some period of time, instead of working out how to implement a major change incrementally.

But what if I’m working on a feature that isn’t going to be ready in time for the next release? Firstly, are there any smaller increments that we can release to production and get benefit earlier? If not, then we need to release partial work into production, without it changing the current behaviour of the production system until the feature is complete and can be activated. This involves the introduction of ‘feature toggles’ – configuration that disables the new feature implementation in production until it is ready.

This doesn’t have to be runtime configuration – simple switches introduced to environment-specific config files will usually be enough. There is a cost in introducing this conditional behaviour, but in my opinion this is far outweighed by the enablement of single code line and regular metronomic releases.

The approach is also more challenging when altering the behaviour of an existing feature – sometimes requiring significant refactoring to introduce the switch. Sometimes we need to introduce a whole abstraction to be able to switch implementations – this is an enabler for significant ‘architectural refactorings’. This is referred to by Paul Hammant as Branch by Abstraction – and is a very powerful technique.

Further reading:

Continuous Integration – If something hurts, do it more often

The prevailing attitude in software development still seems to be that if something is difficult or expensive (or even just not much fun), we try to do it as few times as possible.  This usually means deferring it until as late as possible.


  • merging and integrating the work of multiple people
  • merging and integrating the work of multiple teams
  • execution of tests
  • testing the integration of components
  • deploying into a production like environment
  • deploying into production

Most of these things are difficult and expensive, and the temptation is to make more rapid progress early in a project or release by deferring these things until late.  Unfortunately these things are also very difficult to predict – in complexity and effort.  This means that we often find that we have a significant amount of complexity late in a project or release, just as the pressure on the team starts to rise to deliver.  This inevitably leads to delays being announced very late in the delivery of a project or release, or the team to abandon quality.

I don’t want one day to be significantly harder or anxiety-inducing than any other.  I don’t like the ‘deployment day’ being a time which people dread, or merging the work of multiple teams to be an unloved task which is risky and error-prone.

My goal with CI is to do these ‘hard’ tasks as often as possible, to invent ways to make these things easier, and to keep this up until the ‘hard’ tasks are painless and risk-free.

Blogging and Continuous Integration

I find I have a high barrier to writing – getting started is hard, if I write few things then I get obsessive about the quality of my writing.  I need to get back on the horse – and write a few short topics that can get me flowing.

I’m going to write about old topics – things that I talk about and teach on a daily basis, and get down in this journal the views that I hold right now.  I’m going to start with a favourite topic – Continuous Integration.

I’m a big advocate of Continuous Integration, but I regularly find myself having to explain the wider concept.  Lots of folk I find think of CI as the server itself (Hudson, CruiseControl etc.) which performs automated build processes and runs your automated test suite.  That’s a useful tool to achieve Continuous Integration, but it certainly doesn’t stop there.  I think of CI more as a holistic approach – one in which we seek accelerated feedback at any opportunity.

I’ll drill down into a few topics here, unfortunately all of these topics are linked together in some way, so hopefully this makes sense…

Disclaimer: I’m writing these entries to distill some of the conversations I have with customers on a regular basis – I hope it’ll help with my thinking and expression and provide some feedback.  None of this is original thought – it’s based on a few years of reading and listening to smart people.  I probably don’t remember where I heard a lot of this, so I won’t do a good job of attribution.

Production releases – 0.875 per month

I’m coming up to four years with ThoughtWorks next month, and I’m in a reflective mood.  I did a quick count – since joining ThoughtWorks I’ve been involved in the delivery of forty-two (42) application releases into production (give or take one or two).  Some have been fun, some have been hard, some have been terrifying, others have been so regular and predictable that hardly anyone notices.

I’m sure that’s no great feat amongst my peers – but the reason it’s so astonishing to me is that in the five years prior to joining ThoughtWorks I had written lots of code (some great, some not so great), but not one line had ever made it to production.

Thank you ThoughtWorks.

Setter injection sucks

I know it’s not trendy to re-hash old java programming discussions – I should be discussing some new amazing functional language, however I’m in a grumpy mood. I have a beef with what appears to be the dominant trend of using setter-based dependency injection. I’m an old-skool dependency inject-er – I started with picocontainer when Spring was unheard of (well at least by me). What I particularly like was the way it changed the way I designed my java code. (For the spring-kiddies, PicoContainer *only* supported constructor injection)

A common complaint I hear about constructor injection is “I don’t like having so many constructor arguments”. Yup this crossed my mind when I first came across it too – then it dawned on me: perhaps having too many dependencies injected into a class is a code smell? How many other classes should this class be collaborating with? Am I missing an abstraction in my model? Many times I’ve looked at a class and simply by looking at it’s constructor signature I’ve had a nagging doubt about it’s hygiene.

A big reason I prefer constructor injection is that I can ensure that when an object is constructed is ready to do it’s job. I do not have to concern myself with whether the class has been ‘wired’ correctly in configuration and that all dependencies have been met. If I introduce a new dependency, I can find all places in the code (and pretty quickly in configuration) where I’ve caused a breakage.

However it’s now 2009 and I’m still (occasionally) writing java, but pretty regularly I’m in a team that has settled on setter injection as a standard. It’s the default option in the Rod Johnson scriptures I guess. I also believe in most cases it’s better to be consistent than argue what some see as a fine point, so again I fall into line and am forced to use setter injection. But I do want to bitch about it.

How about this test setup code:

        WidgetGetter widgetGetter = new WidgetGetter();
        widgetGetter.setHttpClient(new HttpClient());
        WeeResourceFactory resourceFactory = new WeeResourceFactory();
        resourceFactory.setSomeParameter("the quick brown fox");
        FooRepository repo = new FooRepository();
        repo.setBarFactory(new BarFactory());

versus this?:

        WidgetGetter widgetGetter = new WidgetGetter(new HttpClient());
        WeeResourceFactory resourceFactory = 
            new WeeResourceFactory(widgetGetter, "the quick brown fox");
        FooRepository repo = new FooRepository(resourceFactory, new BarFactory());

I’m also into configuration in code rather than in XML, but I worry I’ll get burned at the stake if I bring THAT up again.

Danger Will Robinson

I wrote my previous post about running linux on my desktop a couple of weeks back and realise I didn’t put a disclaimer on the post, so here it is…


What I didn’t mention in my post is that the first time I installed vmware and loaded up the windows partition it worked great.  Then I was tooling about in windows (I think uninstalling some cruft) and it required a reboot.  No worries, but then my attention drifted and I went off to another desktop on the linux host.  When I looked back a few minutes later I realised in great horror that the vm had automatically booted via GRUB into the ubuntu partition.  I was looking at the ubuntu login screen, in a vm running on the same ubuntu install.  Nasty feeling.  Power off button on the VM.

Too late – everything in the running host OS started to unravel and I quickly crashed and burned.  Someone in the audience will probably pipe up at this point and say “you could have recovered by xyz” but after a couple of people had a poke around with no success I took the quick route and reinstalled ubuntu again.

Not recommended…

Luckily the windows partition was untouched.

Linux at work again!

For the past couple of months I’ve been working in a very nice workplace with a bunch of nice people and even better some of them run Linux on their desktops without any fear of the SOE-police coming to march them out of the building! I’m also working again with a chap who despite an unnatural love of emacs is a great help at solving any problems I face running Linux at work. So off I set to install Ubuntu Hardy on the client desktop – with their blessing!

I love Linux (and particularly Ubuntu) as a working environment. Most of the applications I’ve worked on in the last 10 years have been deployed on a flavour of unix, and despite doing a lot of Java I also do a lot of scripting and glue code, build and deployment tooling in particular. Working on Windows even with cygwin is just a world of pain.

Unfortunately I’m not able to avoid Microsoft Outlook *sigh*. I spend more time in meetings than I care to admit, and I regularly have to set up meetings with invitees and meeting room resources. Webmail doesn’t cut it. Evolution really doesn’t cut it for much more than reading mail – even sending mail it’s a bit flakey. Great effort, I really hope one day someone gets involved who can make it stable.

So what I and some others are doing is running VMWare Server under Ubuntu, booting the physical Windows partition from disk. When I installed Ubuntu I resized the existing single partition and left the client SOE Windows installation fully functional. It works incredibly well booted under VMWare once everything is loaded – we run Outlook 2007 which is slow as a dog even non-virtualized – and I’m never going to look back. The best part is if I ever NEED to I can boot the same partition directly.

It’s a little tricky getting VMWare Server to run on Ubuntu, as you have to apply a patch (vmware-any-any) to make it install. I followed the instructions here on howtoforge, and they worked on Hardy. I found I had to run twice – once when the patch is applied, and then once more to make it work properly.

Once VMWare is installed it is important to boot under Windows directly, and set up a new hardware profile for “virtual boot”. When you boot Windows under VMWare the first time, choose the virtual boot profile. It will detect a bunch of new VMWare hardware, which will then be associated with the virtual boot profile. This means when you boot directly for some reason later, you can choose the default boot profile and everything will work as it always did.

I’m stoked at how well this works. I have VMWare tools installed so that it gives focus to Windows when I move my mouse over the VMWare window (no Ctrl-Alt-Esc). Highly recommended.


Whenever I think I’ll write a blog post, I get as far as logging into my wordpress admin and it’s always the case that a new version of wordpress has been released and it nags me to upgrade.  Ooh I think and head off down a nice little alleyway of distraction.  I usually take a few minutes to remember my username/password for my hosting provider, upload and unpack files yada yada.  That normally takes care of my urge to write a blog post…

I’m frighteningly similar when it comes to writing code on my desktop machine at home – of course there are 227 incoming critical updates from ubuntu.  Or maybe I need to upgrade from Hefty to Iggy (or whatever).  Of course my Eclipse is 3 versions out of date, and I couldn’t possibly use that Java version (or ruby or pascal, or whatever it is I use these days).  By the time I get everything up to date I’ve usually forgotten what it was I came upstairs to work on anyway.

They’re my habits and I love them.

BarCamping (without the actual camping part)

I attended my first BarCamp today and I was really impressed. It was a very full day of interesting presentations by a lot of very smart people, the range of topics was really quite startling. I’m familiar with user groups (although I don’t go along to as many nights as I’d like) and they tend to be quite focussed, however today people talked about whatever their current interest is – from Perl 5.1 to hardware devices to MythTV. I was particularly impressed with the support and encouragement attendees gave to speakers. If you’ve never given a public presentation and you’re unsure of yourself today would have been an excellent opportunity.

Like at user group nights and conferences I’m awkwardly shy and awful at striking conversation so I missed talking to a lot of interesting people today. It’s a different mix of people than I’m used to (like the Ruby user group) with a slant towards freelancing and smaller projects which is so refreshing compared to the behemoth corporate beasts I’m more familiar with these days.

Favorite presentations today: Mark Ryall’s Intro to Scala (second time around), and Paul Fenwick on an Illustrated History of Failure (with sound effects).

I winged a short demo of the Rspec Story Framework which is my current shiny-toy-of-the-week. I might have been better placed to answer questions if I’d actually used it in the wild yet…

Don’t miss BarCamp Melbourne 2009.


Bob Martin:

At the ER they put me in a room and gave me a dose of Morphine. Morphine is a very nice drug. It had the effect of filing the pain away in a convenient subdirectory where I could access it if I needed it, but was otherwise out of the way.

Brilliant description. I recently had a healthy diet of Morphine for a couple of days (followed by a few more days of oxynorm, which is similar). After my first dose of Morphine the emergency doctor came to see me and I couldn’t help grinning like a loony, despite the horror of my injury. Good stuff when you need it.

// TODO: think of a witty and intelligent tagline