The majority of organisations I’ve worked with deliver new system functionality as development projects. These are funded with capex, and have a start and an end. Even projects that are ‘agile’ are still expected to finish at some date in the future, then once the system has been delivered it will undergo ‘handover’ to ‘BAU’. The project team usually moves on to new projects, developing remarkable cases of mass-amnesia along the way.
Projects deliver exactly what they promise. Project teams have little incentive to invest in the long term operation and maintenance of the systems that they create. I’m not saying that the team doesn’t care or are intentionally acting irresponsibly, but when delivery pressure is applied the first things to be dropped from the project schedule will be the cross-functional concerns that make the system reliable, monitorable, deployable, and maintainable ongoing.
The project effect:
- the project team do not have to live with the long term results of their own architectural and design decisions.
- BAU support/maintenance teams are generally under-resourced, have extremely limited opportunity for handover from project teams, and have to support many different systems. This usually leads to less than ideal development practices and deteriorating quality over time.
- the project team never have to be involved in problem analysis for production outages. They’re never forced to put the right kind of monitoring and logging in place to find root causes.
- the project team only do a limited number of releases to production, so have little incentive to invest in reliable automation or production-like test environments.
Therefore – I believe that many projects are the source of ‘instant legacy’, and a major cause of the development and operations divide.
What’s the alternative? Form long-lived teams around applications/products, or sets of features. A team works from a prioritised backlog of work that contains a mix of larger initiatives, minor enhancements, or BAU-style bug fixes and maintenance. Second-level support should be handled by people in the product team. Everyone in the team should work with common process and a clear understanding of technical design and business vision.
This approach is not easy – it introduces new challenges particularly around balancing priorities and budgeting. I’ve observed that the benefits in terms of long term system health definitely outweigh the drawbacks. Like everything – hire good people who care, and give them the right incentives, good things will happen.
Spot on Evan!!! 🙂
This post is just perfect.. Thinking about root cause analysis, this is really one of the biggest reasons why people behave the way they do on “projects”…
We have been talking about these concepts on our current engagement.
Merging as much as possible BAU and Projects… Eventually, they will be only one.
Very well put 😉
Cheers
Terrific post. I’ve referenced it in my own post on Structure Agile for Product Teams.
Keen to read more about your recommendations on “balancing priorities and budgeting” for product teams.
Is this complicated by the fact that some of the work will be funded by capex and some will be funded by opex? Might a develoepr not work on stuff in 1 iteration that has to be attributed to different cost centers? That’s not too big a deal but are there other ramifications to a single team taking on both operational and feature based work? I’ve worked on teams like this before where someone in the team had to wear a pager at all times (rotating) and respond to unresolved L2 tech support calls. While we did realize some of the benefits you describe, it was also very difficult to poredictably complete feature work due the the unpredictable and bursty nature of the support work. We simply worked to make this labor distribution very visible to all stakeholders BUT when we did that people started clamoring for a seperate team to handle all the operational work. Square.One!
I can identify with all of it. Our latest endeavour has been to do exactly as you’ve suggested, but like Gregory Smith says, our ‘dedicated’ (rotating) support person regularly gets sucked into support problems, and if they need help from other team members, they also get sucked into the vortex, causing real impact on other progress.
I’ve used this as customer bait to get them to prioritise unsexy, but supportability focused stories/requirements in the hope that it eventually makes development team progress more focused, with the added benefit that we can hand over a more simple system to a ‘BAU’ team.
I am so surprised this article has not received so much more attention. For many IT organisations this is indeed the fundamental root cause of a lot of their problems in IT (costs to high, projects over budget, projects exceeding deadlines, no productivity, no delivery of quality software and much more). I have the strong feeling management heavily underestimates the way a project is financed impacts how people behave (incentives) and thus the results.
Yes! I’ve seen (and was horrified by) the effects of designating “break/fix” teams in an organization that professed to be Agile and some regards actually was.
Since everything else about the way Agile methods are constructed is about breaking down the silos, i.e. no longer tossing a work product over-the-fence to the next group of “experts”, it’s astonishing to see ops still treated as a silo, both from a budgeting and an operational perspective. Refusing to let go of “project” based development teams and creating break/fix teams to mop-up after them is filled with so many wrong-headed incentives.
If your organization cannot be convinced that the ongoing maintenance cost is a direct result of the cost of product development, then the next-best solution is a hybrid: require rotation of members from the pool of resources used to man project teams through the break/fix teams. At least with a couple of rotations through a break/fix team individuals will experience first-hand which practices to avoid and/or pay more attention to, and the true cost of failure to do so.
Shades of FlowChain and http://smsexemplar.com/knowledge/whats-wrong-with-the-project-approach-to-software-development/
So why is this idea re-invented time and again with no impact? (Not a rhetorical question).
– Bob
I agree with the points raised about the project effect, and the alternative approach sounds more congruent with the ethos of agile development. My concern is that unless the team work exclusively on the interests of a single product owner then the often political struggle for budget and priority that is usually played out at a to get a project started and finished will just move down to the team level, where the team members may be poorly positioned to deal with it. I would be interested to hear about teams that have managed to work in this manner whilst working with multiple product owners.
We’ve had some success by backfilling support staff out of their BAU activities to become part of the project team, so they can take some ownership in the implementation, transition to BAU, along with any knowledge transfer and support documentation.
Yes @flowchainsensei, this will keep coming up (orig post was from 2010). In @kentbeck’s Software G Forces: The Effects of Acceleration he mentions folding design and testing into the development work to facilitate more frequent releases.
To facilitate even more frequent releases, he then talks about folding-in the deployment or release management to get to continuos, or near continuos, deployment. What’s funny is that one of the results is that this requires pretty much error free automated deployment and the immunity of fully automated roll-backs. So getting to these levels of release frequency makes the need for “break/fix” teams, or other separate ops support teams, obsolete. It folds in and eliminates the “DevOps” divide.
It not easy, and some of the required changes go way beyond the development teams (e.g. subscription funding model vs. paid “releases” or “updates”), but the payoffs are huge. It’s tough to see how moving to daily releases would not rightshift.
They won’t die because the people who push for the “project-based” approach are wedded to other things such as count-accounting and methods that really don’t apply in the knowledge work space. We’re years away from seeing a transition to something else unfortunately.
Getting rid of projects is not necessarily a panacea, however, Even if the team remains responsible for long-term maintenance, team members come and go, and areas of responsibility expand. You have to guard against the tendency for projects to arise organically: eventually James becomes responsible for the entire job-scheduling subsystem, and when he quits and Parul replaces him, she throws up her hands and says. “This needs a complete rewrite! I’ll have to do a project!”
I’ve been on both sides of this issue and I’ve argued both sides as well. (Hey I’m a Libra, I’m up for a good debate anyday). I’ve often said it depends on the culture of the company. While I do see the value in the team who did the work to support the work, I wonder if that’s too idealistic at times. It would in fact force the product owner to prioritize better, which is a great thing. BUT, if there is a critical piece of work that teams needs to do and a host of support items come up, you are in a rut, especially if the release date is fast approaching or can’t be moved (but then again you get to the prioritization issue). Lately, I’ve been leaning to the hybrid approach that was mentioned above..in fact I truly love that idea. The one issue though is when you have a large number of applications to support and maintain, it then takes so much time to get devs up to speed on the ones they don’t know.
If the development team has done a good job, surely the number of people needed to maintain the system, once it is running, is often much smaller than the number of people needed to create it (unless you create it rather slowly)? So how does one sustain this model over a long period? I can see it working just fine for software that needs constant evolution (so the initial pace of development never really slows much), but what about systems that the customer is happy with and just wants to be kept running as-is?
Spot on Evan.
I’m primarily a developer in an institution. A lot of times, top management don’t understand the need for everyone to “Calm Down”. There’s always so much pressure that I know I sometimes deploy “shaky” code.
Some of my managers understand the need to expand the team but management is wary of staff costs.. Hmm..
Having worked in some projects as a support contractor, the long-term concerns really are apparent to me.
Projects delivered are generally documented as deeds done, but without looking to the future. Building a long term team around a solution is not often possible, specifically when engaging external expertise to implement it…
Perhpas a middle ground would be adding as a project-level requirement the delivery of documentation and code/procedure examples of integration of the solution with others, and documented and demonstrable upgrade procedures that can be followed — essentially building the forward-lookingness into the project, with working examples from the get go?
BAU = Business As Usual ?