Sometime ago, in a post entitled The Destination Not the Route, I shared some of my thoughts about Agile development. Since then, a number of people have come up to me or written to me asking for a similar sideways look at the time dimension, particularly in the context of Agile versus Waterfall development.
Agile development is often about time-boxing or time-placing, the two techniques are sisters under the skin. Since time-boxing involves splitting projects into fairly small chunks of time (keeping the time fixed, the team small and varying aspects of the deliverables), people often consider Agile as synonymous with Short-Term.
If this wasn’t bad enough, the same people often compound the error by then considering Waterfall as synonymous with Long-Term.
This is a fundamental fallacy. Let me try and explain, using an everyday nothing-to-do-with-software analogy.
Let’s assume you’re married, with two kids of schoolgoing age, and that as a family you had a yearning to go away somewhere hot for Christmas, just for once. Let’s say you live in the UK, and that you decided to go to South Africa for Christmas in 2008.
South Africa. Christmas. 2008. Long-term enough for you?
That’s the headline plan. When it comes to preparing for it, you have a choice. You can do it the Agile way, or you can do it the Waterfall way.
The Waterfall Way:
- You check the flights and hotels out straightaway, select some hotel in Cape Town, book flights direct from Heathrow to Cape Town, put down a deposit in sterling, forward-buy the rand you think you may need for the holiday, get the necessary visas (if you’re Indian, like me, you need visas to go to the loo!), sit back and think, great. You’ve done everything ahead of time, you’ve saved money, all you need to do is wait to go on that holiday.
- And throughout 2007, you’re pretty relaxed about things. One day, while travelling elsewhere, you realise that the visas you thought you had were no longer valid, that they’d expired 30 days after you got them. This irks you, but not much. You can always get new ones closer to the time. But you wish you’d known. Then the rand weakens against sterling, and another set of niggles set in. Should you have done the forward trade? But you’re not a foreign exchange specialist, the sums involved aren’t large, so you remain relaxed.
- Then, in January 2008, you get the dates for school closures. Your children are missing more school than you intended them to miss. Oops. You contact the travel agency. They try and help, but the tour operator you chose went bust, and the guys who took them on are offering flights to Jo’burg in compensation. In panic you check out the hotel you thought you booked, only to find it’s been partially demolished and that work will continue on the wing you planned to stay in, for the foreseeable future.
- I won’t go on. Suffice it to say that you have to do things more than once, that you have to change things, throw away things, spend and waste more energy than is sensible. And at the end of it you’re left feeling frustrated, dissatisfied and angry.
The Agile Way:
- You plan out what you need to do, but in sensible time-chunks. You do what you must do right now, and nothing more. And you keep adhering to that principle all the way to December 2008. Doing what you must do at the time, and nothing more. No wastage. No rework. An inherent flexibility in the way you exploit your resources, allowing you to change your mind as better information emerges, with minimal cost.
I’m not going to labour the point, but it is important. So here’s the takeaway:
1. Using Agile, you feel insecure (-ish, especially if you’ve not tried it before) at the start of a project, and satisfied at the end. Using Waterfall you feel good at the start and crap at the end. If there is an end.
2. Agile is about a way of doing things, and it does use time-boxing as part of the technique. Waterfall is also about a way of doing things, and it doesn’t use time-boxing as part of the technique. Most projects use Waterfall. Most projects fail.
3. Agile does not mean short-term, in fact it is the opposite. It creates sustainable incremental value. Waterfall, on the other hand, seems to be long-term, but is actually about the worst form of lock-in. A design lock-in before you have enough information, just to prevent you from having kittens right now. Is there anything more short-term than that?
4. I used to be more tolerant in the past, and used to accept that Agile and Waterfall are different horses for different courses. I now realise that’s a lie, an invidious one. Why? Because I accepted the theory that some projects were large, didn’t change much, could be planned in real detail in advance, had everything known about them, and didn’t need Agile. Now I have a better word for such projects. Failures. Expensive ones.
5. A coda. Projects like EMU and Y2K and Basle II and Sarbanes Oxley were called Death March projects because they were long and expensive and unnecessary and ineffably boring and yet had to be done and required immensely large quantities of resources. [Well if you will believe what consultants tell you, and abdicate responsibility altogether, you deserve everything you get!]
Yet we never heard of any failures for projects like these. Why? Weren’t they Waterfall?
No. These oil tankers were actually Agile. Why?
- (a) Nobody could change the end date, a key criterion of Agile Not Waterfall.
- (b) As a result, people traded on the deliverables and not the time. Another key Agile criterion.
- (c) These were “compliance” projects, but with one major flaw. Nobody, not even the consultants who kicked up the storm, actually knew what to comply with. There was a long period of amorphousness while people worked out what the meaninglessness meant. This was true Discovery, another Agile criterion.
I could go on, but won’t. Enough said.
Thanks for the post JP. I did work on some Y2K projects masquerading as SAP re-engineering projects and you’re right, they succeeded much better than similar scale projects that didn’t have such a dead line in place. So looking back now, yes, perhaps they weren’t as waterfall as we thought at the time.
Where I am now I like to leave as much of the specification as possible up to the people doing the actual development. Often the prototypes don’t do what we expected but it’s faster, cheaper and easier to change than the alternative which is to try and dream up a firm set of theoretical requirements. It’s also more fun for everyone involved.
Having said all this, do you think there is a practical limit to the team size that can realistically deliver a time-boxed project while still remaining, err, agile?
Thanks Alan. My heart says the maximum team size is Thé Magical Number Seven, give or take a few.
Small enough to have a meal together. Large enough to change the world like apostles.
Is there a practical maximum? I don’t know. But I’ve never seen great stuff come from large teams. Not unless they are really small pieces loosely joined, masquerading as a large team.
Even in the Opensource model. The hardcore centre is probably a dozen or less.
Using Agile in an offshore development always poses challenges. I have seen a number of projects fail (or rather being blamed on Agile) when one tries to use this using different time zones. What’s the magic bullet to make this work in a distributed scenario?
Ranjit Narula Says:
As usual, there’s no magic bullet. However, a good start is to make the decision to either outsource/offshore a project, or not. At least then (if you do), the offshore team can work in an agile way, and only have to solve the issue of distance from the customer.
Where the problems start IMO is when you have management onshore and implementation off, or (possibly even worse) design on and development off. If they’re serious about agile, IT managers in the UK (and presumably the US etc too) need to realise that they can’t treat ‘coding’ as a low-value acivity that can be shipped off to the lowest-cost supplier once they’ve ‘done the design’.
“Where the problems start IMO is when you have management onshore and implementation off, or (possibly even worse) design on and development off. If they’re serious about agile, IT managers in the UK (and presumably the US etc too) need to realize that they can’t treat ‘coding’ as a low-value activity that can be shipped off to the lowest-cost supplier once they’ve ‘done the design’.”
I could not agree more. The traditional attitude to off shoring has definitely been that you can throw a design over the water and expect a solution to come back. I guess this comes down to two things;
1. Economics: it doesn’t matter if it takes 40 people off shore to do what 10 developers on shore can do, they are cheap; it is cost not results that count.
2. Ignorance: I’ve told them what to build, it’s easy. Successful software development cannot be achieved by throwing things over the wall and saying I’ve done my bit; but that is what happens in many cases.
There is no Silver Bullet to off shoring Software Development. Off shoring faces all of the same challenges as doing development locally. The major issue being that the Communication Challenge is exacerbated. If you are good at communication you will succeed, if you are poor at communication you will fail; it is that simple.
Unfortunately, there is a tiny percentage of organisations who are comfortable with an Agile approach. It relies on significant trust in the ability of the Project Managers to deliver, and an inability to perform is usually blamed on ‘poor organsational skills’.
At Timeboxing.com you may find related articles and resource information. I hope it will become a valuable resource for
many on how to implement, into their own lives, some of the methods used by the clear-thinkers of our modern age.
Good article. Waterfall gives people the feeling of certainty and comfort, yet as you point out it is a false sense of security. Conversely Agile can seem vague and unplanned because it offers little certainty early on. Here’s a picture of what I’m getting at:
http://www.dancingmango.com/blog/2006/10/16/getting-attached-to-stories/
I read a very good statement about Agile the other day. In Agile the ‘plan’ reduces importance and ‘planning’ moves to the top of the list . Well put I thought
As Marx said, when history repeats itself, what is tragedy the first time around is farce the second. How else can we explain changing the names around as a lip-service excuse for ignoring the practices? Back in the Seventies, Peter Keen knew that you could only implement a Decision Support System successfully if the implementation process entailed an ONGOING CONVERSATION with the client with ongoing sanity checks over appropriate time-chunks. (Hey, JP, see how long folks have been trying to flog the virtue of conversation?)
I think that Ranjit has hit on the fundamental truth that explains the problem. The IT world is a world of nouns and noun phrases, because that is the sort of stuff that the technology can process. Unfortunately, “planning” is a verb; so we try to dress it up as a noun and call it a “plan.” However, the “costume change” (in the spirit of my dramatistic thinking) takes away necessary verb-like qualities involving not only the actions themselves but the subtle ways in which issues of tense and mood can impact making the right decisions in real-time. I suspect that we would all learn a lot more from reading a good grammar book than from reading more about Agile!
are there any examples of cases where agile methodologies have failed? im writing a paper and need both sucess stories, as well as a failure story. any help would be greatly appreciated.
This is great! I think reading this, I loved every word. Seriously, keep posting the good information, bloggers like myself need it.