John Dodds, Lars Plougmann and Stephen Smoliar (sorry Stephen, haven’t found a simple way to link to your blog) were kind enough to share their views with me following my earlier post on the subject. There I quoted from Richard Feynman’sRogers Commission looking into the Challenger disaster. submission to the

The comments touched on three aspects of the communication problem. John felt that we need to focus on “accentuating the negative” in order to avoid everyone building castles of sand in the air, suggesting that both “clients” as well as “IT people” have a problem with understanding what is to be built and what can be built. Lars inferred that project managers wilt under pressure and agree to unachievable timescales. Stephen felt that the willingness for self-deceit came from a postmodern “withdrawal into fictions”.

As with many of these things, I have some sympathy with all three views. I posted what I posted primarily to learn more about the issue. Let me try and take the argument further, quoting again from Feynman’s submission:

Official management, on the other hand, claims to believe the
probability of failure is a thousand times less. One reason for this
may be an attempt to assure the government of NASA perfection and
success in order to ensure the supply of funds. The other may be that
they sincerely believed it to be true, demonstrating an almost
incredible lack of communication between themselves and their working
engineers.

In my previous post, I was trying to understand the root cause of the deviance between the “manager” view and the “engineer” view, why and where the management self-deceit was happening. My gut feel is that it has to do with biases we bring in, biases related to incentives, measurement, even the tools we use for managing projects. Let me try and explain.

When John raised the question of accentuating the negative, he was trying to ensure that people build things that can be built, that solve a real problem, that allow value to be delivered in small measurable increments. Of course I agree with all this. But I have a nagging worry that we have tried this and failed. When we accentuate the negative, quite often we seek to solve yesterday’s problem. Sometimes that is the right thing to do. Too often it is “paving the cowpaths”. Even if it makes me sound more idealistic than I want to sound, I feel driven towards avoiding this accentuation of the negative. Yes we should build things that can be built. Yes we should try and deliver value in small increments. But not necessarily by focusing on the negatives.

Similarly, of course I agree with with what Lars says. There is no dearth of project managers accepting unachievable deadlines as a result of some external pressure. But then there is also no dearth of project managers failing to deliver without any such pressure…. So where does it go wrong? How does the management view start deviating so much from the engineer view? This is not just about schedules, it’s about functionality and performance as well. When the rot sets in, it affects many things. And maybe Stephen’s right, but then I have to give up hope for a better way of doing things, and this I am not yet prepared to do.

In a previous post I have already spoken of waterfall/cascade development styles versus agile styles, and accepted that there are horses for courses rather than just one sledgehammer way. So maybe I need to understand all this within the context of the two styles.

In a waterfall style project, most of what we need to know is known from the start. At least that’s what the pro-waterfallers argue. In which case bad estimates are primarily due to bad estimation, either by the engineers themselves or by those who “edit” the engineering view en route to management. Bad execution is due to bad planning, bad quality is due to bad testing, and so on and so forth. Waterfalls are good when there is nothing to discover, nothing to learn, so we have to look for the faults elsewhere. They are not in the definition of the problem, but caused by corruptions to the original data relating to effort and timeline and functionality. Or corruptions to the resources applied, in terms of availability and skillset. Or corruptions to the processes applied, in terms of quality of test cases, the testing process itself, and often the quality of end-to-end-testing.

In an agile project the story is different. There is much that is unknown, and there is a considerable risk that corruption can occur between the engineer and management views. Engineers make best guesses based on the information they have, and quite often management, without a deep understanding of the “guess” aspects, convert them into hard deadlines and functionality and cost. And then we have a period of House-Of-Cards-meets-Emperor’s-New-Clothes, until we have a Yeatsian Second Coming:

Things fall apart; the centre cannot hold;

Mere anarchy is loosed upon the world,

The blood-dimmed tide is loosed, and everywhere

The ceremony of innocence is drowned;

The best lack all conviction, while the worst

Are full of passionate intensity.

When this happens (and it happens often enough) we have a traditional wailing and gnashing of teeth, and blame cultures revel in firing some and promoting others. Such is life. But rarely is all this done as sins of commission, as malevolent acts. More often than not, this road to hell, like most others, is paved with good intentions. I think Feynman’s right on both his points quoted above. Often there is “an incredible lack of communication between [management] and their working engineers”. Often this is to “assure perfection and success …[…]… to ensure the supply of funds”. And this is where I think the incentive bias, the measurement bias and the corruptions caused by the project management tools come in. And I think we do have new ways of working, new tools to use, that can get around these biases.

To paraphrase Einstein, engineers don’t fail, they find ways that don’t work. And then they find ways that do work. In bits. And then they have to test the whole thing, end-to-end, to ensure that the bits work together, to create the experience that the customer wants to have and own. The opensource community can teach us a lot about testing the bits and testing the end-to-end. Given enough eyeballs the bugs are shallow. Web 2.0 companies can teach us a lot about ensuring that the bits work together to provide the customer with the requisite experience. When they say Beta they mean “It works, but we’re still ironing stuff out”. They don’t ship until it works, and when they ship, they listen to customers who do the real end-to-end tests, who exercise the code and at the same time suggest ways to improve the overall experience. These approaches, covering community development and real customer functionally-integral Beta releases, they’re often intertwined. And they tend to rely on social software to improve communications throughout the process.

Success in project delivery takes place, as Kathy Sierra suggested, when the customer says “My software isn’t working” rather than “Your software isn’t working”. This emotional shift, when the customer takes ownership of the output, is absolutely critical. And it doesn’t take place unless there is something for the customer to experience in totality. Nobody takes a car to a garage and says “Your car isn’t working”. We have to let the customer experience the product as quickly as possible, so that we can pick up the bugs and deficiencies that cloud that experience. This requires agile mindsets, community development approaches, a release-quick-and-release-often schedule, good social-software based communications processes, and a willingness to ship Beta.

But all this is difficult to describe, plan for, estimate and monitor using traditional tools and methodologies. Agile development is likely to become more common as we continue to deal with uncertainties. Unless we solve this, we are likely to see increasing divergence between engineering views and those of management, as waterfall approaches get used to govern agile practices.

Having worked in large organisations for most of my adult life, I have regularly been bemused by what happens in project estimation and reporting; how a series of intermediaries go through some sort of Chinese Whispers processes until there is an unsustainable deviance between what is reported and what is real. Quite often, that which is reported becomes that which is perceived as real, and we all know that perception is reality.

There is an incredibly large body of literature to do with projects, their management and their regular failures to meet cost, time and quality objectives. Most of the time, the “blame”, for want of a better word, is laid squarely at the feet of IT as a profession. We’ve tried many things in our varied attempts to “solve” this, focusing on the requirements gathering process via time-boxing and fast iteration, improving estimation processes using a plethora of tools, seeking to simplify coordination failures by disaggregating work packages and keeping project teams small, using enterprise bus architectures with reusable common components in order to simplify the design process, having regular design and code walkthroughs, sophisticated post-implementation reviews, what-have-you.

And yet the cases of sustained success are rare.

More recently, I have been looking at the psychological aspects of all this, both from an individual viewpoint as well as from an organisational one. Is there a group-think problem? Do I (and people like me) fall into a trap of self-deceit? And if so, where does this self-deceit enter the process? Unless you recognise a problem you cannot fix it.

It is with all this in mind that I chanced upon Richard Feynman’s submission to the Rogers Commission on the reliability of the Shuttle. Tragic as the occurrence was, we are all honour and duty bound to learn from it so that we can avoid repeating it.

You can find the full text of the submission here. I think his last sentence says it all:

For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled.

More to follow; let me hear what you think.

Some of you may have read my post yesterday on Hugh De Pree’s Business As Unusual, the book charting the principles by which Herman Miller became the venerated company it is.

Hugh records four principles enshrined by the founders of the firm: Trust. Stewardship. Equity. Innovation. [I guess he could have held the record for the shortest management book in history by publishing it as a four-word book. Could have been reminiscent of Ogden Nash‘s poem On The Antiquity Of Microbes: “Adam. Had ’em.” And yes, despite Wikipedia, I’m sticking to Nash as the author.]

Much has been written about trust, about equity and about innovation. Stewardship rarely gets a look in, and for that topic alone the book is worth reading. In fact you don’t need an excuse to read it, it’s that good.

While on the subject of stewardship, Gerry Acher, the incoming Chairman of the RSA,  has issued a challenge that’s worth taking a look at:

Reduce personal carbon emissions to five tonnes a year

It’s intriguing, placing the onus of dealing with at least some aspects of the global warming crisis on people like you and me. It could just work. I’ve signed up anyway and will try the challenge out. Read the blurb. Read the blog. Decide for yourself.

Who knows, we may be approaching a time when personal carbon emission limits aren’t science fiction, and where we will start trading limits and capacity on a P2P basis. In which case we have everything to gain and nothing to lose. Bit like Pascal’s Wager.

I must have missed it the first time around, and only saw it via Boing Boing (thanks, Cory!).

Reuters reported last Friday that “Book sales get a lift from Google scan plan“.

I didn’t know whether to laugh or cry when I read the story. Read it for yourself.

Someone’s finally figured out that letting people ‘taste” books actually helps sell books. Even obscure ones. Especially obscure ones.

I guess the penny had to drop sometime. As Doc is wont to say, they will make money because of the excerpts rather than with the excerpts.

I’ve been so taken with the writings of Max de Pree on leadership that, some time ago, I decided to look deeper into Herman Miller the company. And in doing so, I came across a book written two decades ago called Business As Unusual, by Hugh De Pree, who preceded Max as President and CEO.

In that book, Hugh quotes “DJ” De Pree, talking about how he felt at the start of the Great Depression.

“I figured we had one year to go before bankruptcy. When you face something like this, you realise something happens inside you. My soul searched and I realised the evils of the industry, which needed correction if I was to stay in business. I listed the following problems in the furniture industry:

• Four markets a year.
• Short-lived designs and the buyer’s first question: What’s new?
• Closeouts had to be sold, at up to fifty per cent off.
• Commission salesmen. Some had seven or eight lines and sold what was easiest to sell. They showed only what they thought were the very best values, travelled only when they wanted to travel.
• Stores took over. We were only a fabricator, with no business identity or plans.
• We had no opportunity for repetitive manufacturing.
• There was no contact with the ultimate user of the product.
• Low wages. Ignoring the ideas of people. Tough attitude towards labour. Responding tough attitude towards management.
• Up and down schedules as a result of all this, with layoffs on short notice, uncertain hours, short hours and short paychecks.” 

Amazing stuff, written three-quarters of a century ago.And I thought to myself, is this a manifesto-in-reverse for the Intention Economy and Vendor Relationship Management, as  evinced by Doc et al? Is this a manifesto-in-reverse for those firms that seek to make something of themselves in years to come?
I wonder.