John Dodds, Lars Plougmann and Stephen Smoliar (sorry Stephen, haven’t found a simple way to link to your blog) were kind enough to share their views with me following my earlier post on the subject. There I quoted from Richard Feynman’sRogers Commission looking into the Challenger disaster. submission to the
The comments touched on three aspects of the communication problem. John felt that we need to focus on “accentuating the negative” in order to avoid everyone building castles of sand in the air, suggesting that both “clients” as well as “IT people” have a problem with understanding what is to be built and what can be built. Lars inferred that project managers wilt under pressure and agree to unachievable timescales. Stephen felt that the willingness for self-deceit came from a postmodern “withdrawal into fictions”.
As with many of these things, I have some sympathy with all three views. I posted what I posted primarily to learn more about the issue. Let me try and take the argument further, quoting again from Feynman’s submission:
Official management, on the other hand, claims to believe the
probability of failure is a thousand times less. One reason for this
may be an attempt to assure the government of NASA perfection and
success in order to ensure the supply of funds. The other may be that
they sincerely believed it to be true, demonstrating an almost
incredible lack of communication between themselves and their working
engineers.
In my previous post, I was trying to understand the root cause of the deviance between the “manager” view and the “engineer” view, why and where the management self-deceit was happening. My gut feel is that it has to do with biases we bring in, biases related to incentives, measurement, even the tools we use for managing projects. Let me try and explain.
When John raised the question of accentuating the negative, he was trying to ensure that people build things that can be built, that solve a real problem, that allow value to be delivered in small measurable increments. Of course I agree with all this. But I have a nagging worry that we have tried this and failed. When we accentuate the negative, quite often we seek to solve yesterday’s problem. Sometimes that is the right thing to do. Too often it is “paving the cowpaths”. Even if it makes me sound more idealistic than I want to sound, I feel driven towards avoiding this accentuation of the negative. Yes we should build things that can be built. Yes we should try and deliver value in small increments. But not necessarily by focusing on the negatives.
Similarly, of course I agree with with what Lars says. There is no dearth of project managers accepting unachievable deadlines as a result of some external pressure. But then there is also no dearth of project managers failing to deliver without any such pressure…. So where does it go wrong? How does the management view start deviating so much from the engineer view? This is not just about schedules, it’s about functionality and performance as well. When the rot sets in, it affects many things. And maybe Stephen’s right, but then I have to give up hope for a better way of doing things, and this I am not yet prepared to do.
In a previous post I have already spoken of waterfall/cascade development styles versus agile styles, and accepted that there are horses for courses rather than just one sledgehammer way. So maybe I need to understand all this within the context of the two styles.
In a waterfall style project, most of what we need to know is known from the start. At least that’s what the pro-waterfallers argue. In which case bad estimates are primarily due to bad estimation, either by the engineers themselves or by those who “edit” the engineering view en route to management. Bad execution is due to bad planning, bad quality is due to bad testing, and so on and so forth. Waterfalls are good when there is nothing to discover, nothing to learn, so we have to look for the faults elsewhere. They are not in the definition of the problem, but caused by corruptions to the original data relating to effort and timeline and functionality. Or corruptions to the resources applied, in terms of availability and skillset. Or corruptions to the processes applied, in terms of quality of test cases, the testing process itself, and often the quality of end-to-end-testing.
In an agile project the story is different. There is much that is unknown, and there is a considerable risk that corruption can occur between the engineer and management views. Engineers make best guesses based on the information they have, and quite often management, without a deep understanding of the “guess” aspects, convert them into hard deadlines and functionality and cost. And then we have a period of House-Of-Cards-meets-Emperor’s-New-Clothes, until we have a Yeatsian Second Coming:
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world,
The blood-dimmed tide is loosed, and everywhere
The ceremony of innocence is drowned;
The best lack all conviction, while the worst
Are full of passionate intensity.
When this happens (and it happens often enough) we have a traditional wailing and gnashing of teeth, and blame cultures revel in firing some and promoting others. Such is life. But rarely is all this done as sins of commission, as malevolent acts. More often than not, this road to hell, like most others, is paved with good intentions. I think Feynman’s right on both his points quoted above. Often there is “an incredible lack of communication between [management] and their working engineers”. Often this is to “assure perfection and success …[…]… to ensure the supply of funds”. And this is where I think the incentive bias, the measurement bias and the corruptions caused by the project management tools come in. And I think we do have new ways of working, new tools to use, that can get around these biases.
To paraphrase Einstein, engineers don’t fail, they find ways that don’t work. And then they find ways that do work. In bits. And then they have to test the whole thing, end-to-end, to ensure that the bits work together, to create the experience that the customer wants to have and own. The opensource community can teach us a lot about testing the bits and testing the end-to-end. Given enough eyeballs the bugs are shallow. Web 2.0 companies can teach us a lot about ensuring that the bits work together to provide the customer with the requisite experience. When they say Beta they mean “It works, but we’re still ironing stuff out”. They don’t ship until it works, and when they ship, they listen to customers who do the real end-to-end tests, who exercise the code and at the same time suggest ways to improve the overall experience. These approaches, covering community development and real customer functionally-integral Beta releases, they’re often intertwined. And they tend to rely on social software to improve communications throughout the process.
Success in project delivery takes place, as Kathy Sierra suggested, when the customer says “My software isn’t working” rather than “Your software isn’t working”. This emotional shift, when the customer takes ownership of the output, is absolutely critical. And it doesn’t take place unless there is something for the customer to experience in totality. Nobody takes a car to a garage and says “Your car isn’t working”. We have to let the customer experience the product as quickly as possible, so that we can pick up the bugs and deficiencies that cloud that experience. This requires agile mindsets, community development approaches, a release-quick-and-release-often schedule, good social-software based communications processes, and a willingness to ship Beta.
But all this is difficult to describe, plan for, estimate and monitor using traditional tools and methodologies. Agile development is likely to become more common as we continue to deal with uncertainties. Unless we solve this, we are likely to see increasing divergence between engineering views and those of management, as waterfall approaches get used to govern agile practices.
Rather to long for a comment so I’ve blogged it. http://notwaving.net/wordpress/2006/10/15/truth-you-cant-handle-the-truth/
JP
Negativity is such a loaded term and I use it somewhat ironically. I don’t eschew blue-sky thinking at all but equally don’t want to throw the baby out with the bathwater when continuous incremental improvements can generate great progress. Grand ambition is great, grand gestures that fall short are definitely not.
More specifically, I use “accentuate the negative” as a counterpoint to “accentuate the positive” which is all well and good but has become a macho management maxim that tends to ignore various elephants in the room. But I don’t see focussing on the negative as a defensive move – if you alter a negative, you don’t have to end up with a neutral. My intention would be to improve the negative so much that you end up with a definite positive.
JP, you touched on this, but here is something no tool can help or address. The management WANTS TO BE FOOLED!
I recently left a company because they were trying to build a product to a completely arbitrary schedule. Every single engineer, over 40 people said it could never happen. I did not believe. As a middle manager who actually listened to my people, I fought, I railed, I really tried to make people hear.
I went to my boss. I went to the CEO. They believed the outrageous claims that half working software would somehow fall into place and just work in 2-3 weeks. Finally I went to the VP of sales and said “Stop, do not tell your customers anything!” This was two weeks before the big deadline would be missed. If they had gone in front of the customers they would have looked like idiots. I won points with him, but was chastised by my management for TELLING THE TRUTH!
No one wanted to know the truth. It was uncomfortable, and made the marketing and sales people upset.
I left. Other people who stood up with me and said “The emperor has no clothes” are now getting fired.
The only metaphor that fits is religious faith. In many organizations you have to be one of the true believers, or your going to hell. No amount of reasoned argument can change that.
Tell me how to fight against that jihad and I will be forever grateful.
Doug, I suspect that we have to push the argument a bit deeper than the premise that management wants to be fooled and see if we can tease out any explanation. I think that my own premise that denial trumps rationality, which Tom Ayerst used as a point of departure in his own blog, is more a symptom than an explanation (and it is a symptom of “the postmodern condition,” which, itself is more a symptom of a prevailing context than an explanation). I think that we more more likely to find an explanation through an analysis of how one accounts for doing one’s job. This raises the familiar question of whether an enterprise is more accountable to its shareholders than it is to its customers. Shareholders have no problems with the well-wrought fictions of postmodernism, because, at the end of the day, they are only interested in the impact of those fictions on the price of the shares. It is only the customers who have to worry about whether or not the business is trafficking in fictions; but, if the business prefers treating its customer as OBJECTS, rather than SUBJECTS, they are unlikely to signify in its strategic decision making. I tried to address this in my own blog on September 2 in a first pass at an analysis of what I have come to call “CRM Society:”
http://blog.360.yahoo.com/blog-Mff23hgidqmHGqbcv.lfskakEtS6qLVHUEMFUG4-?cq=1&p=93
What if the lines between shareholder, customer and employee get more and more blurred?
Tom, John, Doug, Stephen, thanks for the comments. John, I take your point on your intent behind the use of “accentuating the negative”, and the irony inherent in it.
You have all given me food for thought for another post.
JP, what IF those lines get more and more blurred? The cynic in me immediately remembers that old saw: “If wishes were horses, then beggars would ride!” (No talk about courses here!) On the other hand my positive side (yes, I DO have one) asks, “Well, HOW could those lines get blurred?” One answer may lie in my comment on your most recent post advocating participatory design:
https://confusedofcalcutta.com/2006/10/15/continuing-with-musing-on-project-management-and-communication/#comment-11866
The heart of my advocacy revolves around the idea that participatory design is ultimately all about ONGOING CONVERSATIONS that need to be enabled through the entire development process. At the very least those conversations need to involve both producers and consumers, and there is nothing like a good conversation to blur the categorical boundaries that are imposed upon us! Should shareholders also be included in those conversations? Perhaps that is not a bad idea. My guess is that many (most?) shareholders give very little thought to the customers of the businesses in which they invest; and, for that matter, they probably give more thought to the financial reports than they do from any accounts of what is actually happening “on the shop floor” (metaphorically speaking). So, to the extent that shareholders could achieve a better understanding of how business is actually conducted, includion in the conversation would be a good thing. Nevertheless, there is probably a risk that shareholders too interested in quarterly returns could disrupt conversations between producers and consumers that arise when things are not going according to plan. I suspect that, once again, it all comes down to governance. There is no doubting the value of the conversations, but there is no ignoring that they need to be properly managed in such a way that all engaged interests are honored.
Somewhere on your page, JP, is an unclosed “strong” or “bold” tag.
Indeed, a brittle formatting system is a good stalking horse for a brittle engineering system.
The relationship in which a person promises to pay more than he has to someone who promises to deliver more than he can is not a new kind of problem and not specific either to technology nor even to exchange.
But, I fail to see how it is ‘post-modern,’ ala Stephen Smoliar’s comment.
I mght mention as well that the line breaks in your Yeats quotation are all wrong. In any case, given the origin of this theme in Feynman’s engagement w/ the Challenger disaster, perhaps you should have quoted a different Yeats poem?
An Irish Airman Foresees His Death
I know that I shall meet my fate
Somewhere among the clouds above:
Those that I fight I do not hate,
Those that I guard I do not love:
My country is Kiltartan Cross,
My countrymen Kiltartan’s poor,
No likely end could bring them loss
Or leave them happier than before.
Nor law, nor duty bade me fight,
Nor public men, nor cheering crowds,
A lonely impulse of delight
Drove to this tumult in the clouds;
I balanced all, brought all to mind,
The years to come seemed waste of breath,
A waste of breath the years behind
In balance with this life, this death.
And, finally, that I had thought to meet you at Jerry’s Retreat last weekend. I was going to recommend the Michael Ohayon novels by Batya Gur — quite unique in the genre of crime fiction and quite wonderful. I wondered whether you knew them?
Tom, I’ve never heard of Michael Ohayon, thanks very much. I’ll check him out. Think I found the errant emphasis tag, hadn’t noticed, comment much appreciated.
I know, I was meant to be at Jerry’s Retreat, but life has been somewhat hectic for me these past few months…..
Yes, JP, I should say that’s true – hectic indeed!
The author of the detective novels, btw, is Batya Gur; Michael Ohayon is her character.
I was looking forward to meeting you at the Retreat, above all, because I wanted to introduce you to a startup I’m involved with — you have worked closely, I know, with my friends at SocialText, and I rather think this will interest you extremely. In place of my spamming your blog, will you very kindly drop me an email when you have a chance?
will do. Rgds
Tom, you are right that there is nothing new about a “relationship in which a person promises to pay more than he has to someone who promises to deliver more than he can.” The “postmodern condition” to which I referred is one in which discourse devolves into an exchange of “fictions of convenience,” which is basically what you are describing:
http://blog.360.yahoo.com/blog-Mff23hgidqmHGqbcv.lfskakEtS6qLVHUEMFUG4-?cq=1&p=125
The important point is that, even if the customer sees through the fiction, the fiction is intended more for the shareholders, since, at the end of the day, the price of the shares is a speculation that can be just as easily driven by fictions as by the “hard data” on the books. In other words the fictions become the tail that wags the dog, which is why the movie of that name was so aptly titled!
that was a cool advice and i will definitely follow it.