Recently I wrote about meeting Doug Engelbart for the first time, courtesy of a dinner invite from Tom Malone. Before dinner, I had the opportunity to hear Doug speak at a Center for Collective Intelligence seminar at MIT. As you would expect, he covered a lot of ground very quickly, and I won’t attempt to document all of it here.
One particular thread, however, intrigued me so much I feel driven to share it here, to see what people think. The quotes are as close to verbatim as I could make them; apologies for any inaccuracies or misrepresentations:
Tackling a large-scale problem requires a strategic rather than a tactical approach. The paradigms that shape our individual and collective perceptions of big problems, and of their possible solutions, they tend to evolve much more slowly than the problems themselves, due to the inherent complexities of the big problems. [….]Far too many of the possible improvement steps will change the design environment for other improvement candidates. So we have to depend on an evolutionary process. We can learn to facilitate and accelerate that process.
What struck me about that line of thought was its potential applicability in aiding root cause analysis of complex systems problems. More and more, systems environments are growing more and more complex, as wave upon wave of device descend on our network shores; boundaries of the enterprise keep getting stretched, both in the supply chain as well as in the customer chain; the impacts of globalisation and disintermediation, of offshore and outsource, continue to be felt.
While all this has been happening, there have been a few other shifts as well. Each enterprise has tended to become multivendor in itself, with a greater number of hybrid environments; if anything, this has been accelerated by the opensource movement. Then, as telephony becomes software, there is also a movement of intelligence from the core to the edge of the network, and this tends to aid customer-to-customer interaction at that edge. As a result, looking at any given large enterprise, we tend to see the following characteristics:
- Sharp increases in device population and proliferation
- Steady creeping-out of the enterprise boundary
- An increase in hybrid environments
- Significant extensions to supply and customer chains
- Greater and more complex electronic relationships
- All happening over a global footprint
- All happening 24 x 7
When something goes wrong, it isn’t always that trivial to work out the root cause. The more complex the problem, the more likely we are to apply some sort of serial process to solving it. And maybe that’s where collective intelligence should meet Agile. Where we use the power of well-established knowledge bases and tie it up to the experience of a large collective in order to focus on a problem, then use an accelerated evolutionary process to iterate through the possible solutions, taking care to avoid “changing the design environment for other improvement candidates”.
Initially, I used to think about Agile somewhat narrowly, keeping to the bounds of systems development. Over the years, I’ve come to realise that, more than anything else, Agile is a mindset and a business strategy. It is only more recently that I’ve come to consider Agile as a problem-solving tool and an aid to bug-fixing, but I wasn’t sure about that. Now, having heard what Doug had to say, I’m beginning to think there is something to it after all.
I’d love to hear your views, particularly from people who use a combination of agile methods and collective intelligence techniques to solve complex systems problems.
An aside: In a many-sided marketplace, is every participant a fractal representation of the marketplace itself? Sometimes when I look at a large enterprise, it seems to behave like an open software platform in its own right. Just a thought…. for some time now I’ve been intrigued by the fractal nature of organisation structures, and at the comic ways we use to try and de-fractalise them by PowerPoint and memorandum.
JP,
My recent reading has been in and around Corporate Democracy. This relates to all you are saying about an Agile and fractal nature of an organization.
Some related links:
http://www.gbn.com/BookClubSelectionDisplayServlet.srv?si=69
http://www.designnine.com/library/docs/other_papers/Democratic_Company.pdf
and http://worldblu.com/ although this site has not been updated for a while.
These ideas and what you have said in this post are swimming in a mixup in my brain. I would love to have some discussion to help put them in order.
The organization should not be de-fractalised. It needs to embrace the Agile and open methods. By changing the basic organization from a management hierarchy, into smaller groups making decisions, and putting the power at the edges of the network all the benefits of the Agile methods are available.
The issue is really about the people with power giving up control and trusting that the folks on the edges know what they are doing. The same issue keeps corporate CIOs from using OSS. They don’t have someone to blame, another form of control.
The discussion needs to be wider than just using OSS or Agile methods. The whole structure of the organization should be considered.
I worry that all this talk about fractals amounts to a rehash of Haeckel’s theory of recapitulation (“ontogeny recapituaaltes phylogeny”) seen through the lenses of Mandelbrot’s “fractal geometry of nature” and mapped over to “organic organizations!” I prefer to put my money on Giddens’ structuration theory. This assumes a dialectical relationship between the STRUCTURES of rules and resources in any social organization and the “regular social practices” that actually take place around those rules and resources. Giddens is less interested in the question of whether an organization should be hierarchical and more interested in how to REPRODUCE successful engagements between structures and practices.
I have come to a very similar conclusion via my experience with the US DOD environment and it’s forced expansion of it area of responsibilities due to current events.
The world is so large and complex that we need to evolve a living organism model where we bring together smart and experienced people and support them with rapid access to whatever knowledge base required. Let them rapidly consider different options in an iterative process so they can evolve optimal solutions and react/adjust in agile manner to the changing environment.
Excellent post, thank you! About the same time you met with Doug Engelbart, I was in Cambridge for a session at the New England Complex Systems Institute. Two of MIT’s finest business minds were there, Peter Senge and John Sterman. Yaneer Bar-Yam led the session. The presentations and discussions focused on how to apply complex systems thinking in a business context. The concept of evolutionary processes, or in the case of IT, evolutionary engineering were front and center. Your post, and the following comments, are spot on in so many ways.
A significant challenge I face in my practice is finding a means for large and established organizations to transform into a more evolutionary type business and IT model. Newer and smaller organizations have the benefit of either starting off “correctly” or transforming quickly due to their small size. I firmly believe that the transformative process for larger organizations is an evolutionary process as well.
I agree with Doug that the whole structure should be considered. Too often though, the complexity of the whole structure is used as a justification for why the whole structure is too big to consider. I think that the “too big to consider” argument assumes that a firm control structure must be in place and thus all of the variables must be closely managed. Clearly this is impossible. As Doug points out, this type of control thinking is what is causing the problem in the first place. So, it is in this context that the transformative process must also incorporate the same concepts as the “agile” environment we seek.