Musing about collective intelligence and Agile and complex systems problems

Recently I wrote about meeting Doug Engelbart for the first time, courtesy of a dinner invite from Tom Malone. Before dinner, I had the opportunity to hear Doug speak at a Center for Collective Intelligence seminar at MIT. As you would expect, he covered a lot of ground very quickly, and I won’t attempt to document all of it here.

One particular thread, however, intrigued me so much I feel driven to share it here, to see what people think. The quotes are as close to verbatim as I could make them; apologies for any inaccuracies or misrepresentations:

Tackling a large-scale problem requires a strategic rather than a tactical approach. The paradigms that shape our individual and collective perceptions of big problems, and of their possible solutions, they tend to evolve much more slowly than the problems themselves, due to the inherent complexities of the big problems. [….]Far too many of the possible improvement steps will change the design environment for other improvement candidates. So we have to depend on an evolutionary process. We can learn to facilitate and accelerate that process.

What struck me about that line of thought was its potential applicability in aiding root cause analysis of complex systems problems. More and more, systems environments are growing more and more complex, as wave upon wave of device descend on our network shores; boundaries of the enterprise keep getting stretched, both in the supply chain as well as in the customer chain; the impacts of globalisation and disintermediation, of offshore and outsource, continue to be felt.

While all this has been happening, there have been a few other shifts as well. Each enterprise has tended to become multivendor in itself, with a greater number of hybrid environments; if anything, this has been accelerated by the opensource movement. Then, as telephony becomes software, there is also a  movement of intelligence from the core to the edge of the network, and this tends to aid customer-to-customer interaction at that edge. As a result, looking at any given large enterprise, we tend to see the following characteristics:

  • Sharp increases in device population and proliferation
  • Steady creeping-out of the enterprise boundary
  • An increase in hybrid environments
  • Significant extensions to supply and customer chains
  • Greater and more complex electronic relationships
  • All happening over a global footprint
  • All happening 24 x 7

When something goes wrong, it isn’t always that trivial to work out the root cause. The more complex the problem, the more likely we are to apply some sort of serial process to solving it. And maybe that’s where collective intelligence should meet Agile. Where we use the power of well-established knowledge bases and tie it up to the experience of a large collective in order to focus on a problem, then use an accelerated evolutionary process to iterate through the possible solutions, taking care to avoid “changing the design environment for other improvement candidates”.

Initially, I used to think about Agile somewhat narrowly, keeping to the bounds of systems development. Over the years, I’ve come to realise that, more than anything else, Agile is a mindset and a business strategy. It is only more recently that I’ve come to consider Agile as a problem-solving tool and an aid to bug-fixing, but I wasn’t sure about that. Now, having heard what Doug had to say, I’m beginning to think there is something to it after all.

I’d love to hear your views, particularly from people who use a combination of agile methods and collective intelligence techniques to solve complex systems problems.

An aside: In a many-sided marketplace, is every participant a fractal representation of the marketplace itself? Sometimes when I look at a large enterprise, it seems to behave like an open software platform in its own right. Just a thought…. for some time now I’ve been intrigued by the fractal nature of organisation structures, and at the comic ways we use to try and de-fractalise them by PowerPoint and memorandum.

The Mother Of All Demos

Funny place the internet. Or maybe we’re the ones that are funny. Maybe it’s just me, funny peculiar. You know, I never thought of looking for The Mother Of All Demos on the web.

Then, last week, I had the opportunity of attending a Doug Engelbart seminar at MIT, and the incredible privilege of having a private dinner with him, courtesy Tom Malone and the MIT Center for Collective Intelligence. [My thanks to all concerned.] Naturally, I looked for the video soon after, and there it was.

If you haven’t seen it already, do look at it: link. I’ve also shown it in my VodPod in the sidebar to this blog. It’s an hour and 15 minutes long; do not adjust your set when watching it, there is no sound for the first few minutes. I think everyone who has any interest in computing should watch it and hear from the man who, along with his team, gave us the mouse, hypertext, and the precursor to today’s GUI.

It is also worth going to this link, if you have the time.

Most of you know my views on the unusability of today’s patent regime. So you can imagine how I felt when I learnt about this (I quote from Wikipedia, but I had heard of it independently):

In 1967, Engelbart applied for, and in 1970 he received a patent for the wooden shell with two metal wheels (computer mouse U.S. Patent 3,541,541 ), describing it in the patent application as an “X-Y position indicator for a display system”. Engelbart later revealed that it was nicknamed the “mouse” because the tail came out the end. His group also called the on-screen cursor a “bug,” but this term was not widely adopted.

He never received any royalties for his mouse invention, partly because his patent expired in 1987, before the personal computer revolution made the mouse an indispensable input device, and also because subsequent mice used different mechanisms that did not infringe upon the original patent.

I’m glad to hear that Logitech provide Doug with free office space and support; other companies (and there are a few slightly bigger ones (!) ) should note and follow suit.

Of American Idol and software platforms

Have you ever noticed that whenever you try and describe something “new”, there is a tendency to use words that relate to the “old” thing it replaces? I guess it’s human nature. The trouble is, quite often this leads to a misunderstanding of what the new thing is about. As Einstein is reputed to have said,

The significant problems we face cannot be solved at the same level of thinking we were at when we created them

Which is why, whenever I get a chance, I try and use an example from a completely different field to try and explain what I’m saying.

Take software platforms. A month ago I wrote this post,  describing them as multisided open marketplaces. Many of you “got” what I was talking about, many of you didn’t. So here’s a view from a different perspective.

What American Idol can teach us about software platforms:

Let us think of American Idol as a platform, and see what happens. [In the UK, if you prefer, you can use the same arguments for X Factor, there is no real difference.]

  • The platform is open to all, with no barriers to entry. You only have to watch the early stages to know this is true.
  • There are no fees to be paid by entrants. Instead, the platform makes money by advertising, and by charging for the voting process. An interesting version of The Because Effect.
  • Everyone understands what the platform is for, but not everyone uses the platform as it was intended. There is a whole industry of spin-offs generated, ranging from merchandising and out-takes through to the alternative careers that have been launched. How else can you explain a Sanjaya or an Eye-Of-The-Tiger? So no surprise, you now have American Idol video games and syndicated series as well.
  • While success has its roots in openness of access, the real value is in the quality of what emerges, which requires excellent moderation. Yes, it’s true, Simon Cowell is a 1000lb gorilla.
  • While moderation is important, it is the market that sets the standard: what really matters is what the people think. Which is why the moderators can’t start believing in their own propaganda. 
  • A good platform transcends national and cultural boundaries. It’s not just about X Factor and American Idol and Indian Idol (I understand there are 10 X Factors and ) but also Classical Idol and Country & Western Idol and Left-Handed Yodelling While Drunk Idol. Believe it or not, there is actually an American Inventor series.
  • Scalability is an essential ingredient. If people couldn’t get through at vote times, there would be no platform; if people could not choose from a variety of ways to get through, there would be no platform.

Enough said.  Just stuff to think about. Comments welcome.

Uploading text

Early comments and conversations suggest that I didn’t get my point across when discussing the moods and changes of various armed services organisations with respect to emerging technologies.

The point I was trying to make was this:

An integral, essential part of the web as it is today is its writeability, its “liveness”. When you comment on a blog or add an entry to Wikipedia, what you are doing is uploading text. It is no different from what you do when you contribute a photograph to Flickr or a video to YouTube.

This writeability is key. It is what allows conversations to take place, learning to take place, democratised innovation to take place, culture to form and morph. It is what makes today’s web what it is.

You don’t have to participate. But you must have the right to. That is what makes today’s web different from yesterday’s web. Any organisation which seeks to gain value from today’s web needs to understand this. The web is two-way. So when you want to take advantage of YouTube, you need to understand this two-way-ness. And be part of it.

Sure, there is a difference between text and audio and image and video. To us. But not to the computer. As Bob Frankston keeps reminding me, it’s all bits. Nothing else. Deciding not to allow access to YouTube or Flickr or last.fm is a perfectly reasonable thing for the military to do. But that decision is a two-way decision. Neither up nor down. Or both.

That’s what I meant when I saw the “usual suspects being wheeled out”. Argue about security, sure. Argue about money, sure. But don’t argue about the two-way nature of the web and still expect to gain value from it.

George Dyson’s lecture on John Von Neumann on IT Conversations

I was reminded of the existence of this during a completely different conversation over at Gordon Cook’s discussion group. Fascinating talk, provides an unusual and lively view of aspects of computing history. I think it’s important for all of us to revisit such conversations and discussions, it will help us do the right thing when it comes to the internet, intellectual property rights and identity.

For people who have the time and the inclination, there’s a hefty slide presentation available via the same link, you may find it worth while to listen to the audio while working the presentation yourself. Don’t be put off by the double misspelling of Von Neumann on the site; the talk is of high quality, as are a number of the other sessions organised by the IT Conversations guys.