Four Pillars: Some musings on enterprise search

Have you been following what’s been happening at WWW 2006 in Edinburgh recently? One of my favourite cities, wish I could have been there.

It’s worth checking the site out. Some interesting papers and discussions. Incidentally, there’s a footnote to each paper:

Copyright is held by the International World Wide Web Conference Committee (IW3C2). Distribution of these papers is limited to classroom use, and personal use by others.

Ironic, isn’t it? To see a condition like that placed on research about the web, on the web, relating to a conference attended by Sir Tim Berners-Lee? Oh well.

But I digress. Some interesting papers. Like this one, on Using Annotations in Enterprise Search. The gist of the argument appears to be that intranets are different from the Internet, and that we can use annotation and similar feedback loops to improve the enterprise search experience. Nothing particularly earth-shattering, but worth a read. We can always learn.
I was more intrigued by some of the assertions made in the paper. Examples:

  • “….company employees cannot freely create their own Web pages in the intranet”
  • “….algorithms based on link structure analysis….do not apply to intranets the same way as they apply to the Internet”
  • “….the amount of anchor text, one of the major factors used by Internet search engines….is very limited in intranets”
  • “….One such characteristic is the absence of spam in intranets”

I wonder. Don’t blogs and wikis lower the barriers to web page entry? Can’t we improve the quality and quantity of anchor text by using such tools? Sure, I believe that annotation, tagging, feedback loops and collaborative filtering all need to migrate from the world outside to the enterprise. But it’s more than that. All this is not worth doing unless we’ve lowered the barriers to conversation, to publishing, in the first place.

I wonder. Isn’t spam nothing more than an act of commission or omission that clogs up arteries? Everyone understands spam in an e-mail context. Spam exists all over the place in organisations. Yes there is internal e-mail spam as well. There’s “Meeting-Agenda” spam. There’s “Things-that-need-deciding” spam. There’s “Things-that-need-prioritising” spam. Fundamentally there’s spam in every workflow channel; spam covers a multitude of techniques used by professional organisational men to deny service, particularly when it comes to prioritisation or even decision.

I wonder.

Amongst other papers of interest are:

Collaborative exchange of news feeds

Visualising tags over time

Semantic Wikipedia

The impact of online music services on the demand for stars in the music industry

I don’t agree with everything that’s in those papers. In fact, for some of the papers, the disagree bit outweighs the agree bit. But that’s what makes it worth my while, seeing other points of view and figuring out what the differences mean. As Gregory Bateson said about information, it is “a difference that makes a difference”.

Four Pillars: Serendipity knows no borders

I’ve been gently restoring the blog back to the way it used to be. And, while waiting for things to happen, I took a random walk. I read the news today, oh boy. And while I avoided finding out how many holes it took to fill the Albert Hall, I chanced across Bill Thompson’s reference to a Steven Johnson post about an op-ed on “the endangered joys of serendipity“.

Go Bill. Go Steven. Mr McKeen does not know what he’s talking about.

There are many reasons to believe that serendipity is increased, not decreased, as a result of the web. That creativity is imbued with new dimensions and in no way diminished. I can think of many arguments; let me concentrate on just one.

A quote from the excellent Judy Breck book 109 ideas for virtual learning:

The huge shift under way for learning is that the virtual knowledge ecology is not geographical, it is global. It transcends localities and cultures and is available in common to students everywhere, along with anyone else who is interested. The emergence of the virtual knowledge ecology represents a titanic shift from localised learning to a common global knowledge resource.

This is key. Serendipity knows no borders.
An aside. Over forty years ago, Leo Goodman suggested that we get the word Serendipity from Serendip, a corruption of the word Saradip, used in Hindusthani to describe what was then Ceylon. Arthur Clarke then made sure we knew about it.

Serendipity knows no borders. Particularly now we have the web.

Four Pillars: The Personal Wayback Machine Rides Again

Well, it looks like normal service has been restored, to all practical intents and purposes. The community has been fantastic, coming up with rich and varied suggestions as to how I could salvage the blog. A number of you scraped Google caches and sent the salvage on, particularly Chris and Doc. One, Myrto, had a complete set of my posts in Outlook via Newsgator. Some of you, particularly Malc, pointed me at the feedburner cache where the last 73 posts were available. My Mac account had a faithful copy of all comments received for moderation. While Google, Wayback Machine and Feedburner were less than complete, Niall found that Alexa had the complete store.

So there were lots of suggestions, all of them good; lots of routes to take, all of them good, but with varying results and time: particularly in the context of timed and dated posts, integrity of past inbound links, visibility of comments and links and images. Malc, Steve and Tim did the thinking about the best way to do it, and then Tim made it happen. In seconds flat.

Thank you everyone. Particularly Steve and Tim.
As promised, I will share what I learnt, once I have had the chance to think it through. I’ve learnt many things. But the first thing that occurs to me is: Wow. P2P backup? But then I’m weird that way.

Four Pillars: On savouring knowledge en passant, and more on blogs as the opensourcing of ideas

A few days ago I mentioned that my copy of Richard Gabriel’s Patterns of Software had briefly passed through my hands en route the next reader. One of the things that characterises a good book is the urge it places on the reader, the urge to delve into it again en passant, as it were, and acquiring something new in the process; in its own small way, it is akin to the revelation known and loved by Bible scholars as they dwell on scripture.
And so it was with Patterns of Software. I dug. And delved. And tried to find the passage I was looking for, one which Doc Searls quoted in Making a New World. And failed to find it.

Which was good in its own way, because it made me go and find my battered copy of Open Sources 2.0, in order to re-read Doc’s piece. Within which I found said passage. Richard Gabriel is quoted as saying “Habitability is the characteristic of source code that enables programmers coming to the code later in its life to understand its construction and intentions and to change it comfortably and confidently”. Wow.

Another random walk had begun. This time leafing through Open Sources 2.0. And as I read Making a New World again, I was drawn repeatedly to the following diagram, part of a mesmerising sequence, at least for me:

commoditization.JPG

And then I was reminded of something Kim Polese said in the Foreword to the same book, the same collection of essays:

“There is an important difference, however, between open source commodities and those derived from raw materials (like wood or steel) that is harvested or mined. It’s a difference that will make the new, mature, software marketplace incalculably large.

The difference is this: open source commodities are produced by creative and resourceful human minds. Not by geology, biology, and botany. This means there is neither a limit to the number of open source products, nor a limit to the number of improvements.

Yet every one of those open source projects is concerned mostly with the improvement of their own products. While they care about how those products interoperate with other products, they can’t begin to account for all the combined possibilities where interoperation is required. That means there is room for businesses to test, certify and support combinations of open source products.

That’s what blogs are for. To test, certify and support combinations of open source ideas.

To turn kernels into snowballs.

The blogosphere becomes the CollabNet, the SourceForge of ideas.

As Doc said, commoditisation is contribution. And we’re all in the business of building and certifying idea stacks.

Four Pillars: On doubts and certainties

If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.

Francis Bacon (1561-1626)

Francis_Bacon.jpg

One of my favourite quotes.

Before my blog went walkabout, I had been working on the next “recap” of Four Pillars, and taking somewhat more time than I had for the previous ones. The draft posts for that recap may never materialise.

So what I thought I’d do is to share what I was writing about, an emergent “Ten Principles for Constructing the Four Pillars”: but in a summarised form, while I go about restoring the rest of the blog.

I claim no ownership of these principles, they represent a melange of ideas collected from conversations I have had and books and articles I have read. Snowballs in passing. Other than Cluetrain, of course. Which was The Kernel.

  1. Don’t get too hung up with your own propaganda. Building an enterprise architecture for the 21st century around the Four Pillars concept is not going to be easy; there is much that is nascent and emergent about the technology, and much to be learnt through experimentation; balancing an open mind with a healthy dose of scepticism is what is needed.
  2. Markets are conversations, as the Cluetrain guys said.
  3. Conversations are based on relationships. No “sound of one hand clapping” zen koan-types make relationships.
  4. Relationships thrive on trust and integrity. Whatever you have with people you cannot trust or rely on, find another word for it. Unction of the serpentine kind.
  5. Trust is based on transparency and reliability. Not evidence. But on openness. It is OK to have faith. Past-predicts-the-future is at best about transactions, and fails every time something new comes along.
  6. Transparency and reliability are rooted in identity. Relationships are only built between people who know and understand and respect each other’s identity.
  7. Identity cannot flourish without respect for privacy and confidentiality. Many laws were created with this is mind; yet the risk remains that the same laws are subverted to destroy privacy, particularly in the name of “security”. Of information, of company, of market, even of state.
  8. Privacy of the individual, the client, the patient, the citizen is paramount. This is the First Law of [Four Pillars] Information. Which is why the Dick Hardts of this world are right.
  9. No information asset can be monetised if doing so breaks the First Law of Information. This is the Second Law of [Four Pillars] Information. Any attempt to enforce or build anew Intellectual Property Rights must adhere to the First and Second Laws. Any DRM system that focuses on the monetisation before the privacy is fundamentally and fatally flawed.
  10. Only the market, a community of individuals, can establish or amend laws like these. It cannot be done by individuals like me acting alone. Or even with the help of Grassy Knoll.

Please don’t take the principles above too seriously.They sound too pompous for me right now, but it’s late, I’ve just come back from a Clapton concert, and it may be me reverting to type. There is something there, something that needs working on. Maybe a lot of working on. But something’s there.

Why was I going down this Ten Principles route? Because I felt we needed a yardstick, a frame of reference, that could be used to help us make sense of the Blefuscudian discussions we appear to be having at present. Polarised arguments relating to net neutrality, to intellectual property, to digital rights management, to identity; polarised arguments involving everyone; the more important the argument, the more impassioned and extreme the arguments.

And it’s not just Calcutta being Confused.

We need simple independent yardsticks. Even if all ten of the principles above are jettisoned in toto, only to be replaced by n better ones, then this post has succeeded.

That’s what I was trying to do in the recap that never was. Which moved from the Sermon on the Mount through Justin Hawkins of the Darkness to end with Jerry Garcia. Who knows, I may get the chance to rewrite it one day.

In the meantime, please keep the comments coming, tear my post apart and help me learn.