Getting Confused about Search

Over the last few months, I’ve been hearing this term quite a bit: “search fatigue”. Since I had absolutely no idea what it meant, I thought I’d check. So I Googled it.

Until I saw what came up, if I’d been asked to guess what it meant, I’d have tried some variant on information overload or infoglut. Instead, what I got as search result number one was this article, suitably headlined Search Fatigue.

I read it. And I didn’t understand it. I quote from the article (which itself quotes from an article by Jeffrey Beall in a magazine called American Libraries):

Search fatigue, according to Beall, is a feeling of dissatisfaction when search results do not return the desired information.

“The root cause of search fatigue,” Beall told me, “is a lack of rich metadata and a system that can exploit the metadata.”

I read that, and re-read it. So search fatigue is an academic-sounding term for Google rage. Okay, I got that. And the next sentence was also fine; the richer the metadata, the richer one can make the search experience. So far so good. But soon after that I got confused. Beall went on:

For example, metadata-enabled searching, as you find in a library when it searches through its databases for resources, “allows for precise author, title, and subject searches,” Beall says. In other words, it looks only in the fields you request, rather than searching through the entire document. If you name the author, it looks only in the author field of each document, thus returning only relevant hits.

If Beall is being quoted correctly, he is asserting that deterministic search actually improves the search experience.

Everything I have learnt about search points the other way. Formal data structures, key fields, primary keys, these are all the ways we lost information in the first place. In fact tree structures were probably more responsible for losing things than everything else put together; have you ever tried looking for archived mail or files you Saved rather than Saved As on your PC?

I thought we were moving away from deterministic search to probabilistic models. I thought people at Google and Technorati and wherever else were finding ways to raise the relevance of amorphous poorly-filed information, using a variety of artifices to reflect the value of links and references. I thought we would be heading towards the next generation of collaborative filtering, where I can “pass” my search bias to someone else, or for that matter perform Boolean operations on search bias.

The problem that needs solving is not to do with finding things that have been well labelled and well filed. That we have always been able to do. What we haven’t been able to do is to find the messy stuff, partially named, partially remembered, often misfiled, often misclassified.

Valuing opensource

It’s getting more and more fashionable to buy an opensource company. Quite often, the buyer is a closed-source company.

For about a decade I earned a living as a salesman, selling software services. And when I started, I remember being told “a salesman is only as good as his next sale”.

I think that something similar needs to be said about opensource companies. Their value is not in the IP they produced in the past; instead, it vests in the IP they will produce in the future. Which is why the community is important. I am aware that a very small number of people really contribute to an opensource project, any opensource project; but what they contribute becomes valuable because of the community.

No community, no value. Caveat emptor.

More random musings on opensource: States and Transitions

I said I would revert to the theme with which Dan Farber ended his opensource post: the inevitability of hybrids.

Why are hybrids inevitable? I think it all boils down to what Stephen Smoliar and Gordon Cook have been talking about recently, both in the shape of comments here as well as in their own blogs. [Blogs overlap and underlap all over the place, like real conversations….].

States and Transitions.

Many of us know where we want to go, but the path is not yet clear. We know what today represents. We know what tomorrow should look like. But we struggle with the in-between.

That in-between, the transition, tends to have three characteristics

1. Everyone’s Entitled to My Opinion

Polarisation: Opinions get polarised. Everything is about Big Ends versus Little Ends. Blefuscu Redux. Proprietary versus open. Petrol versus electric. VHS versus Betamax.

2. You say Tomahto, I say Tomayto

Meme battles: The battle switches to ideas and terms as we strive to make sense of it all. The language gets less civil, more intense, as passions move past simmering point. When words are the only weapons the air tends to get its own tinge of blue.

3. Plus ca change, plus c’est la meme chose

While 1 and 2 are going on, with each side battling in glorious technicolor, the world carries on. And pragmatic people build pragmatic business models and exchange pragmatic value.

We’re all part of this ecosystem. It is said that each plant has its own specific parasite and its own specific pest, that famines were caused by people who carried plants to far-off lands without the apposite pest-parasite pairing. It’s not always clear what our roles are, who the pest is, who the parasite is. Maybe some of us who blog are pests. Maybe the media that feeds on our doings are parasites. Whatever it is, the outcome can and should be a healthy plant.

Hybrids represent transition. And some process of natural selection, as different hybrid strains battle it out for top slot; some atrophy and die, some adapt and survive.

So we have hybrids.

An aside. Hugh Macleod had this to say in a recent post. Over 95% of all Microsoft revenues come from their partners.

Think about it. What keeps the ecosystem going? Who is the pest? Who is the parasite? And is the plant healthy as a result?

Distribution channels are partners. Ecosystem members are partners. Customers are partners.

As we move from proprietary to open worlds, we are seeing another transition. The customer is becoming the partner. And not a day too soon.

Random musings on opensource

As you would expect, I spent a lot of time with my wife and children over the Easter break. And then stayed up to watch the golf. When I wasn’t doing either of these things, I was catching up on my reading.

Dan Farber’s True Nature Of Open Source post got me thinking. Go take a look yourself, Between The Lines is a place I visit regularly; in fact I read most of the ZDNet bloggers pretty often.

Dan ends with the following:

Economics don’t favor pure open source. The future is hybrids–cars, software, people, pets. It’s better for the planet…

And that’s what set me thinking. As usual, I’m sharing that thinking with you, in the knowledge it is provisional and only-partially-formed. For the sake of brevity I’m making this a bullet-point list. [Yes even I can be brief sometimes!]

1. Economics can favour opensource. Commodity economics, based on scarcity, does not. But gift economics, based on abundance, can and does favour opensource. There’s a lot we have to learn about the economics of abundance. It is at the heart of Doc Searls’ Because Effect; it connects Stallman and Raymond and Torvalds to Brand and even Garcia.

2. Gift economics relies, to quite an extent, on delayed gratification. The same delayed gratification that is at the heart of Daniel Goleman‘s Emotional Intelligence work. The same human capacity that engenders the perseverance that characterises so much of innovation, of invention. For that matter, the same human capacity that allows people to have faith.

3. The willingness to accept delayed gratification (besides being central to Goleman’s themes) is critical to building community, to engendering teamwork and collaboration. Communities are defined by their shared purposes, their treasures in heaven. They evolve and grow despite their differences, held together by their common goals. I would go so far as to say they evolve and grow because of their differences, they learn more from the differences than from the similarities. But they stick together because of their shared vision.

4. Teamwork and collaboration are essential for the success of any 21st century organisation. Collaboration within the enterprise, collaboration across the supply chain, collaboration across the customer chain. Collaboration with the customer.

5. Opensource connects the customer with the coder in different and powerful ways. Transparency of demand and supply. The impact of Linus’s Law. True future-proofing. The wisdom of crowds. Evolution of software as a result of natural selection, driven by open market adoption rather than the slave trade of vendor-locks. [If you think about it, vendor-locks are really a form of slavery].

So for me there is a golden thread that links opensource with abundance with delayed gratification with collaboration and teamwork all the way to co-creation of value with the customer. All economically sound, just not scarcity-economy sound.

What Dan says about opensource is true, but we must understand why before we can make the right calls. I will post separately about the inevitability of hybrids.

A plea

Both Chutki (at Cuckoo’s call) as well as Nandini (at Travelling Light) pointed me towards Patrick Ghose’s post about a very special reunion.

Sadly there’s no way I can make it to Cal by Thursday. Much as I’d like to.

So Bertie, Mel, Fuzz, I hope it all goes well. Wish I could be there.

Here comes the plea. Would you guys please let someone in the audience record and upload five minutes of the reunion on to YouTube or similar? That way people like me can be there, out of time, out of space yet vicariously present.