On firehoses and filters: Part 1

Image above courtesy of Library of Congress, Prints and Photographs Online Catalog.


I’ve never been worried about information overload, tending to treat it as a problem of consumption rather than one of production or availability: you don’t have to listen to everything, read everything, watch everything. As a result, when, some years ago, I heard Clay Shirky describe it as “filter failure”, I found myself nodding vigorously (as us Indians are wont to do, occasionally sending confusing signals to onlookers and observers).


Filtering at the point of consumption rather than production. Photo courtesy The National Archives UK.


Ever since then, I’ve been spending time thinking about the hows and whys of filtering information, and have arrived “provisionally” at the following conclusions, my three laws of information filtering:

1. Where possible, avoid filtering “on the way in”; let the brain work out what is valuable and what is not.

2. Always filter “on the way out”: think hard about what you say or write for public consumption: why you share what you share.

3. If you must filter “on the way in”, then make sure the filter is at the edge, the consumer, the receiver, the subscriber, and not at the source or publisher.


What am I basing all this on? Let’s take each point in turn:

a. Not filtering at all on inputs

One of the primary justifications for even thinking about this came from my childhood and youth in India, surrounded by mothers and children and crowds and noise. Lots of mothers and children. Lots and lots of mothers and children, amidst lots and lots of crowds. And some serious noise as well. Which is why I was fascinated by the way mothers somehow managed to recognise the cry of their own children, and could remain singularly unperturbed, going placidly about their business amidst the noise and haste. This ability to ignore the cries of all the other babies while being watchful and responsive to one particular cry fascinated me. Years later, I experienced it as a parent, nowhere near as good at is as my wife was, but the capacity was there. And it made me marvel at how the brain evolves to do this.

Photo courtesy BBC

There are many other justifications. Over the years I’ve spent quite a lot of time reading Michael Polanyi, who originally introduced the “Rumsfeld” “unknown unknowns” concept to us (the things we know we know; the things we know we don’t know and the things we don’t know we don’t know). I was left with the view that I should absorb everything like a new sponge, letting my brain work out what is worth responding to, what should be stored for later action, what should be discarded. And, largely, it’s worked for me. Okay, so what? Why should my personal experience have any bearing on this? I agree. Which is why I would encourage you to read The Aha! Moment: The Cognitive Neuroscience of Insight, by Kounlos and Beeman. Or, if you prefer your reading a little bit less academic, try The Unleashed Mind: Why Creative People are Eccentric. In fact, as shown below, the cover of the latest issue of Scientific American MIND actually uses the phrase “An Unfiltered Mind” when promoting that particular article.


b. Filtering outputs

We live in a world where more and more people have the ability to publish what they think, feel or learn about, via web sites, blogs, microblogs and social networks. We live in a world where this “democratised” publishing has the ability to reach millions, perhaps billions. These are powerful abilities. And with those powerful abilities comes powerful responsibilities. Responsibilities related to truth and accuracy, responsibilities related to wisdom and sensitivity. Responsibilities related to curation and verification. None of this is new. Every day we fill forms in with caveats that state that what we say is true to the best of our knowledge and ability; every day, as decent human beings, we take care not to offend or handicap people because of their caste, creed, race, gender, age. Every day we take care to protect minors, to uphold the confidentiality of our families and friends and colleagues and employers and trading partners and customers. Sometimes, some of these things are enforced within contracts of employment. All of them, however, should come under the umbrella term “common decency”.


These principles have always been at the forefront of cyberspace, and were memorably and succintly put for WELL members as YOYOW, You Own Your Own Words. Every one of us does own our own words. Whatever the law says. It’s not about the law, it’s about human decency. We owe it to our fellow humans.

When we share, it’s worth thinking about why we share, something I wrote about here and here.

c. Filtering by subscriber, not by publisher

Most readers of this blog are used to having a relatively free press around them, despite superinjunctions and despite the actions taken to suppress Wikileaks. A relatively free press, with intrinsic weaknesses. Weaknesses brought about by largely narrow ownership of media properties, weaknesses exacerbated by proprietary anchors and frames, the biases that can corrupt publication, weaknesses underpinned by the inbuilt corruptibility of broadcast models. Nevertheless, a relatively free press.

The augmentation of mainstream media by the web in general, and by “social media” in particular, is often seen as the cause of information overload. With the predictable consequence that the world looks to the big web players to solve the problem.

Which they are keen to do.

Google, Facebook, Microsoft et al are all out there, trying to figure out the best way of giving you what you want. And implementing the filtering mechanisms to do this. Filtering mechanisms that operate at source.

There is a growing risk that you will only be presented with information that someone else thinks is what you want to see, read or hear. Accentuating your biases and prejudices. Increasing groupthink. Narrowing your frame of reference. If you want to know more about this, it is worth reading Eli Pariser’s book on The Filter Bubble. Not much of a reader? Then try this TED talk instead. Jonathan Zittrain, in The Future Of The Internet and How to Stop it, has already been warning us of this for a while.

Now Google, Microsoft, Facebook, all mean well. They want to help us. The filters-at-source are there to personalise service to us, to make things simple and convenient for us. The risks that Pariser and Zittrain speak of are, to an extent, unintended consequences of well-meaning design.

But there’s a darker side to it. Once you concentrate solely on the design of filterability at source, it is there to be used. By agencies and bodies of all sorts and descriptions, ranging from less-than-trustworthy companies to out-and-out malevolent governments. And everything in between.

We need to be very careful. Very very careful. Which is why I want to concentrate on subscriber-filters, not publisher-filters.

Otherwise, while we’re all so busy trying to prevent Orwell’s Nineteen Eighty-Four, we’re going to find ourselves bringing about Huxley’s Brave New World. And, as Huxley predicted, perhaps actually feeling good about it.


More to follow. Views in the meantime?



The Silent Spring of the Internet: Part II: Understanding “unpaid”

Yesterday I spent some time thinking about what Rachel Carson experienced in the period leading up to her writing The Sea Around Us, and following that up a decade or so later with Silent Spring. How we can learn from those experiences as we hurtle towards wholesale destruction of the internet and all it stands for, particularly with phenomena like the Digital Economy Act, the DMCA, Hadopi and the most appalling of them all, ACTA. I shared some of those thoughts with you here.

Today I want to spend a little more time on the same subject, but from a different perspective. Let me explain why.

Ever since I got visibly involved in the Digital Economy Bill debate, I have been dismayed by the number of people who spend time accusing me of complete naivete when it comes to the download and fileshare debate. The accusations usually begin with an assumption (on the part of the accusers) that I (and people like me) do not want to see “creators” properly rewarded for their work; this is then extrapolated into further accusations that classify unpaid digital downloading as theft, somehow taking the civil offence of copyright infringement and converting it into a criminal offence, despite the “owner” of the asset continuing to have complete and unfettered access to the asset, despite the extreme nonrival nature of the asset.

When I’ve tried to debate with the accusers, their usual stance has been “don’t talk to me about the need to change intellectual property law, don’t talk to me about how badly broken copyright law is, don’t talk to me about downloaders being the primary buyers, don’t talk to me about fair use and free speech and all that jazz. What you’re talking about is theft, pure and simple. Don’t come back until you’ve got sensible proposals for how creative people get paid for their work.”

So that’s where I want to begin.

Making sure creative people get proper payment for their work.

You see, where I come from, software is a creative business. Software is a creative industry… it must be: after all, the fancy figures for illegal downloads include the “lost revenue” for pirated software. [I am now trying desperately not to give in to the temptation to make up sentences that have words like “hoist” and “with” and “own” and “petard”. After all, this is a smelly enough business as it is].

Where was I? Oh yes. Creative people getting paid for their work.


Let’s start with Linux. 60% of all web servers run Linux.  “It would take $10.8 billion dollars to build the Fedora 9 distribution in today’s dollars“. Just one distribution.

Or let’s look at the Apache HTTP Server, which went past the 100 million web sites landmark a year or two ago.

Or let’s look at the volunteers who keep the Internet Storm Center manned and productive.

Or let’s go back in time and look at the volunteers who wrote RFC 675, without which there would be no internet.

Or let’s look at the people who work for and with industry bodies like ICANN and W3C and IETF and, more recently, the Web Science Trust.

All possible because of volunteers. Yes the volunteers may get paid by organisations that can perceive the value generated by such voluntary activity; but this form of payment is closer to patronage than anything else.


I could go on and on, but I won’t. I hope I’ve made the point already. The point is that for the internet to exist, many things have to be in place. There have to be people willing to invest in stuff; people willing to connect the stuff up; people willing to run the many-headed beast that emerges as a result of connecting the stuff up; people willing to protect the beast as it mutates organically, naturally; people willing to keep trying to find faster, cheaper, better ways of doing things.

It all begins with a state of mind. A willingness to share. A focus on being open, a focus on enabling people at the edge to do things they would otherwise not be able to do.

Without that state of mind there are no volunteers, there is no set of standards and protocols, there is no process, cumbersome or otherwise, to let the internet evolve: there is no internet.

Without that internet there is no goldmine for “rightsholders” to strip of all value. Without that internet artists will get paid even less than they do currently, however unlikely that sounds.

Incidentally, here’s a very instructive method of visualising what musicians get paid: [My thanks to @gapingvoid and to @psfk for sharing it with me].

[Also incidentally, Hugh is a good friend, I love the way he thinks, and I really like his recent passion “Remember Who You Are”. He’s got some really great posts together under that banner. Which is why it was a privilege for me to be able to contribute this post over at Gapingvoid.]

Which brings me to the end of this particular post.

We need to remember who we are. Stewards of the internet. The internet, a concept, a state of mind, a set of values, a network of networks of people, things and infrastructure. Where people live and work and learn and read and create. Oh yes, and where people occasionally listen to music or watch videos.

I’m going to continue to think about the internet, particularly in the context of writings like Jonathan Zittrain’s Future of the Internet; Eben Moglen’s recent speech on Freedom In The Cloud and David Gelernter’s Time to Start Taking the Internet Seriously

The internet was built for sharing. The internet relies on people who share their time freely and passionately.

There is a catch, however. These people expect something in return for the investment they made, the investment they make, the investment they are prepared to continue to make. And that something is this: a free, unfettered internet.

So when the talk in cafes and dinner tables turns to creative people and the need to make sure creative people get paid properly, do make sure you include all creative people and all modes of payment.

The silent spring of the internet: cyberspace needs its stewards

Maybe it’s because of the events leading up to the Digital Economy Bill becoming an Act here in the UK. It’s been a bit like Chinese water torture for many months; then, more recently, as the BPI saw their chance to corrupt parliamentary process and took it, it felt more like being waterboarded. I have had it up to here with people who think the internet was built to become a distribution mechanism for Hollywood and Universal Music and David Geffen.

My first objection to the Digital Economy Bill was to do with technical difficulties in proving who downloaded what: the complexity and inefficacy of technical solutions, the guaranteed waste of time and money, the likelihood of erroneous accusations, the unwanted consequence of driving dissent underground. My second objection was to do with the nature of the punishment, completely out of proportion with the crime, possibly illegal in human rights terms and with definite and unnecessary collateral damage on non-participants. My third objection was to do with the manipulation of data, the extrapolation of questionable samples into WMD-like justifications, but then I have to accept that statistics and lies have been kissing cousins for many years now. My fourth objection was to to with the corruption of process, the way the Bill was timed, how debate was avoided, how all parties achieved nothing but grubbiness in the process. And my final objection was to do with the people involved, the vestedness of their interests.

Many of us who opposed the Bill vehemently were quite happy to see legitimate and proportionate action taken against thieves. Legitimate. Proportionate. Against thieves. Sadly the Bill had nothing to do with words like those.

The industry lobby did their work well. Now we have to get used to a world where filesharing and downloading are both wrongly equated with theft, where damaging action can be taken on mere suspicion, and where dictatorial powers may be assumed almost at will. All to try and hold on to a dying business model. There will be consequences, unexpected consequences. [For those of you who are interested, I wrote about the data here, here and here, about the Bill’s inappropriateness of punishment here, about the unreasonable bias here and about the core issues related to the Bill here and here. And if you want to understand how retrograde all this is, read this. ]

What’s done is done. And we will live with the consequences. And learn from them, and maybe even change as a result. The Digital Economy Bill was a skirmish, maybe even a battle, but it wasn’t the war.

The War is about the internet: what it is, what it means, what it stands for, how it works, who it works for, and many such related questions.

It’s been an interesting week or so in this context.

Apple and their SDK terms; Twitter and Tweetie; the Appeals Court and their ruling on the FCC and Net Neutrality; Microsoft and Kin. European telcos catching the Ed Whitacre disease. All this in an environment that has Google and ChinaAndroid, the Droid and the Nexus One, all apparently living in perfect harmony.

By the pricking of my thumbs…..

I think we’re heading towards the cyber equivalent of what Rachel Carson saw and understood when she wrote Silent Spring nearly 50 years ago, having established her reputation with The Sea Around Us.

  • The internet is a sea around us, and we’re polluting it. We’re polluting it for short-term gain, we’re polluting it without really understanding the ecosystem that has evolved around it, the creatures that live in it.
  • The internet is an ocean around us, still somewhat unknown, still being mapped. It is capable of nourishing and sustaining us, capable of supporting and encouraging trade and commerce, capable of giving us incredible enjoyment, helping keep us clean and healthy.
  • The internet is all the rivers around us, capable of being dammed and isolated, capable of being corrupted and polluted at industrial levels, capable of being poisoned, capable of drying up, capable of killing us.

[And yes, the internet is capable of supporting piracy as well. But let us first understand what extreme nonrival goods are, how copyright infringements are different from theft. If Labour use unlicensed images in a campaign advertisement, is it called theft? When John Fogarty can be accused of plagiarising himself, is it called theft?]

We will soon begin to understand what the internet is. What identity means in an internet context. What intellectual property means in an internet context. The establishment of a Web Science Trust may well accelerate all this.

When we do learn about all this, we will begin to enact laws. Laws that protect the internet. Laws that make criminals of people who damage the internet.

Rachel Carson may have helped us with an understanding of what it is to become stewards of physical space. We now need to become stewards of cyberspace as well.

In that sense, the Digital Economy Bill may actually be a godsend, bringing together disparate groups of people with common, passionately held aims.

No them out there, just an awful lot of us


Been travelling for a while. While I was catching up on my reading, I was reminded of a much-loved Douglas Adams quote in a post by Kevin Anderson. [Now Kevin is someone I read regularly and would recommend wholeheartedly]. Anyway, I just had to share the Adams quote again, for those who may not have seen it first time round a decade or so ago:

Because the Internet is so new we still don’t really understand what it is. We mistake it for a type of publishing or broadcasting, because that’s what we’re used to. So people complain that there’s a lot of rubbish online, or that it’s dominated by Americans, or that you can’t necessarily trust what you read on the web. Imagine trying to apply any of those criticisms to what you hear on the telephone. Of course you can’t ‘trust’ what people tell you on the web anymore than you can ‘trust’ what people tell you on megaphones, postcards or in restaurants. Working out the social politics of who you can trust and why is, quite literally, what a very large part of our brain has evolved to do. For some batty reason we turn off this natural scepticism when we see things in any medium which require a lot of work or resources to work in, or in which we can’t easily answer back – like newspapers, television or granite. Hence ‘carved in stone.’ What should concern us is not that we can’t take what we read on the internet on trust – of course you can’t, it’s just people talking – but that we ever got into the dangerous habit of believing what we read in the newspapers or saw on the TV – a mistake that no one who has met an actual journalist would ever make. One of the most important things you learn from the internet is that there is no ‘them’ out there. It’s just an awful lot of ‘us’.

It’s worth reading the whole Adams post, which you can find here. Things that were synchronous are becoming asynchronous as well; things that were asynchronous are becoming synchronous as well; we have a lot to learn about whom and what and when we trust as a result.

I’ve got your number

First you call 1176 households. You find that 136 people actually admit to using file sharing software. Not necessarily illegal, but who cares when you have a good story?

Then you take the percentage that those numbers represent, 11.6, and bump it up to 16.3; why? because people lie, of course. Not you.

Then you take the 33.9m people published as online by ONS and bump that up as well, to make it 40m. Because statistics also lie.

Mash the percentage with the target population. Et voilà, you get to 7m.

The 7m illegal downloaders referred to in the onslaught on “illegal file sharing”.

Sounds better than 3.9m, doesn’t it? Has a nice whole ring to it.

Sounds so much better than 136.

Full story at http://www.pcpro.co.UK/news/351331/how-UK-government-spun-136-people-into-7m-illegal-file-sharer