On firehoses and filters: Part 2

Note: This is a follow-up to an earlier post on the subject, written in May this year. You may find it worth the while to read that one first. But if you don’t feel like it, no problem. This post is readable standalone.

I love the very concept of publish-subscribe: if you search for the term in this blog, you’ll find I’ve written maybe a dozen posts on the subject over the last six years or so. So I thought it would be worth talking about filters and firehoses in that context.

So let me start with the three “laws” of information filtering that I laid out in that earlier post: (if you want to know why, please read the post; I’ve linked it it earlier in this post)

1. Where possible, avoid filtering “on the way in”; let the brain work out what is valuable and what is not.
2. Always filter “on the way out”: think hard about what you say or write for public consumption, why you share what you share.
3. If you must filter “on the way in” then make sure that the filter is at the edge, the consumer, the receiver, the subscriber, and not at the source or publisher.

Yes, as Clay Shirky put it, we don’t live in an age of information overload; rather, we live in an age of filter failure.

So everyone’s been looking for better filters.

Which is fine.

What’s not fine is when we expect the publishers to do the filtering. Because that allows bad actors to come and spoil the party, whether they’re bad corporations or bad governments.

It’s not the job of the publisher to do the filtering. It should not be the job of the publisher to do the filtering.

But there is a job for the publisher to do. And that is to provide the tools by which subscribers can filter.

Let me expand on this. [Incidentally, when I started using Google+, I raised this issue under the guise of asking for “circles” to be built by publishers as well as by subscribers…. with some interesting discussions and comments as a result].

We live in a world of publish-subscribe, and this world has three facets.

There is an infrastructure, allowing individuals and aggregations of individuals to publish stuff, and to subscribe to stuff.

We exist as publishers, sharing stuff on this infrastructure.

We also exist as subscribers, sharing stuff on this infrastructure.

So now let me look at what I want to do as a subscriber. I want to choose whom and what I “follow” or subscribe to. Most of the time, I expect to be allowed to follow whatever I choose. Sometimes the publisher places a restriction on following, permission is needed. You may just have to ask for permission, register in some form or the other. In some cases registration alone is not enough, there is a gatekeeper who decides whether you qualify. And in extreme cases there is a paywall as well.

Subscriptions can therefore be open or closed, paid or unpaid. But they remain subscriptions. Choices made by the subscriber.

As a subscriber I now receive information. And I’d like that information to have certain characteristics. One, I want that information available to me wherever I am, whenever I am, whatever device I am using. [And I don’t want to have to pay multiple times for the same information as it goes through format transformation]. Two, I want to be able to annotate that information, add notes, tags, links, images, whatever. Three, I want to be able to share that information, via twitter, facebook, google+, chatter, whatever; as in the case of opensource licences, it makes sense that I have to share-alike, share the information with the same constraints under which it was shared with me. Four, I want to be able to filter that information very granularly: for example, I may want to follow a person on twitter, but only for her music tweets; I may want to follow someone else on Google+, but for everything but his music posts. Five, I want to be able to persist the bookmarks, tags, shares, links, pointers, whatever, somewhere, so that I can recreate, “play back”, the shared information, for a specific date or range of dates, and with specific filters.

Simple, isn’t it?

So what should I do as a publisher? Even simpler. As a publisher I need to be able to share what I share in such a way that subscribers can do what I’ve described.

And infrastructure providers have an even simpler job: all they have to do is to provide the tools that publishers and subscriber need to do what I’ve described.

For too long, we’ve kept looking at all this from the viewpoint of the publisher. The publisher in each of us has to work much harder at publishing in a way that makes it easier for others to subscribe. To filter us out when needed. To find us easily when needed. To aggregate us, synthesise us, annotate us, edit us to shreds. Platform-independent, location-independent, device-independent. As private as the information requires us to be, and no more. As public as the information requires us to be, and no more.

So when I tweet, I don’t want to restrict what I tweet about. I want to be able to tweet about food, about cricket, about etymology, about idiocy, about work, about me, about my beliefs, whatever. I want to be able to blog about all these as well. And write books about all this, speak on the topics, and so on.

And I want to be able to do all this in such a way you can find me when you want to, block me when you want to, block me by subject, block me for a time, only follow me for a narrow subset of what I do. It has to be your choice. And the infrastructure I use has to be able to do this. It has to be able to let me do all this, so that you can do what you want out of it.

Which is where the fun begins. Because you don’t think of all this as a winner-takes-all arms race. Because we don’t this of all this as an arms race where we have to choose between the Betamax of Google+ and the VHS of Facebook.

Each of us, as subscribers, will choose how and where and when we will subscribe.

Each of us, as publishers, will choose the environment and infrastructure that most suits us to do what the subscribers want.

So federation and sharing between social networks is unavoidable. A multiplicity of such networks will exist, for cultural, technical and style reasons. One size will never fit all, when we’re seven billion people.

And each social network will come with its tools for sharing, for publishing, for subscribing, for filtering, for helping filter.

For helping filter.

Which is what this post has been about. What do I have to do in order to make it easier for you to find me or block me, find this post or block it, save it or share it, add to it or shred it?

Because you will decide. You will decide whether it’s Betamax or VHS or both or something else as well.

We can only fix filter failure by providing subscribers with better filters, by providing publishers with tools that allow subscribers to filter better.

More later.