Skip to content


On firehoses and filters: Part 2

Note: This is a follow-up to an earlier post on the subject, written in May this year. You may find it worth the while to read that one first. But if you don’t feel like it, no problem. This post is readable standalone.

I love the very concept of publish-subscribe: if you search for the term in this blog, you’ll find I’ve written maybe a dozen posts on the subject over the last six years or so. So I thought it would be worth talking about filters and firehoses in that context.

So let me start with the three “laws” of information filtering that I laid out in that earlier post: (if you want to know why, please read the post; I’ve linked it it earlier in this post)

  1. Where possible, avoid filtering “on the way in”; let the brain work out what is valuable and what is not.
  2. Always filter “on the way out”: think hard about what you say or write for public consumption, why you share what you share.
  3. If you must filter “on the way in” then make sure that the filter is at the edge, the consumer, the receiver, the subscriber, and not at the source or publisher.

Yes, as Clay Shirky put it, we don’t live in an age of information overload; rather, we live in an age of filter failure.

So everyone’s been looking for better filters.

Which is fine.

What’s not fine is when we expect the publishers to do the filtering. Because that allows bad actors to come and spoil the party, whether they’re bad corporations or bad governments.

It’s not the job of the publisher to do the filtering. It should not be the job of the publisher to do the filtering.

But there is a job for the publisher to do. And that is to provide the tools by which subscribers can filter.

Let me expand on this. [Incidentally, when I started using Google+, I raised this issue under the guise of asking for "circles" to be built by publishers as well as by subscribers.... with some interesting discussions and comments as a result].

We live in a world of publish-subscribe, and this world has three facets.

There is an infrastructure, allowing individuals and aggregations of individuals to publish stuff, and to subscribe to stuff.

We exist as publishers, sharing stuff on this infrastructure.

We also exist as subscribers, sharing stuff on this infrastructure.

So now let me look at what I want to do as a subscriber. I want to choose whom and what I “follow” or subscribe to. Most of the time, I expect to be allowed to follow whatever I choose. Sometimes the publisher places a restriction on following, permission is needed. You may just have to ask for permission, register in some form or the other. In some cases registration alone is not enough, there is a gatekeeper who decides whether you qualify. And in extreme cases there is a paywall as well.

Subscriptions can therefore be open or closed, paid or unpaid. But they remain subscriptions. Choices made by the subscriber.

As a subscriber I now receive information. And I’d like that information to have certain characteristics. One, I want that information available to me wherever I am, whenever I am, whatever device I am using. [And I don't want to have to pay multiple times for the same information as it goes through format transformation]. Two, I want to be able to annotate that information, add notes, tags, links, images, whatever. Three, I want to be able to share that information, via twitter, facebook, google+, chatter, whatever; as in the case of opensource licences, it makes sense that I have to share-alike, share the information with the same constraints under which it was shared with me. Four, I want to be able to filter that information very granularly: for example, I may want to follow a person on twitter, but only for her music tweets; I may want to follow someone else on Google+, but for everything but his music posts. Five, I want to be able to persist the bookmarks, tags, shares, links, pointers, whatever, somewhere, so that I can recreate, “play back”, the shared information, for a specific date or range of dates, and with specific filters.

Simple, isn’t it?

So what should I do as a publisher? Even simpler. As a publisher I need to be able to share what I share in such a way that subscribers can do what I’ve described.

And infrastructure providers have an even simpler job: all they have to do is to provide the tools that publishers and subscriber need to do what I’ve described.

For too long, we’ve kept looking at all this from the viewpoint of the publisher. The publisher in each of us has to work much harder at publishing in a way that makes it easier for others to subscribe. To filter us out when needed. To find us easily when needed. To aggregate us, synthesise us, annotate us, edit us to shreds. Platform-independent, location-independent, device-independent. As private as the information requires us to be, and no more. As public as the information requires us to be, and no more.

So when I tweet, I don’t want to restrict what I tweet about. I want to be able to tweet about food, about cricket, about etymology, about idiocy, about work, about me, about my beliefs, whatever. I want to be able to blog about all these as well. And write books about all this, speak on the topics, and so on.

And I want to be able to do all this in such a way you can find me when you want to, block me when you want to, block me by subject, block me for a time, only follow me for a narrow subset of what I do. It has to be your choice. And the infrastructure I use has to be able to do this. It has to be able to let me do all this, so that you can do what you want out of it.

Which is where the fun begins. Because you don’t think of all this as a winner-takes-all arms race. Because we don’t this of all this as an arms race where we have to choose between the Betamax of Google+ and the VHS of Facebook.

Each of us, as subscribers, will choose how and where and when we will subscribe.

Each of us, as publishers, will choose the environment and infrastructure that most suits us to do what the subscribers want.

So federation and sharing between social networks is unavoidable. A multiplicity of such networks will exist, for cultural, technical and style reasons. One size will never fit all, when we’re seven billion people.

And each social network will come with its tools for sharing, for publishing, for subscribing, for filtering, for helping filter.

For helping filter.

Which is what this post has been about. What do I have to do in order to make it easier for you to find me or block me, find this post or block it, save it or share it, add to it or shred it?

Because you will decide. You will decide whether it’s Betamax or VHS or both or something else as well.

We can only fix filter failure by providing subscribers with better filters, by providing publishers with tools that allow subscribers to filter better.

More later.

 

Posted in Filtering.


14 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

  1. Matthias says

    The culminating (bold) sentence of your article has a typo: Because we don’t -this- think. That’s too bad. Feel free to delete this comment after correcting.

  2. Brian Driggs says

    I’m afraid I’m a bit confused, JP.

    You start by saying we should filter on output, but not input, as we remain open to new possibilities. Then you close by saying we should provide subscribers with better filters. Truly, every piece of information is both input/output at the same time. Is this a function of differentiating roles (ie: publisher/subscriber)?

    I’m thinking of a comment I just left on Brian Solis’s blog about Foursquare and 10MM users. Personally, the platform and it’s often useless updates in my stream are an annoyance causing me to reconsider following people on Twitter (when those 4sq updates go out on their streams). Should these people be filtering their output, perhaps being more considerate of their followers across platforms? Or should they be able to share any and all updates as part of their social experience and let their followers (subscribers) filter?

    I, for one, would love an application wherein I could participate in all my social streams – G+, Twitter, FB, etc. – and selectively filter content by APPLICATION over USER. It’s unfortunate our relationships with people online are subject to the self-serving broadcast aims of each publisher/platform they wish to use.

    So, input/output, publish/subscribe, how do we differentiate?

  3. leon says

    Like all people I just want exactly what I want in a really easy to get way…to use JP’s example, I want to be able to consume all his tweets, but when he’s engaged in a long music listening session, I wouldn’t mind being able to just see one of those posts in my stream…

    Facebook has been spectacularly successful by just getting UX, and an arms race with google can only be good for the consumer…

    If the tools make it easy for me to see whether a post is “public”, “family”, “cricket” or “what music i’m listening to” – google+ is getting there but it seems to miss one specific use-case of prolific bloggers and public commentators who have families and friends who don’t want to see *everything* they post. They will doubtless get there…

    As a slight aside, my interest in this particular area stems from my love of wine…cellartracker (a wonderful project run by the brilliant Eric LeVine) is a crowdsourced database of wines with tasting notes. Each time you buy or consume wine or post a tasting note its integration tools allow you to post to twitter or facebook – I found this a hoot but soon started getting complaints from fb friends that I was spamming them….now how any right-thinking person can see an exciting purchase of 97 Rousseau Clos de la Roche is beyond me, but apparently these people exist. Circles seems like a perfect way to get around this.

  4. clive boulton says

    16-24 seems to be choosing Tumblr for effortless social filtering.

    Until existing blogs get effortless pub/sub filters it’s easier to find or block Confused of Calcutta on Google Reader. Although I’m intrigued by JP’s Google+ Circle experiments.

  5. Douglas says

    “We don’t live in an age of information overload; rather, we live in an age of filter failure.”

    So true

  6. John Tropea says

    Some more thinking here on algorithms http://johntropea.tumblr.com/post/7255182502

    I also posted about this in an organisational context, here’s an excerpt:
    http://johntropea.tumblr.com/post/3454440340

    “We don’t want to smash silos, we don’t want the absurd notion of replacing hierachies with personal networks…what we want is for them to officially co-exist as they both have benefits.

    I think the best we can do is for silos to work in the open ie. open online silos so people can visit and subscribe…and for people to connect online so they can be ambiently aware…the onus is on a person to subscribe to people and other units (who openly narrate and discuss their work)

    ie. publish (post) / subscribe or follow…rather than sender and receiver

    How do I know who will benefit from my work or group…who are the receivers I should make aware of my work
    – This doesn’t work and email is based on this
    – I mean sure, you can do this to a certain degree…but you can’t possibly know in an org of 8000 people who can benefit from being informed about what you are doing right now
    – Instead the onus is on people to tap into you (as long as the “you” is findable ie. you narrate your work, you do tasks online…this is observable work via digital means as we all don’t work in the same office where we can’t help bumping into each other and noticing work in progress, as like would happen in Andrew’s office of 12 people)”

Continuing the Discussion

  1. Google Plus – its really all about *you* | Storage according to a dixie chick linked to this post on July 11, 2011

    [...] the firehose of information that comes at you from the people you follow.  Some people think this filtering should be done from the application side, but I think its something that has to be done on a personal [...]

  2. Today’s Shared Links for July 10, 2011 – Chuqui 3.0 linked to this post on July 11, 2011

    [...] On firehoses and filters: Part 2 [...]

  3. IBM'ers poles apart on Google Plus | ZDNet linked to this post on July 12, 2011

    [...] Like JP Ranagswami, I want to effortlessly filter in the same manner I do in the real world and not worry about what I might miss going on in the firehose while at the same time handling the in-moment requirements I have today. And I want it all archivable and searchable so that I can go back and reference what went before because history matters and memories are short. [...]

  4. Update on Filters linked to this post on July 17, 2011

    [...] work much better when you combine your activity as publisher with tweaking your preferences as reader/learner. G+ provided the answer I found so useful on FriendFeed to fine tune what comes [...]

  5. The democratization of communication: In filter we trust | TechDiem.com linked to this post on July 23, 2011

    [...] the proverbial firehose and drowning in the ensuing deluge. Or, as Clay Shirky put it (by way of JP Rangaswami, who writes eloquently about the subject), we don’t live in an age of information overload, but of “filter failure.” Quite [...]

  6. The democratization of communication: In filter we trust — Tech News and Analysis linked to this post on July 23, 2011

    [...] the proverbial firehose and drowning in the ensuing deluge. Or, as Clay Shirky put it (by way of JP Rangaswami, who writes eloquently about the subject), we don’t live in an age of information overload, but of “filter failure.” Quite [...]

  7. The democratization of communication: In filter we trust | Information Technology | Technology News linked to this post on July 23, 2011

    [...] the proverbial firehose and drowning in the ensuing deluge. Or, as Clay Shirky put it (by way of JP Rangaswami, who writes eloquently about the subject[5]), we don’t live in an age of information overload, but of “filter failure.” Quite [...]

  8. E L S U A ~ A KM Blog Thinking Outside The Inbox by Luis Suarez » It’s Not Filter Failure, But Thought For Food linked to this post on May 10, 2012

    [...] filter failure” at the Web 2.0 Expo in New York, back in 2008, there has been an ongoing, everlasting, but rather passionate discussion from both sides of the story pondering whether it’s [...]



Some HTML is OK

or, reply to this post via trackback.



%d bloggers like this: