Author: JP

Note: This is the third in a series of posts I’m committed to writing about filters; I started with the principles of filtering, and will proceed to blow up each of the principles in as much detail as makes sense at this stage. Earlier I looked at network-based filters. Today I want to spend time on routing.

When I talk about routing insofar as filters are concerned, I’m talking about three things. One, the time when the filtered message is delivered to the subscriber. Two, the place that it is delivered. And three, the device to which it is delivered. Nothing less, nothing more. There are more things to consider but I want to keep it simple for now.

Some years ago I was on vacation in Austin, Texas. We used to love going there every summer, when everyone else appeared to leave Austin. We tended to stay at the Barton Creek Resort there, unkind people would call it Connally’s Folly.

I was woken up in the middle of the night, around 130am, by the sound of a car alarm going off. Tried to go back to sleep, but couldn’t. The alarm wouldn’t stop. Then I noticed my wife had been awakened as well. So I muttered to her that I was going to ring the hotel reception and give them a piece of my mind and ask them to do something about this pesky alarm. She took all this in quietly. I should have known something was up. Then, just as I picked up the phone to call reception, she spoke.

JP, I think the alarm is coming from you.

That stopped me in my tracks. As usual, she was right. It was my pacemaker. It sounded like I had swallowed a tiny ambulance which was then complaining, sirens blazing, about the way it was being treated.

I tried to remember what I’d been told. Three types of alarms. One that let me know my battery was running low, and that I had to check in within the next month or so. A second that said it was a problem with my leads, could be a disconnection, could be fluid buildup. And a third which said drop everything and see a doctor immediately. Which one was it? Which sound was which? What was I meant to do?

It sounded like type 2, lead or fluid problems, but the timing didn’t make sense. I had been assured that type 1 and type 2 alarms went off sedately around 830am, so that I knew not to panic. Why, if it was type 2, was it going off at 130am?

And then I thought about it. The device was probably somehow hard wired to think it was in London, and that it was December. Made sense, I’d had it implanted in December in London. But I wasn’t in London. And it wasn’t December. I was in Austin, in late July. And 130am in Austin in July was ….. 830am in London in December.

Problem solved. I slept, and slept well. And woke up at an earthly hour, rested and refreshed, spoke to the cardiologist back in London and all was well.

Context matters. We live in an age where we expect alerts to be delivered to us sensitive to the time and place of delivery, all the more because the cost of knowing where we are and what time it is there is trivial.

When we convert firehoses into value, the filter process needs to bear this in mind. Discover location and time where possible, test for delivery conditions in consequence.

There are other important considerations, more social than anything else. In the early days of smart devices, we used to have this strange phenomenon jocularly referred to as Blackberry Prayer. This was where you were seated, in company, and then proceeded to clasp your hands and stare at your crotch for a few minutes, in the benighted belief that no one else would notice what you were doing.

As we move into wearable computing, the social side of what we do will start mattering more. You know how it is when you’re chatting with someone and you realise that he/she gets distracted every time someone enters the room? There are people who just have to “work the room”, and many of them don’t realise how discourteous it is, and what signal they’re giving you. Some of them don’t care about the signal they’re giving, but that’s another matter.

Soon, with wearables, particularly with glass-style devices, we run the risk of regular inadvertent dissing, as you watch the person you’re in conversation with “wipe the windscreen” of their eye-borne device.

In social company, it’s one thing to glance surreptitiously at your watch, or to take a quick gander at your phone. It’s something else altogether to pull out your tablet and start reading it in public. Whatever you do, your engagement with the message has to be quick, which means it has to be short. If it cuts across conversation, it had better be urgent as well as important. Say if your partner or child was in hospital… Then people would understand.

Context matters.

As we continue to separate signal from the stream, we’re going to have to learn a lot about conditional routing. What IF THIS THEN THAT or IFTTT seeks to solve. Years ago, when I was at BT and we were talking about the connected home, the example I would use to explain things ran as follows:

If the doorbell rings and you know that the only person at home is your aged aunt who happens to be hard of hearing, then please make the light that’s near the TV blink a few times because that’s the only way she’ll know that someone is ringing the doorbell.

The doorbell is a node on the network. The light near the TV is a node on the network. So is the hard-of-hearing aged aunt.

Firehoses exist because everything can publish. Filters exist so everyone can subscribe.

As we learn more about conditional routing, we will see that the principles behind stuff like IFTTT will pervade our consciousness in ways we haven’t considered. Children will learn to write the code needed to make conditional routing happen, just like they learnt to program VCRs and work their smart devices. These won’t be specialist things.

And as this happens the routing mechanisms will learn to know more not just about person and time and place but device as well. Form factors, graphic capabilities, likely locations of use, all these will matter and form part of what we need from filters.

Why do we filter? Because we want to get the right information to the right person at the right time in the right location and to the right device.

Who chooses all these “right” things? The subscriber. How can these choices be made possible? By designing filters than subscribers can use to make these choices, using all the contextual information possible.

It’s past one am, it’s been a long day, so I’ll sign off for now. If you want to see more of this, then do let me know that you’re finding this series useful.

Note: This is a continuation of my earlier post Filtering: Seven Principles. Over the next few weeks I hope to expand on each of the principles, adjusting and refining as I learn from your comments, observations and guidance.

Learning from email

There was a time when I liked email. A time when it was quick and informal, when typing in lowercase was fine, abbreviations were in common use, messages tended to be short, externally-initiated spam was very rare and internally-initiated “corporate” spam was but a glimmer in centralist eyes. That was a long time ago.

Over time email became more and more formal; as happens in so many cases, there was a tendency to force-fit the future into the construct of the past, a variant of paving cowpaths. Soon there were formal beginnings and endings, names and addresses and dates; layouts started imitating snail-mail. The carbon-copy of the past, a useful way of keeping a copy of what was sent, became the cc ass-c0ver of the present; strangely, even though most mail systems had a Save Sent Mail function, the cc persisted: probably because of the sheer gravity of the asses being covered. I trust you so much that I’ll keep copying your boss in when I talk to you. Worse was to come. The blind copy “bc” button, a means to solve propagating distribution list contents, was subverted into something far more insidious: I trust you so much that I’ll copy your boss in to our conversation without telling you.

As mail became an enterprise utility, more and more of its collaborative function was corrupted, as signalled above. It could not be a trusted medium with functions like cc and bc in common use. There were other problems. Email was fundamentally a broadcast mechanism. Control lay in the hands of the sender. And there was no real cost to sending. Unless there was some meaningful price or penalty, spam was inevitable, both external as well as internal. Furthermore, threading was not always available, so discussions became fragmented and hard to follow. As people began to use attachments, storage vendors chortled in their joy and version mismatch became a common problem in meetings. Which presentation are you looking at? That’s not what my slide 5 says.

Fragmented conversations were a real problem in other ways. Hierarchical organisations have inbuilt frictions, and as they scale the risk of internal politics increases. In such organisations, the fragmentation caused by email sometimes takes a darker route. Person A sends an email to a group of people. Some of them reply-all, seeing that it is the right thing to do. A few others then corrupt the conversation, by taking a few people off the recipient list and adding a few more, with liberal doses of cc and bc. Before you know it there are now multiple conversations with different carefully-chosen groups of people, with only a few, usually politically-motivated, members playing puppetmaster to all the conversations. Cut-and-paste then comes into play, as segments of one set of conversations get viewed in other, exclusive, environments.

And then we have the ultimate, the Infinite Loop. The phenomenon that grinds decision-making to a halt as people strive to obtain consensus via mail. I was on vacation at the time. I didn’t see that message. Constant re-openings of the same debate as people try and get a synchronous outcome out of an asynchronous tool without the agreements and conventions in place to do it. Sometimes I think that Infinite Loopery is the single biggest cause of male pattern baldness. Tear your hair out time.

When we think about a world where everyone is connected, where everything is a node on the network, where every node can publish and subscribe, we can understand the need for the stream/filter/drain architecture. When I speak of network-based filters in this context, it behooves me to view the stream as a successor to mail, at least to begin with.

The network as filter

People use “social” to mean many things. A social worker; social sciences; social media; the social enterprise. I am not here to debate all these meanings or fight for one or the other. One of the joys of any language is the natural ambiguity that uses context to help discover meaning. For example, I love the way that in many Indian Sanskrit-based language, the word for yesterday is the same as the word for tomorrow.

I think of social as a filter. Let me explain.

In email days, if I went on vacation, my inbox would pile up agonisingly. A week away meant a few thousand emails to read. And to respond to, given that social conventions now expect you to answer all emails. One of my erstwhile colleagues, Stu Berwick, when talking about different modes of communications, remarked that instant messaging was unusual in that it was “polite to be silent”. When new modes of communication emerge, this is often the case. It was so with email as well.

Back to me on vacation. Thousands of emails in mailbox when I return. What do I do? Start with the oldest, onslaught of replies, frustration level growing as I see later mails on the same subject, mails which would have had me reply differently. So with that experience go on to plan B, start with the newest. Same problem, because the conversation thread is not quite integral. Fragmentation frustration.

So what’s the solution? More precisely, what was the solution? For many people, it was this. Feign some mild stomach disorder or plumbing problem. Go regularly to loo. Sneak a quick look at the blackberry (remember them). Respond to urgent and important items before they complexify.

That was in a publisher-has-power world. Today’s social-network firehose-stream world is different. You choose whom you follow.

Amongst the people you follow is this class of person called your friend. At work and at play, in business as well as in personal life.

These friends know you, know what’s important to you. Sometimes they even know what’s important to you despite your not recognising or acknowledging that importance.

These friends are your social filters. You no longer have to read every email. When you come back from vacation, whatever has passed in the stream unread can stay unread.

Why? Because you have a network of friends. They will DM you or private message you about the things that are important. They will SMS you or text you or IM you or Whatsapp you about the things that are urgent.

And they, as a friends collective, will RT and +1 and Like stuff as well. As part of an even greater collective, the firehose, they will make things trend.

You get all those filtering benefits. Because it’s based on subscriber power and not publisher power, the spam risk is lowered. (Although I am sure there are organisations that mandate your following someone or something and thereby creating network spam).

Your friends will tell you what you missed, which conversations you need to be part of. What’s important. What’s urgent. What’s trending. What’s not.

Your friends, people you trust. People who trust you. People who know you. As individuals and as a collective. People with whom you have a relationship, with the investment of time and effort. A relationship, with all the openness and vulnerability that brings.

Your friends. A very powerful filter. A very very powerful filter.

Your friends don’t just filter in this form. They annotate, comment on, rate,review, recommend every digitally shared social object there is. More on that later.

They solve other problems as well, problems that mechanical filters need help with. Ontologies and taxonomies. You say tomayto. Tagging, hashtags, semantic notation. Integral components of sense making in this flood of information.

Collaborative filters are just the icing on this cake, the ability to discover and be educated by patterns. People who did this also did.

More on this later. Please keep the comments and observations coming.

At least three of the books below made my reading list because one of you told me it was worth it…..So here’s my current batch of ten, just in case they make you think of something else I should be reading. Who knows, you may find something of interest there as well.

 

• Mr Penumbra’s 24-Hour Bookstore. Robin Sloan. Nearly finished. Relishing it, slowing it down as I approach the end.
• Gray Men. Tomotake Ishikawa. An intriguing Noir Anonymous-era novel, translated from the Japanese, written by someone from my children’s generation. Midway.
• Eminent Hipsters. Donald Fagen. Too much of a Steely Dan fan to miss this one. But haven’t started it yet.
• Shoot The Woman First. Wallace Stroby. His two previous books were both Kirkus-starred. Rare. I liked them both. Just started this one.
• No Man Is An Island. Thomas Merton. Recommended by a friend after I wrote the Forgiveness post. Loving it. Almost done.
• Pure, White and Deadly. John Yurkin. Been fascinated/revolted by the sugar vs fats vs industry vs regulator shenanigans. Had to check this ’72 warning. Part way.
• Trust Me, I’m Lying. Ryan Holliday. Recommended by a reader, focused on how people game today’s media, social and otherwise. Unstarted.
• The Lowland. Jhumpa Lahiri. How can I not read a novel set at least partly in the Calcutta of my youth? Just getting into it.
• Steps to An Ecology of Mind. Gregory Bateson. Reading it for the second time, slowly. Will probably be reading it for the next six months. Need to understand it better.
• The Burglar Who Counted The Spoons. Lawrence Block. Been waiting patiently for this Rhodenbarr. Holding it off till my next daytime flight to San Francisco.

In earlier posts towards the tail end of last year and early this year, I committed to writing a number of posts on filtering. The background is simple:

• soon, everything and everyone will be connected
• that includes people, devices, creatures, inanimate objects, even concepts (like a tweet or a theme)
• at the same time, the cost of sensors and actuators is dropping at least as fast as compute and storage
• so that means everything and everyone can now publish status and alerts of pretty much anything
• there’s the potential for a whole lotta publishing to happen
• which in turn means it’s firehose time
• so we need filters
• which is why the stream/filter/drain approach is becoming more common
• and which is why I want to spend time on all this during 2014, starting with the filter

So here goes.

1. Filters should be built such that they are selectable by subscriber, not publisher There should be no publisher-level filters. Allow the firehose to happen. We know how to solve the firehose. What we don’t know is how to solve a much bigger problem: what to do when there are filters at publisher level. Once you allow this, the first thing that happens is that an entry point is created for bad actors to impose some form of censorship. In some cases it will be governments, sometimes overtly, sometimes covertly; at other times it will be traditional forces of the media; it may be generals of the army or captains of industry. The nature of the bad actor is irrelevant; what matters is that a back door has been created, one that can be used to suppress reports about a particular event/location/topic/person. If we keep making sure that it’s not easy to filter at publisher level, the bad actor is left with the strain of large-scale filtering of firehoses. Not easy.

 

2. Filters should intrinsically be dynamic, not static In keeping with the firehose that’s being filtered, the act of filtering should itself be one of flows and not stocks. There is a place for canned filters, to support trend analysis, pattern recognition, predictive analytics. But the norm should be that the subscriber can reset filters anytime without any loss of time or value.

 

3. Filters should have inbuilt “serendipity” functionality Have you ever chosen “random” when presented with a choice of things to look at, to listen to, to read, to follow? It’s a simple insurance policy to take out in order to avoid digital bigotry or heretical thinking or tunnel vision or herd instinct or groupthink or whichever other buzzphrase a la mode excites you. You must have something that takes you outside the pattern of what you do normally. And you must be able to switch that something on at will. The StumbleUpon approach is useful, but since you can “train” it you run the risk of “filter bubble” in Eli Pariser terms. In fact any publisher-level filter can create a filter bubble. Which, at its worst, allows someone else to determine what you can see/touch/listen to/engage with.

 

4. Filters should be interchangeable, exchangeable, even tradeable I should be able to give someone else my collection of filters; similarly, someone should be able to give me their filter set. Their transient filter set, nothing permanent as I said earlier. The idea is that one person is given an opportunity to engage with the firehoses of the digital universe while “walking in someone else’s shoes”. So I should be able to view news as if I was a 21-year old Iranian. Not by selecting the publisher-side filter for “21 year old Iranians” but by being able to exchange filters with a real live person who has those characteristics. Again, we need to watch for static, hierarchical filters and avoid them like the plague.

 

5. The principal filters should be by choosing a variable and a value (or range of values) to include or exclude The variable could be anything. Place. Time. Person. group. Topic. Temperature. Degree of wetness. Humidity. Blood pressure. Relative density. Weight. State. Number or count. Size. Type. Part number. SIC or NACE code. Tag. Hashtag. Label. Length. Material. Language. Species. Duration. Anything and everything. And filtered again, if needed, by the associated value. Hotter than. Lighter than. Higher than. Containing. And then filtered again for inclusion or exclusion.

 

6. Secondary filters should then be about routing This is where the concepts behind If This Then That come into their own. The universe that IFTTT represents is one of conditional filtering and routing. The filtered information, having passed the conditions set, needs to go somewhere. Devices now form part of the world of filters. A person who has a laptop, tablet, phone and wearable does not want the same filters for each, the same notifications to each. For one thing, the social conventions for each form factor are different; for a second, the readability and “actionability” will differ as well. So we will use IFTTT and similar constructs to filter by notification type, intensity, device, perhaps even recipient time of day and location.

 

7. Network-based filters, “collaborative filtering” should then complete the set Collaborative filtering is also critical. Show me the tweets that are trending with my friends that I haven’t seen yet. Let me know the restaurants frequented by people in my network who like spicy food and who’ve posted on TripAdvisor about those restaurants in the last six months; make it relevant to my location and the current time.

 

So that’s a starter set, seven principles that inform me when I think about these things. I shall expand on each in days to come. In the meantime, keep your observations, advice, questions and comments flowing, choosing whichever means or channel you prefer. Comment here. Respond to the link in Twitter, Facebook, Google+, LinkedIn. WhatsApp me if you want. Talk to me via @jobsworth. If you don’t like any of these, then I suppose you can email me via [email protected] but be warned that I look at email rarely and that too only under duress.

Last January, around this time, as people came back from hibernation into the plastic warmth and sparkle of CES, we heard about luggage that can tweet. Hello, I’m here. In Rio. While you wait for it. In Riyadh.

Trakdot was one of the stories I latched upon at that time.

You could put a Trakdot into your sharkskin suitcase and bingo, not only could it tell you where it was, it could also tell you when it approached you. Via SMS if needed. [Incidentally, no sharks were harmed in producing the sharkskin suitcase, it’s not real sharkskin, just in case you were getting ready to complain].

A year on the story gets better. Why bother with mock sharkskin when you can have the real thing? Real sharkskin, still worn by the shark, while it’s alive and breathing?

Apparently over 300 sharks have been tagged and equipped to let bathers and swimmers know of their proximity. And many other things besides.

Everything is a node on the network.Even sharks.

Everything can publish, everything can subscribe.

Which means firehoses and filters.

That’s why all this publishing and subscribing takes place against thresholds we set. Like location. Or size. Or temperature. Or time. Or whatever. In whatever combination. Static as well as dynamic.

Over the next few weeks, as I spend more time writing about filtering in an age of streams and drains and filters, the tweeting shark is a good place to start.

What a difference a year makes.