Filters: Part 2: Thinking about the network as filter

Note: This is a continuation of my earlier post Filtering: Seven Principles. Over the next few weeks I hope to expand on each of the principles, adjusting and refining as I learn from your comments, observations and guidance.

Learning from email

There was a time when I liked email. A time when it was quick and informal, when typing in lowercase was fine, abbreviations were in common use, messages tended to be short, externally-initiated spam was very rare and internally-initiated “corporate” spam was but a glimmer in centralist eyes. That was a long time ago.

Over time email became more and more formal; as happens in so many cases, there was a tendency to force-fit the future into the construct of the past, a variant of paving cowpaths. Soon there were formal beginnings and endings, names and addresses and dates; layouts started imitating snail-mail. The carbon-copy of the past, a useful way of keeping a copy of what was sent, became the cc ass-c0ver of the present; strangely, even though most mail systems had a Save Sent Mail function, the cc persisted: probably because of the sheer gravity of the asses being covered. I trust you so much that I’ll keep copying your boss in when I talk to you. Worse was to come. The blind copy “bc” button, a means to solve propagating distribution list contents, was subverted into something far more insidious: I trust you so much that I’ll copy your boss in to our conversation without telling you.

As mail became an enterprise utility, more and more of its collaborative function was corrupted, as signalled above. It could not be a trusted medium with functions like cc and bc in common use. There were other problems. Email was fundamentally a broadcast mechanism. Control lay in the hands of the sender. And there was no real cost to sending. Unless there was some meaningful price or penalty, spam was inevitable, both external as well as internal. Furthermore, threading was not always available, so discussions became fragmented and hard to follow. As people began to use attachments, storage vendors chortled in their joy and version mismatch became a common problem in meetings. Which presentation are you looking at? That’s not what my slide 5 says.

Fragmented conversations were a real problem in other ways. Hierarchical organisations have inbuilt frictions, and as they scale the risk of internal politics increases. In such organisations, the fragmentation caused by email sometimes takes a darker route. Person A sends an email to a group of people. Some of them reply-all, seeing that it is the right thing to do. A few others then corrupt the conversation, by taking a few people off the recipient list and adding a few more, with liberal doses of cc and bc. Before you know it there are now multiple conversations with different carefully-chosen groups of people, with only a few, usually politically-motivated, members playing puppetmaster to all the conversations. Cut-and-paste then comes into play, as segments of one set of conversations get viewed in other, exclusive, environments.

And then we have the ultimate, the Infinite Loop. The phenomenon that grinds decision-making to a halt as people strive to obtain consensus via mail. I was on vacation at the time. I didn’t see that message. Constant re-openings of the same debate as people try and get a synchronous outcome out of an asynchronous tool without the agreements and conventions in place to do it. Sometimes I think that Infinite Loopery is the single biggest cause of male pattern baldness. Tear your hair out time.

When we think about a world where everyone is connected, where everything is a node on the network, where every node can publish and subscribe, we can understand the need for the stream/filter/drain architecture. When I speak of network-based filters in this context, it behooves me to view the stream as a successor to mail, at least to begin with.

    The network as filter

People use “social” to mean many things. A social worker; social sciences; social media; the social enterprise. I am not here to debate all these meanings or fight for one or the other. One of the joys of any language is the natural ambiguity that uses context to help discover meaning. For example, I love the way that in many Indian Sanskrit-based language, the word for yesterday is the same as the word for tomorrow.

I think of social as a filter. Let me explain.

In email days, if I went on vacation, my inbox would pile up agonisingly. A week away meant a few thousand emails to read. And to respond to, given that social conventions now expect you to answer all emails. One of my erstwhile colleagues, Stu Berwick, when talking about different modes of communications, remarked that instant messaging was unusual in that it was “polite to be silent”. When new modes of communication emerge, this is often the case. It was so with email as well.

Back to me on vacation. Thousands of emails in mailbox when I return. What do I do? Start with the oldest, onslaught of replies, frustration level growing as I see later mails on the same subject, mails which would have had me reply differently. So with that experience go on to plan B, start with the newest. Same problem, because the conversation thread is not quite integral. Fragmentation frustration.

So what’s the solution? More precisely, what was the solution? For many people, it was this. Feign some mild stomach disorder or plumbing problem. Go regularly to loo. Sneak a quick look at the blackberry (remember them). Respond to urgent and important items before they complexify.

That was in a publisher-has-power world. Today’s social-network firehose-stream world is different. You choose whom you follow.

Amongst the people you follow is this class of person called your friend. At work and at play, in business as well as in personal life.

These friends know you, know what’s important to you. Sometimes they even know what’s important to you despite your not recognising or acknowledging that importance.

These friends are your social filters. You no longer have to read every email. When you come back from vacation, whatever has passed in the stream unread can stay unread.

Why? Because you have a network of friends. They will DM you or private message you about the things that are important. They will SMS you or text you or IM you or Whatsapp you about the things that are urgent.

And they, as a friends collective, will RT and +1 and Like stuff as well. As part of an even greater collective, the firehose, they will make things trend.

You get all those filtering benefits. Because it’s based on subscriber power and not publisher power, the spam risk is lowered. (Although I am sure there are organisations that mandate your following someone or something and thereby creating network spam).

Your friends will tell you what you missed, which conversations you need to be part of. What’s important. What’s urgent. What’s trending. What’s not.

Your friends, people you trust. People who trust you. People who know you. As individuals and as a collective. People with whom you have a relationship, with the investment of time and effort. A relationship, with all the openness and vulnerability that brings.

Your friends. A very powerful filter. A very very powerful filter.

Your friends don’t just filter in this form. They annotate, comment on, rate,review, recommend every digitally shared social object there is. More on that later.

They solve other problems as well, problems that mechanical filters need help with. Ontologies and taxonomies. You say tomayto. Tagging, hashtags, semantic notation. Integral components of sense making in this flood of information.

Collaborative filters are just the icing on this cake, the ability to discover and be educated by patterns. People who did this also did.

More on this later. Please keep the comments and observations coming.

What I’m reading at present

At least three of the books below made my reading list because one of you told me it was worth it…..So here’s my current batch of ten, just in case they make you think of something else I should be reading. Who knows, you may find something of interest there as well.

 

  • Mr Penumbra’s 24-Hour Bookstore. Robin Sloan. Nearly finished. Relishing it, slowing it down as I approach the end.
  • Gray Men. Tomotake Ishikawa. An intriguing Noir Anonymous-era novel, translated from the Japanese, written by someone from my children’s generation. Midway.
  • Eminent Hipsters. Donald Fagen. Too much of a Steely Dan fan to miss this one. But haven’t started it yet.
  • Shoot The Woman First. Wallace Stroby. His two previous books were both Kirkus-starred. Rare. I liked them both. Just started this one.
  • No Man Is An Island. Thomas Merton. Recommended by a friend after I wrote the Forgiveness post. Loving it. Almost done.
  • Pure, White and Deadly. John Yurkin. Been fascinated/revolted by the sugar vs fats vs industry vs regulator shenanigans. Had to check this ’72 warning. Part way.
  • Trust Me, I’m Lying. Ryan Holliday. Recommended by a reader, focused on how people game today’s media, social and otherwise. Unstarted.
  • The Lowland. Jhumpa Lahiri. How can I not read a novel set at least partly in the Calcutta of my youth? Just getting into it.
  • Steps to An Ecology of Mind. Gregory Bateson. Reading it for the second time, slowly. Will probably be reading it for the next six months. Need to understand it better.
  • The Burglar Who Counted The Spoons. Lawrence Block. Been waiting patiently for this Rhodenbarr. Holding it off till my next daytime flight to San Francisco.

Filtering: Seven Principles

In earlier posts towards the tail end of last year and early this year, I committed to writing a number of posts on filtering. The background is simple:

  • soon, everything and everyone will be connected
  • that includes people, devices, creatures, inanimate objects, even concepts (like a tweet or a theme)
  • at the same time, the cost of sensors and actuators is dropping at least as fast as compute and storage
  • so that means everything and everyone can now publish status and alerts of pretty much anything
  • there’s the potential for a whole lotta publishing to happen
  • which in turn means it’s firehose time
  • so we need filters
  • which is why the stream/filter/drain approach is becoming more common
  • and which is why I want to spend time on all this during 2014, starting with the filter

So here goes.

1. Filters should be built such that they are selectable by subscriber, not publisher There should be no publisher-level filters. Allow the firehose to happen. We know how to solve the firehose. What we don’t know is how to solve a much bigger problem: what to do when there are filters at publisher level. Once you allow this, the first thing that happens is that an entry point is created for bad actors to impose some form of censorship. In some cases it will be governments, sometimes overtly, sometimes covertly; at other times it will be traditional forces of the media; it may be generals of the army or captains of industry. The nature of the bad actor is irrelevant; what matters is that a back door has been created, one that can be used to suppress reports about a particular event/location/topic/person. If we keep making sure that it’s not easy to filter at publisher level, the bad actor is left with the strain of large-scale filtering of firehoses. Not easy.

 

2. Filters should intrinsically be dynamic, not static In keeping with the firehose that’s being filtered, the act of filtering should itself be one of flows and not stocks. There is a place for canned filters, to support trend analysis, pattern recognition, predictive analytics. But the norm should be that the subscriber can reset filters anytime without any loss of time or value.

 

3. Filters should have inbuilt “serendipity” functionality Have you ever chosen “random” when presented with a choice of things to look at, to listen to, to read, to follow? It’s a simple insurance policy to take out in order to avoid digital bigotry or heretical thinking or tunnel vision or herd instinct or groupthink or whichever other buzzphrase a la mode excites you. You must have something that takes you outside the pattern of what you do normally. And you must be able to switch that something on at will. The StumbleUpon approach is useful, but since you can “train” it you run the risk of “filter bubble” in Eli Pariser terms. In fact any publisher-level filter can create a filter bubble. Which, at its worst, allows someone else to determine what you can see/touch/listen to/engage with.

 

4. Filters should be interchangeable, exchangeable, even tradeable I should be able to give someone else my collection of filters; similarly, someone should be able to give me their filter set. Their transient filter set, nothing permanent as I said earlier. The idea is that one person is given an opportunity to engage with the firehoses of the digital universe while “walking in someone else’s shoes”. So I should be able to view news as if I was a 21-year old Iranian. Not by selecting the publisher-side filter for “21 year old Iranians” but by being able to exchange filters with a real live person who has those characteristics. Again, we need to watch for static, hierarchical filters and avoid them like the plague.

 

5. The principal filters should be by choosing a variable and a value (or range of values) to include or exclude The variable could be anything. Place. Time. Person. group. Topic. Temperature. Degree of wetness. Humidity. Blood pressure. Relative density. Weight. State. Number or count. Size. Type. Part number. SIC or NACE code. Tag. Hashtag. Label. Length. Material. Language. Species. Duration. Anything and everything. And filtered again, if needed, by the associated value. Hotter than. Lighter than. Higher than. Containing. And then filtered again for inclusion or exclusion.

 

6. Secondary filters should then be about routing This is where the concepts behind If This Then That come into their own. The universe that IFTTT represents is one of conditional filtering and routing. The filtered information, having passed the conditions set, needs to go somewhere. Devices now form part of the world of filters. A person who has a laptop, tablet, phone and wearable does not want the same filters for each, the same notifications to each. For one thing, the social conventions for each form factor are different; for a second, the readability and “actionability” will differ as well. So we will use IFTTT and similar constructs to filter by notification type, intensity, device, perhaps even recipient time of day and location.

 

7. Network-based filters, “collaborative filtering” should then complete the set Collaborative filtering is also critical. Show me the tweets that are trending with my friends that I haven’t seen yet. Let me know the restaurants frequented by people in my network who like spicy food and who’ve posted on TripAdvisor about those restaurants in the last six months; make it relevant to my location and the current time.

 

So that’s a starter set, seven principles that inform me when I think about these things. I shall expand on each in days to come. In the meantime, keep your observations, advice, questions and comments flowing, choosing whichever means or channel you prefer. Comment here. Respond to the link in Twitter, Facebook, Google+, LinkedIn. WhatsApp me if you want. Talk to me via @jobsworth. If you don’t like any of these, then I suppose you can email me via [email protected] but be warned that I look at email rarely and that too only under duress.

What a difference a year makes

Last January, around this time, as people came back from hibernation into the plastic warmth and sparkle of CES, we heard about luggage that can tweet. Hello, I’m here. In Rio. While you wait for it. In Riyadh.

Trakdot was one of the stories I latched upon at that time.

You could put a Trakdot into your sharkskin suitcase and bingo, not only could it tell you where it was, it could also tell you when it approached you. Via SMS if needed. [Incidentally, no sharks were harmed in producing the sharkskin suitcase, it’s not real sharkskin, just in case you were getting ready to complain].

A year on the story gets better. Why bother with mock sharkskin when you can have the real thing? Real sharkskin, still worn by the shark, while it’s alive and breathing?

Apparently over 300 sharks have been tagged and equipped to let bathers and swimmers know of their proximity. And many other things besides.

Screen Shot 2014-01-03 at 15.48.06

Screen Shot 2014-01-03 at 15.48.58

Everything is a node on the network.Even sharks.

Everything can publish, everything can subscribe.

Which means firehoses and filters.

That’s why all this publishing and subscribing takes place against thresholds we set. Like location. Or size. Or temperature. Or time. Or whatever. In whatever combination. Static as well as dynamic.

Over the next few weeks, as I spend more time writing about filtering in an age of streams and drains and filters, the tweeting shark is a good place to start.

What a difference a year makes.

We’re Not Gonna Take It: The Age of The Disruptive Customer

httpv://www.youtube.com/watch?v=AmGgcqjrxMc

We’re not gonna take it
Never did and never will
We’re not gonna take it
Gonna break it, gonna shake it,
Let’s forget it better still

We’re Not Gonna Take It: The Who (Townshend, 1970)

We live in inventive times.

Sputnik 1 went into space a few weeks before I was born; shortly before I entered my teens, man had landed on the moon. While mechanical calculators had been around since the 17th century, solid-state “pocket” calculators only emerged in the early 1960s. The radio sets and amplifiers I grew up with were, similarly, valve-based; there was something magical about how they glowed, how the air smelt different as they warmed up. By the time I was thirteen, I’d been given a solid state “transistor” radio by a friend. The typewriters I grew up with were massive mechanical devices. While electric typewriters had been invented at the turn of the previous century, it was only in the early ’60s that the “golf ball” showed up, and “electronic” typewriters took a decade longer.

The first e-mail was sent during my early teens; the first mobile-phone call around the same time; by the time I was in university, the Commodore PET (“for the masses, not the classes”) had started shipping. I touched my first PET in 1978 or so. The countertop microwave oven came out before I turned 10. Compact cassette tapes began to be commercially produced around 1964; the Sony Walkman didn’t turn up for another fifteen years, shortly after I completed university.

TV’s transition to colour programming took place in the US in 1965, and in Europe a couple of years later. I was 10. It would take a further decade before TV became available in India; I’d actually left the country before I saw colour TVs there.

The first patient to receive a fully implantable heart pacemaker did so shortly after my first birthday; the world’s first heart transplant took place shortly after my 10th birthday. The laser was invented the day after I was born.

By the time I was 12, Woodstock had happened. In the twelve years preceding, the Beatles, the Who, the Rolling Stones, the Grateful Dead, Simon and Garfunkel, Bob Dylan, Leonard Cohen, The Band, Crosby Stills Nash and Young, Traffic, Cream, Blind Faith, The Allman Brothers, Jim Croce, the Doors, Chicago, Jefferson Airplane, Ten Years After, Jethro Tull, Peter Paul and Mary, Gordon Lightfoot, Janis Joplin, Stevie Wonder, Led Zeppelin, Joan Baez, Joni Mitchell, Van Morrison, were just part of the incredible talent parade emerging. And we’d started our painful march across formats, as the recording industry sought to take advantage of a short-term blip in performance art.

The world’s first successful commercial jet service began before my first birthday; while there’d been a handful of hijackings since 1929 or so, hijacking went mainstream from around 1958, to peak in 1969.We’d just begun to understand about DNA; I was still at university when the first successful nucleotide sequencing was completed.

I was 6 when ideas about the internet began to surface, 10 by the time ARPANET was approved, 12 by the time the first message was sent; protocols like FTP and TCP-IP emerged as I left school.

You get my drift. I was born into an age of invention and innovation and disruption, an age that continues apace today. But.

The focus has all been on the “supply side”

Invention, innovation, disruption. Popular terms. And for some reason, loaded on to the supply side of the equation. Inventors and innovators and disruptors thought up stuff and made stuff and provided stuff and sold stuff. It was all about them. And yet, while a tiny handful of them succeeded, many failed; the failures were orders of magnitude greater than the successes. And we looked to the successes. And cheered.

There was also a tendency to make invention and innovation and disruption the progeny of individuals. It made things like patent law easier, I suppose, and created wonderful bottlenecks in the creative process that allowed money to be minted. Sometimes history has been kind to us and shown us that actually this wasn’t the case, that much of invention and innovation happens through the sharing of ideas, sometimes cooperatively, sometimes competitively. But most of the time we were allowed to believe that the magic was in the sole inventor. Libraries of books have been written to tell us this is the case; and over the last forty years, similar libraries of books have been written to tell us that this is not the case.

All this is changing. We’re meeting a new class of inventor, a new class of innovator, a new class of disruptor.

The Disruptive Customer

“You are old, Father William,” the young man said,
“And your hair has become very white;
And yet you incessantly stand on your head—
Do you think, at your age, it is right?”

“In my youth,” Father William replied to his son,
“I feared it might injure the brain;
But now that I’m perfectly sure I have none,
Why, I do it again and again.”

[An aside: One of my favourite parodies, Lewis Carroll as inventive and disruptive as ever. Or perhaps I should call him innovative rather than inventive, since he took an existing “technical principle”: Robert Southey’s The Old Man’s Comforts and How He Gained Them: and parodied it. If you read Southey’s version, you’d be surprised at how pragmatic and sensible copyright law was before it was corrupted into the nonsense of today.]

The point of this post is simple. No invention, no innovation, no disruption, none of this takes place without the customer.

It is only through adoption that we hear the tree of invention, of innovation, of disruption, make a sound

No adoption, no invention. No innovation. No disruption. Nothing without the customer’s adoption.

The Maker Generation is upon us. The tools of production are more and more in the hands of the customer. 3D printing will make this happen even faster. Which is why new roles have emerged, “platform” roles. Platforms that enable connections and interactions, conversations and transactions.

When customers become inventors and innovators, the scale of disruption changes, and changes radically. More people are involved in inventive activities, more trials and experiments carried out, more “failures”, more learning, more “successes”. The cost of finding out what someone else has done, learnt, shared is low and continues to get lower. The pace at which such learning takes place is increasing, and again radically. The places it takes place in are also changing. It’s happening all the time. Everywhere.

When we design platforms, let’s bear that in mind. It is the customer who disrupts. We make it possible, we enable it, we may even catalyse it. But what we do is nothing unless there’s a customer present.

As everything gets connected, as everyone gets connected, as we see these enabling platforms emerge and evolve, the customer will invent more, innovate more, disrupt more. What customers want of us is what they’ve always wanted of us: get rid of the friction, get rid of the latency, take out the barriers to entry, let them in.

Successful platform providers will understand that, and as a result

design to enable while getting out of the way

That’s what I was trying to say four or five years ago, when I was working on my “Design for Loss of Control” theme. And it hasn’t changed.

Design to enable. And get out of the way. More to follow, as I concentrate on platforms and control and filtering and getting out of the way.