Filters: Part 5: Designing for Serendipity

Note: This is the fifth in a series of posts I’m committed to writing about filters; I started with the principles of filtering, and will proceed to blow up each of the principles in as much detail as makes sense at this stage. Earlier I looked at network-based filters, and then spent time on routing, then went on a tangent to look at bringing responsibility into publishing. Today I’d like to spend time developing ideas that can help us avoid tunnel vision, blinkers, groupthink, the madness of crowds, heresy, narrowmindedness, herd instinct and a slew of similar ills.

The background

It is normal and natural to spend time with people who have backgrounds similar to you; you get along better with people who share your interests, who have things in common with you. This is as true in the digital world as it is in the analogue world; it comes as no surprise that humans have tendencies to be tribal in outlook, to “flock” as “birds of a feather”, to form homogeneous groups.

In the past, when our ability to migrate quickly and affordably was somewhat constrained, we tended to live our lives within a short distance of where we were born, with our kith and kin and in communities that had been stable for a considerable while. When we did migrate we tended to ghettoise, we formed districts and neighbourhoods full of parvenus. It was hard to integrate easily into a homogeneous community, so the new entrants stayed together as well. Often they shared a common language, a common culture; in many cases they came from similar locations. So we had the Latin Quarters, the Chinatowns, the Koreatowns. We first formed relationships with people we had much in common with, then proceeded to integrate at glacial rates. Which was fine, so long as integration took place.

When it comes to the digital world, the cost of discovering people with similar backgrounds, interests, attitudes has fallen dramatically; the speed at which we can group thus has increased significantly. Our digital tribes form at breakneck speed, and run the risk of being too homogeneous and insular. Maybe it’s a variant of first-mover advantage, this tendency to form like-minded and closed groups, emphasis on the closed.

I don’t tend to meander into politics here at confusedofcalcutta, and don’t intend to start doing that now. What I will say is of late, as our ability to form digital tribes has improved in leaps and bounds, so too has our ability to polarise every debate. Which is not a  good thing in countries that have two-party systems in practice. Everything soon becomes about Blefuscu, about Big-Endians and Little-Endians. [I’ve written about this tendency before as I saw it evolve within the IT industry, spurred by the simplicity and speed of connecting to others]. I used to think that elections could be about Red or Blue, but government had to be about Red and Blue. That there was a distinction to be made between electioneering and governing. But nowadays it appears it’s all electioneering: the purpose of government has somehow been mutated into one of being re-elected and not much else.

[Enough politics! Ed.] When groups form with strong homogeneous characteristics, it’s easy to lose perspective. Our ability to look at multiple sides of an argument weakens, our ability to tolerate dissent erodes, our ability to take in ideas that challenge the status quo atrophies. The importance of diversity is not just in how we evolve and develop as flesh-and-blood humans, it holds as true in our digital lives.

When it comes to “business” there are a few more frills to consider. Firms have boundaries around them, walls build to keep the enemy out. Departments become silos as people with shared incentives optimise what they do to maximise their payoffs against those specific incentives. This tends to have the effect of making departments compete against each other within the firm, with appropriately negative consequences.

Innovative ideas tend to form when two or more well-bounded edges rub against each other; the frictions and latencies that get exposed through this contact become fertile ground for creative people to envision how to make their lives, and those of the people they care for, better.

It is important to ensure that the filters we set, as subscribers, are formed in such a way that the heretical consequences of extreme homogeneity are minimised.

I think it was Einstein who said that common sense is made up of the prejudices we collect by age eighteen. We all have anchors and frames that prevent us from looking beyond the promontories of our proboscides.

There’s also the issue of the filter-bubble, what Eli Pariser has been so eloquent on. It’s a natural consequence of supply-side filters. And it needs fixing.

All this leads me to the importance of serendipity and diversity in our streams. Allowing chance as well as differentiated viewpoints to emerge and surface, to teach us, to challenge our thinking, to inspire us.

Designing for Serendipity

It seems to me that you can do five things by design:

(a) Listen to people outside your cosy circle, starting with customers

When I was at Dresdner Kleinwort I had a fantastic team working for me, people who have gone on to do greater things because that was always going to be their destiny. It was a privilege to have been there. They were so good that I could spend time looking at unusual things, in efforts to try and improve our ability to create business value through teamwork and collaboration.

One of those unusual things was this: every now and then, I looked at the patterns thrown up by our email usage. How much of the mail we generated actually left the bank’s boundaries. Very little. How much of the mail sent by New York left New York (or for that matter any other location bar London”. Very little. London enjoyed “Head Office” status and therefore generated lots of mail for other locations, but in principle the pattern was the same. Most of the time, people sent mail to colleagues in the same location, to people they could have walked over to speak to. And very little of what they sent by mail went outside the firm. If you have the time, the inclination and the ability, take a look at what happens in your firm. It’s unlikely to be pretty.

Designing the network such that externals can participate is a good place to start. Making sure that you can recognise the presence of an external easily is very important, otherwise the environment becomes so error-prone that trust is weakened. Show the presence of externals using differentiated colours and fonts, as an example.

Build mechanisms that track how many of the people you follow are from the outside; from your partner and supply chain; from locations other than yours, from departments other than yours. If the only people you follow are people who look and feel and act like you, you’re not going to learn very much, your thinking isn’t going to be challenged, you’re not likely to spot the frictions in the handoffs.

(b) When listening to customers, bring in two dimensions beyond just traditional communications/conversation

Most companies get complaints from customers. There’s usually an abundance of complaints in comparison to unsolicited goodwill messages. So why not take advantage of this abundance? Make your complaints follow-able, embody them in the form of a person who publishes complaints into the stream. Make it even better, allow the complaints to be classified semantically. Follow the voice of the complainer, by theme or topic if possible. She’s unlikely to have the same biases as you do.

When you do this, try and retain the original words the customer used. Avoid the temptation to summarise-by-triage, throwing away useful and valuable information. Let me give you an example. Customer complains that her broadband is not working. Traditional scripted methods will go through a plethora of well-designed engineer-responses. And fail. Because the reason for her problem is she hasn’t paid her bill. You’re more likely to spot that if you see the words the customer used, rather than the “trusted” commentary that summarises what was said.

A second route is to use some way of collecting ideas from outside the firm and plugging that flow into the network. Companies like Starbucks and Dell have shown what is possible as a result of doing this.

(c) Always bring in some fresh thinking whenever possible

There are so many ways of doing this. Find out who’s joined the party recently (a new hire; someone from graduate intake; a new-name customer; a recently-formed partner). Start following them. In fact from a design perspective every time someone joins the network, they should be discoverable somewhere as newly-joined. Welcome them in by following them.

When someone follows you and you don’t know them, follow them back. I do this as often as possible, once I’ve verified that the person is a person and not a bot, and once I have checked that the person “speaks” rather than spams.

When you follow people, make sure you make a point of following people who are interested in the same subject but with a point of view that’s different from yours. If they’re interested in music and they’re all about country while you’re about folk, follow immediately; if they’re vegan-foodies while you think man and meat were destined as a pair, follow immediately. The point is to be in touch with people that have similar interests but not identical ones.

(d) Learn from the trends around you

Watch closely what’s trending in your network, amongst the people you know and trust. But keep an eye on what’s trending elsewhere. In a perfect world I would want to be able to say “Let me look at the stream as if I am in India; in Somalia; in Iceland”. Let me look at the stream as if I’m Christian, Jewish, Muslim, Pastafarian, atheist. Let me see how 18 year-olds view this. Let me see what a new joiner sees on day 1. This idea, one of being able to encapsulate and transfer lenses, is something I will come back to in a post very soon.

When you see something trending that you don’t understand, do something about it. Find out what it means. Who’s involved. When it began. Stretch yourself.

(e) Go for a random walk

Stumble, just as in StumbleUpon. Go somewhere different; do something that’s not habitual, with people who aren’t habitues, in places that aren’t your natural habitat.

Again, more later. I’m still building on the arguments, delighted with the comments and mails and tweets and links I’ve been given. Please continue.




16 thoughts on “Filters: Part 5: Designing for Serendipity”

  1. I’m enjoying the series very much JP and totally agree with the philosophy of this post. Designing for serendipity is a fine objectve, but I wonder if there’s also a case for designing against homophily and if that’s possible?

  2. Lovely series. Some of the language is so powerful.

    I have used the same analogy ( but a little differently ) in my blog of start up mentoring and growing the business.

    I have tried to say that each business tries to built a business plan out of an linear growth plan that is homogeneous of the factors of the past as it continuing variables of the future. ( not sure if I have made myself clear )

    In other words all the underlying assumptions of the business plan are that the underlying factors are linear / homogeneous / filtered.

    But on the other hand the real world infuses so much of non linearity on the business. Cost accountants have built a body of knowledge called ‘variance analysis’ around this aspect.

  3. Hi JP – fine work as always. The polarisation you mention has been a result of our biased application of filters to sift signal from noise as the data we try to process grows. I’ve written on this in my chapter on Open Data in my book (10 Principle of Open Business.

    Here’s an extract I hope you will find relevant:

    As data scales so does the ratio of noise to signal. But the act of selection of signal adds risk of bias. Bias is a dangerous thing when you are trying to make predictive models…
    The more complex the predictive model, the more restricted and limited it is by the biases of its creators, and the greater the room for disruption from a catastrophic, unpredicted, moment. It is closed to new possibilities, built to deliver understandings based on the possibilities of the past…
    If the rate of information we have at our hands exceeds our ability to process it the more likely we are to fall back on the default short-cut of bias. Played out at a societal level this amplifies – leading to polarisation.

    This resort-to-bias is a natural human state – a short-cut which saves us time and has served us well through most of evolution. We seek information that supports our natural biases rather than that which counters it.

    Some argue this selective bias was amplified by the arrival of the printing press and the explosion of information it delivered. They see (alongside the Cambrian event of the Renaissance) a causal link to polarisation that gave us a century of religious war and persecution.

    The risk of our latest information explosion – the Internet – is a new and potentially more radical divergence of thought – left and right. Both sides will have the potential to prove themselves ‘right’ with the data – according to their biases.

    One magnificent defensive shield against this, then, is the work and ambition of the Open Data Institute. Promoting more interoperable data means more people can access it, understand it and act on it – potentially acting as a dampener on extremes of bias and offering a level of protection against one bias becoming dominant.

    My personal solution is to deliberately keep my ‘silo’ open by following (on twitter) a number of people whose opinions I find difficult to take. I see value in being exposed to alternative views – and avoiding the nodding dog syndrome of the echo chamber.

    btw – hope you planning on making a book of this series.

    best dc

  4. @john I would use slightly different words but yes I think there is a case to be made to design to minimise group bias or something like that

  5. @DE yes. Both in the US as well as in the UK, decades of gerrymandering have led to extremes of homogeneity as parties strive for making safe seats safer still. does not bode well.

  6. thanks David, will refer to this in a follow-up post. haven’t decided about a book yet. Want to get 15-25k words out this month and then see where it goes.

  7. Thanks for your insightful article, I really enjoyed reading it. My passion has been for years trying to understand serendipity, your article gave me again some new perspectives. I would love to continue discussion with you in this topic. Please have a look at my blog of the same topic, maybe you’ll find it interesting. Here is the link for the blog titled “How to avoid being trapped from serendipity during your journey”

  8. Too many of us take serendipity as a given – not something we can influence. At best, in the words of Louis Pasteur, we can only be prepared for it when it comes along. In a world that is more rapidly changing, those who break this mold and learn how to shape serendipity will be the ones to thrive. Your post reminds me of the perspective on shaping serendipity that we explored in The Power of Pull

  9. JP, great series. Thx for that. Re Serendipity, there is a dimension about high quality, intense, human contacts. I know your series is about filters, data, information, knowledge… the filtering, its creation, its ideation. Meeting NEW people that at a quality level are becoming real friends and personal coaches and help you looking beyond the obvious.

  10. Dear JP, many thanks for this great post which I’ve just come across via Twitter. It touches exactly on many of the problems we are seeking to solve with PositionDial. In terms of the stream / and letting in the outside, we have actually designed this, in the form of the ‘antibod’ – The idea is once we know where you stand on your PositionDial, you can see a stream of news & views opposite to your own. Furthermore, to see across the spectrum all at once, we have laid out views across what we call a Position Axis – so you can see from left to right, for example. We’re UK focussed so far but if you get time would be great to hear what you think about how we’re designing for serendipity…

Let me know what you think

This site uses Akismet to reduce spam. Learn how your comment data is processed.