Musing gently about the impact of change and the time it takes

We’re all very connected now, and news travels fast. There was a time when what constituted “news” used to be corroborated before it was sent on its way. There was a time when news had to be new to be news. There was a time when news had to be true to be news.

Those were the days.

Some of you may remember reading Bombardiers, a book by Po Bronson, which came out over 20 years ago. Here’s a review that might intrigue you enough to read it.

The principal reason for mentioning that book is that Bronson’s satire was instrumental in getting me to understand something I’ve written about before, over 10 years ago.

That something is this, paraphrasing Bronson:

In the Stone Age, Might was Right. And you could figure out who was Mighty quite easily. In the Industrial Age, Money Ruled. And you could figure out Who had the Money quite easily. In the Information Age, figuring out Who has The Information isn’t easy. In fact it is Very Hard. Because the time taken to verify the information can exceed the speed at which information changes.

So it doesn’t matter if the UK doesn’t spend £350m per week on the EU, it doesn’t matter if the NHS never receives that theoretical weekly sum. It doesn’t matter if Mexico never pays for the wall, or even if that wall never gets built.

Things move on. Turning back the clock is hard. Very hard.

There is something that is a little easier. Studying the impact of change, looking objectively at the data. Not easy, but easier.

It means knowing the right data to collect, the right way to collect it. It means preserving that data and making it accessible. It means protecting that data from change and from misuse.

It also means looking at the data over a long enough period. That’s what we didn’t do with cigarettes; but at least that battle’s over. That’s what we didn’t do about sugar; and that battle isn’t over yet. That’s what we’ve been trying to do with climate change, and that battle’s barely begun. That’s what we’re probably at the nascent stages on with fracking: the issue is not about hydraulic fracturing per se, not even about horizontal drilling,  but about safe ways of disposing of the waste water, in particular learning from the seismologists who’ve been studying this aspect.

These battles get won and lost by people who are in positions of authority for far shorter periods than the time it takes for their decisions to have an effect.

And for us to learn about that effect, understand it and respond to it. Adapt as needed.

Things move on. Turning back the clock is hard. Very hard.

We need to get better at studying the impact of change over time. Proper, longitudinal study, collecting and preserving the right data sets, with the relevant discipline and safeguards in place. That’s why I have been fascinated by and supportive of Web Science.

This need for good longitudinal data is also the reason why I have been so taken with Amara’s Law, about our tendency to overestimate the effects of a technology in the short run and to underestimate it in the long run.

Take cricket for example. There was wailing and gnashing of teeth when limited-overs cricket was introduced, more of the same when T20 was introduced. The death of test cricket was foretold and prophesied, and later announced. Apparently it’s been dying ever since.

Not according to the data, especially when you look at longitudinal data. This was something I wrote about recently.

Taken over nearly a century and a half, comprising well over two thousand Tests, the data indicated that since the introduction of the changes, Tests were more prone to ending in win/loss results rather than draws, that more runs were being scored and more quickly, and that perhaps counterintuitively, the number of long individual innings (as exemplified by triple centuries scored) was also on the increase.

Events earlier this week have allowed me to look into another data set, suitably longitudinal, which reinforces all this.

I started with the hypothesis that one reason why Tests may be ending in win-loss results more often is that batsmen have learnt to truly accelerate run-scoring in bursts, using skills acquired in the short game. I surmised that we may be seeing more such bursty behaviour in the 3rd innings, thereby setting up for grandstand finishes, sometimes with “sporting declarations”. I also surmised that this bursty behaviour would be able to act as a potential insurance policy against any time lost due to inclement weather. But it was all hypothesis. I needed the facts. At the very least I needed a suitable data set collected over a sensible time period. The recent Australia-Pakistan Test gave me the catalyst I needed. The Australians scored 241 for 2 in their second innings before declaring.  By itself that wasn’t unusual. But they scored the runs at a rate of 7.53 RPO, something I would associate readily with T20, something I would expect in a small percentage of 50-over games, but something I would consider a complete outlier in the five-day game.

So I went and had a look.

In the history of Test cricket, covering somewhere between 15000 and 18000 innings, there have been just 10 instances where a run rate (RPO) of 6 or more per over has been sustained in an innings lasting 20 or more overs.

  • England 237/6d, RPO 6.12, 3rd innings, Mar 2009
  • Pakistan 128/2, RPO 6.19, 4th innings, Oct 1978
  • West Indies 154/1, RPO 6.20, 4th innings, Mar 1977
  • Australia 251/6d, RPO 6.27, 3rd innings, Jan 2015
  • Australia 264/4d, RPO 6.28, 3rd innings, Nov 2015
  • South Africa 189/3d, RPO 6.37, 3rd innings, Mar 2012
  • Pakistan 164/2, RPO 6.60, 4th innings, Nov 1978
  • South Africa 340/3d, RPO 6.80, 2nd innings, Mar 2005
  • West Indies 173/6, RPO 6.82, 4th innings, Feb 1983
  • Australia 241/2d, RPO 7.53, 3rd innings, Jan 2017

The first limited-overs international was played in 1971. All ten instances took place after that date. The first T20 international was played in 2005. 6 out of the 10 instances took place after that date. In all ten cases, the team putting their foot on the accelerator didn’t lose; in half the cases they won.

 As it is with cricket, so it is with many other things. When you change things, it takes time to figure out the real effects. Longitudinal studies are important. This is as true for technology change as for any other change. With all change, there is an Amara’s Law in operation. We tend to overestimate the short term effects and underestimate the longer term impact. 

Tracking the impact of change requires good baseline data and a consistent way of collecting and preserving the data over long periods. That’s not a trivial task. It is made more complex with the need to protect the data from corruption and misuse.

While I love cricket, I only use it as an example  here, to illustrate how longitudinal studies can help assess the impact of change, objectively and reliably.
 

… and crime travel

I didn’t have a passport until I was approaching my 23rd birthday. But that didn’t stop me from travelling far and wide.

Calcutta was a truly cosmopolitan city in those days; people from many cultures would pass through. While one generation of people, rooted in empire, left to find those roots, another, younger generation came for the first time, to “find themselves”. People like Allen Ginsberg. Like the Beatles and Eric Clapton. Like Steve Jobs.

This gentle exodus and influx of foreigners prevailed throughout my youth, and that was one way I left Indian shores, just by spending time with visitors. Some of those visitors stayed, and that’s part of what made Calcutta Calcutta.

I guess I lived trilingually, much like many of my friends and relatives. English was my lingua franca and the language I studied in, the language I conversed with foreigners in. Tamil was what I used to communicate with my mother and with my grandmother. And Bengali was for everything and everyone else.

[I suspect we made a point of not saying we spoke Hindi. The unwritten rule in those days was that every state with a coastline “refused” to speak Hindi and toyed with the idea of seceding from the coastless centre of India, where all the Hindi speakers hung out. At least that’s the way it seemed to me].

When it came to culture, however, the trilingualness faded, everyone accepted the opium of Hindi film music into their lives…. along with the local Baul and the deep-south Carnatic and the traditional classical. And particularly in the cities, even more particularly where English was the language of school, there was a deep English-language based cultural flow: music, literature, the arts in general.

These things represented a second method of travel for the passportless me. We studied other cultures. We listened to their music. We watched their films. We enacted their plays, read their poetry out aloud. [Many years ago, I wrote about the sheer breadth of the songs we used to sing at school, something I still marvel about today.]

Most of us didn’t have passports. Most of us travelled anyway. Vicariously. In many ways.

One of those ways was by being voracious readers. And one road that some of us travelled more often than others was that of detective fiction. [Over time, the genre has broadened to include “mystery” and “suspense” and “thriller”, but when I was doing the travelling at home in Calcutta it was really detective fiction].

We learnt about the 19th century via the America of Edgar Allan Poe and the Britain of Wilkie Collins. People like Chesterton and Philips Oppenheim and Baroness Orczy took us into 2oth century Europe, and then the Agatha Christies and Dorothy Sayers and John Creaseys carried on where they left off. We went into the 1930s New York brownstone of Rex Stout’s Nero Wolfe; the Paris of Simenon’s Maigret; the courtrooms of Erle Stanley Gardner; the Amsterdam of Freeling’s Van der Valk; the Po Valley through the eyes of Guareschi’s Don Camillo. We discovered a new America through the Ross Macdonalds, the Richard Starks, the Robert Parkers, the Ed McBains.

We travelled. Far and wide. While never leaving our seats.

That travel was place-travel. While we did travel back in time, the only reason we did that was because the books were written at that time, they were contemporaneous accounts of the culture.

I think Georgette Heyer was the first person to make me travel in space and time; I don’t speak here of her Regency romances but her detective fiction. She let me taste something I really liked, a hybrid genre: historical detective fiction.

That’s a big field now. I’ve read most of the players, and the standout author for me in the hybrid is Michael Pearce. If you haven’t been to the 19th century Egypt of the Mamur Zapt, you haven’t lived. If you really haven’t, and if you like detective fiction, then you have so much joy awaiting. Michael Pearce is amazing.

More recently, my historical-detective cup has runneth over a few times, and I wanted to make sure I didn’t selfishly hold on to my spoils.

Jason Goodwin was the first to make a real impact on me, with The Janissary Tree, featuring Yashim, a eunuch wandering around early 19th century Turkey. He’s put out four or five books over the last decade or so, and they’re all excellent.

More recently, I came across Miranda “MJ” Carter. Her Blake and Avery books are absolute must-reads for people like me. I found the Strangler Vine unputdownable. I think she’s not fair on me, since her books have much to do with historical Calcutta and historical London… the only two cities in the world I’ve really lived in. I’m reading The Devil’s Feast right now and it is brilliant.

And only a few weeks ago, I chanced upon Abir Mukherjee. Was introduced to Sam Wyndham in A Rising Man. And became hooked.

Michael Pearce and the Mamur Zapt made me yearn for something, something rare, something beautiful. Now, with Miranda Carter, Abir Mukherjee and Jason Goodwin, I have three people who let me really enjoy myself travelling in space and time while reading good detective fiction. My thanks to all four people for having enriched my life.

 

 

Time travel

On any given day I get sent maybe 20-25 messages through one communications channel or other, with links to new sites or apps. Most of them are of no value to me at all. Maybe I’m growing old. A friend sent me a link today; I can usually rely on him to send me interesting things, so I took a look. This, despite the site and app having one of those oh-so-oughties names.

Radiooooo.

I tried it. Not having to register in order to try it out helped, that was a big plus for me. Chose Canada, Slow, 1970s. And up came Neil Young and Vampire Blues. After a while switched to India, stayed Slow and 1970s. And I was served Ananda Shankar and Raghupati. Registered straightaway. Downloaded the app as well. I like being able to vary how I want to engage with such things.

It’s still in beta, and I’m still learning about the site. Some categories are empty. I have no idea how many people have uploaded music, but that number feels low at present, I see the same names come up a few times. That may have to do with the selections I’m making.

I’m intrigued by the Share and by the Buy options song by song; the prominence given to the uploader suggests that over time this is going to become a blip.fm with an edgier UI.

The ability to time-travel around a music site is itself not new; neither is the serendipity offered in various forms. I couldn’t help but smile when I saw India turn to British India when I chose 1930. But being served Yom Hashabat by Nathan Solomon Satimkar was, to say the least, unusual.

Fascinating.

Radiooooo feels a bit like the first time I came across a food hall at a mall in the US. I was like a child in a sweetshop when I realised that I could choose to have something sensibly spicy while other family members could do their own thing, and we could still sit together and eat together.

That’s how I feel about the site right now. It’s fascinating to be able to mix genres so easily. It’s almost as if someone decided to build a mechanism by which each one of us could design our own StumbleUpon for music.

The ease with which I can get to, discover, shuffle through disparate times and places and genres is very attractive. There’s a long-tail aspect that soothes me, I’m not a hit-culture fan. I am even less a hit-culture fan when people I haven’t learnt to trust make the choices for me, but that’s another story.

I haven’t uploaded anything yet, nor shared anything so far. I’m still in early explorer mode.

But what I’ve seen so far, I like.

Radiooooo has possibilities. And I shall continue experimenting, and watch with interest.

 

 

 

Thinking lazily about notifications and alerts: Part 2

This is the second in a series on notifications and alerts, building on what I started sharing earlier today, as promised.

First, a musical interlude.

Someone’s knocking at the door, somebody’s ringing the bell/ Do me a favour/Open the door/And let them in.

Mum, the kettle’s boiling/Daddy, what’s the time/Sis, look what you’re doing/Can’t you see/The baby’s crying

We get used to receiving and processing notifications while we are still children. Doorbells ringing. Kettles boiling. Hearing footsteps approach, learning which ones are friendly, recognising the patterns made by parents and siblings.

And yet the one I remember most vividly is to do with what’s exemplified in the photo below:

kayarakhet31.jpg

[My thanks to VikalpSangam whose photo of Mohan, above, helps make my point].

It’s  a very strong childhood memory, one that is almost as strong as sensing the presence or return of a parent. You see, I wasn’t much of a sleeper. [Never was. And now that I’m approaching 60, it looks like I never will be. The power of habit]. As I lay awake, tossing and turning in the sweltering heat of a Calcutta night, I’d hear a strange sound. A sound I came to treat as a friend. The sound made when a stout stick gently hits a lamppost. A strange sound indeed.

We used to live on a street called Hindustan Park in the 1960s, in Ballygunge in Calcutta. I don’t know the precise history behind the phenomenon, but what I can remember is this. The local darwans, who performed roles of building manager, maintenance man and security guard, used to take turns to walk around the neighbourhood while their colleagues dozed. The one doing the walking would signal his presence and doing of the rounds by twanging the occasional lamppost with his danda, his lathi.

It seemed to me that his lamppost-striking action achieved many outcomes: it alerted his colleagues that he was doing his rounds; it probably alerted would-be burglars as well, but more of that in another post; most importantly, it made me feel that God was in his Heaven and that All was Well with the World. It wasn’t just about presence being signalled over distance, it was about the sense of security implied by that presence.

Which brings me to the first class of notification: All is Well.

There’s a rhythm, a pulse, a cadence, to the All is Well notification. It’s a repeating signal. It’s like hearing the sounds a baby makes while asleep. It’s like seeing an ECG at a hospital bedside. There is no need for alarm, no warning threshold has been breached, no action needs to be taken. But its absence is often a signal for action, for investigation.

As the number of sensors continues to grow exponentially, and as we get better at joining the data collected and creating value via insights, we will learn to build baselines for many such things. Initially these baselines may well pertain to single senses, but as we learn and adapt we will build multisensory baselines as well. We will describe whole environments in multisensory ways: a child’s bedroom; a person working out in the gym; a restaurant kitchen in full flow; a factory floor full of robots; a street with a mixture of driven and driverless vehicles. For each of these, we will have established when All is Well: the temperature, the energy consumption, the heart rate, the breathing sounds, the ambient noise, whatever.

It’s important to distinguish the All is Well notification from all others; I think it would be a mistake to assume that people only want a variant of “management by exception” reporting. It’s like the mother wanting to check that the child is still alive, asleep, relaxed. There’s a wellbeing signal, a Linus’ security blanket involved, and this should not be confused with exception management.

Right now it may not be obvious why we should concern ourselves with the All is Well notification when in the context of enterprise software. But I think it’s only a matter of time. One of the implications of hyperconnectedness appears to be that by the time we find out something’s wrong, it’s too late to avoid damage. Our early warning systems will learn to become more sophisticated in order to deal with the problems of connectedness.

It’s worth taking a leaf out of David Agus’s book in this context. Marc Benioff introduced me to David some years ago, and I found his line of thinking very instructive, concentrating on wellness rather than illness. The human body is just one great example of complex adaptive systems in operation, and there is much we can learn from people like David as a result; it is then up to us to adapt that learning to the enterprise context. With the Second Machine Age  of Brynjolfsson and McAfee now upon us, we have to get a move on in understanding how to keep notified of a state of wellness at work and at play, as collectives and as individuals.

So there’s a lot of work to be done in fleshing out the All is Well notification. How to form the new baselines. How those baselines move from being single-sense to multi sensory. The role of time series in all this. The increasing march of robotics, of augmented reality, of hybrid operating environments. The likely arrays of unintended consequences we will face as we go through the learning: the world-ruled-by-algorithm issues identified by people like Kevin Slavin,  the problems caused by poorly designed filters as described by people like Eli Pariser, modern versions of Asimov’s Three Laws, we have all that and more to face and to adapt around.

That’s only the beginning, as we then learn more about which notifications to receive on which devices, and when;  how those notifications will announce themselves when they arrive; when the receipt of a notification has legal standing.

It’s only the beginning.

In my next post I shall be dealing with the next two classes of notification: the Houston, We Have a Problem class and the I Am Here class. After that I shall try and wrap up the remaining classes of notification quickly, so that I can concentrate on the filtering/subscription processes.

Some of you have suggested that I should hold these thoughts and write a book around them. I’d rather share and learn via places like this one here, even if the conversations are with just a small number of people. It’s not that I don’t want to write a book. I do and I will. Sometime. But not about this. More to the point, I want things like this to be discussed openly so that we can all learn. Maybe I have the most learning to do. I will find out soon enough, even if only in private.

Feel free to engage via Twitter or LinkedIn or Facebook or even here, if it’s not too retro for you. Use e-mail only if you absolutely must, we haven’t had that spirit here since 1969.

 

Thinking lazily about notifications and alerts

There are some things  that I can sense for myself. I can see something, hear something, touch something, smell something, taste something, provided it happens within the range of the sense needing to be used.

With the necessary training, I may be able to add granularity to what I sense. Some people can gauge distance with great precision, assess the weight or height of an object. Sometimes this is done with senses working in combination: a smell may impart information about taste.

Most of the time, granularity is added by our using tools: measuring jugs, scales, thermometers and the like. The dashboards we build at work are just examples of measuring tools, often providing granularity to something that a skilled person can “sense”. For example, a good sales manager will know how she is doing against target, but use a dashboard to add granularity to that knowledge.

Ever since I first heard Kevin Kelly speak of technology as something that can speed up evolution, that concept has fascinated me. The internet of things may mean many things to many people: what intrigues me the most about it is the idea of “more sensors, more actuators”. When those sensors are networked together, then terms like collective intelligence and “global brains” start gaining prominence. When the datasets collected by those sensors are labelled usefully, then terms like “big data” become more meaningful.

Connecting sensors together allows us to conquer “distance”, to sense things that are happening elsewhere. It’s been a few decades since terms like “the death of distance” were used to describe what the internet and the web represented. As mobile devices proliferated and became “smarter” this trend continued.

As it became possible to persist the data collected by these myriad sensors, and to make that data available across networks, we began to talk about our new-found ability to “shift time”. A trend that was probably first seen in messaging (centrally held paper-based phone messages, then paging services, then locally held voicemail, then remotely accessible voicemail, and so on) managed to work its way into other spheres of activity. The television industry was the one that felt it quite deeply, as people decided to record stuff to watch later; it then became only a matter of time before video on demand became normal and TV went “nonlinear”.

Whenever I think of notifications and alerts, I tend to view them with all this in mind. That the notification or the alert is made possible because I can “sense” more. That the sensing is taking place while I am able to shift time and/or place.

There’s one more thing. Filters. Something which regular readers will know I’ve spent quite some time on before, herein this series, and even earlier in this series.

When you have sharp increases in the number of sensors available, when these sensors allow you to sense things far away, when the event being sensed may have taken place earlier, then the ability to filter notifications and alerts becomes very important.

I’ve been spending time thinking about this, and plan to share my thoughts here over the next few weeks, looking for criticism, for feedback, for advice. Stay tuned.