Musing about silos and streams

Screen Shot 2013-10-20 at 22.41.51

 

Silos. They’ve been around for years, millennia even; evidence of silos can be found in Tel Staf, c.5200-4700 BC. Storehouses of valuable produce, protecting and enhancing that value, connected into the supply and demand rituals of local markets.

 

rats-eat-grain

 

They’re still around, but now they’re crumbling and decaying, with rats fighting over the grain that left’s behind. Largely empty, forgotten and forlorn, serving very little purpose other than to signal their obsolescence.

And I’m not just talking about grain silos.

 

the_tilbury_post__telegraph_office_1890_mid

There was a time when the post, the telegraph and the telephone formed separate and successful silos. Each to its own. And as they grew up, helped by the protection provided them through monopoly structures, they lived happily together in an even bigger silo called telecommunications.

Whirlwind-1952

There was a time when the computer and its peripherals and software and services formed separate and successful silos, each to its own. And they too grew up in holy and proprietary ways, making sure that customers had splendid isolation in the name of choice. And all was well.

 

IMG_0958

There was a time when there were even more silos. If your computer wasn’t “general purpose” then you were allowed to build yourself a whole other silo, called embedded systems. And everyone stayed within their silos and honed their skills and built up their markets and never talked to anyone else.

Unlike grain silos, you couldn’t just transport information from one silo to another that easily; it required specialist skills and a lot of time and a lot of expense. Which kept some people very happy and a lot of people very poor. Maybe not that different from grain silos after all.

pollock_convergence

 

Those fairy tales are over. Now we have convergence. Actually we’ve had it for decades, but it’s all becoming more affordable and more visible and more useful.

We have a number of cats at home. Three, to be precise. They’re wonderful creatures.

Two of them, Mudpie and Midnight, are sisters. They’ve been with us for over a decade; we took them over from friends who were emigrating, so they’re really old.

 

IMG_2481

So we wanted to have a younger one as well, to prepare for days to come we’d rather not think about.

IMG_1783

Tiger. He was great. But one day he went missing, never to be seen again.

Some time later Lily joined our household.

IMG_4081

She’s still in charge. Has us eating out of her hand, and she even keeps Mudpie and Midnight in check. Occasionally bullies them, until they realise they’re wiser than she is.

Convergence has affected our cats’ lives. They’ve been chipped. Now they and only they can use the cat flaps to enter our house. No more late-night raids from the neighbouring marauders, unwelcome visitors for nigh on a decade. And if I wanted to, I could set different rules for each cat. Even track where each cat was, and where it went the previous night. All our cats are outdoor cats, still keeping their mouser instincts honed, often bringing in rabbits and birds as well. Most of the time it’s okay; occasionally, when said mouse or rabbit or bird is alive, unharmed and running/flying/hopping freely around in the house, it makes for an interesting half an hour or so.

Lily, the youngest, is part of the iPad generation:

IMG_3988

IMG_3989

IMG_3991

Yes, as the saying goes, there’s an app for that. You can get apps for kittens to play with.

Actually it’s not just kittens. There’s an app for everything.

And everything is connected.

All the time.

Everywhere.

[Pedants will write in and tell me that everyone is not connected, that there is a digital divide, that there are many parts of the world where this is just not true. And I will agree with them. And yet.  Please just go to Mumbai or Beijing or Nairobi and look around you and realise the future’s already here, it’s just not evenly distributed.]

This interconnectedness and smart-phone-ness and app-ness of everyone and everything is blurring the lines between telecoms and general-purpose IT and embedded systems, bringing us a world of opportunity we could only dream of not that long ago.

Soon we will be able to say “If the doorbell rings and if the only person in the house is my aged hard-of-hearing aunt, then please switch on the light near the flat-screen TV in the living room because she doesn’t like using the computer to watch television, she’s old fashioned that way; but before you do that check if the person ringing the doorbell is that pesky brush salesman; if it is he, then don’t disturb my aunt, just let the Alsatian next door know that his favourite trouser leg, still attached to its human owner, is back in town.”

Bart_9Weeks_10-27-09_001

We live in exciting times. Silos have broken down.

Everything that’s connected is capable of sharing streams of information about state and status, accompanied by rich metadata to provide context. We’ve been able to do all this before, but not for everything and not instantly.

The instant bit changes everything. There are so many things we do that are based on information not being real-time. We rely on historical stocks of information because we’ve not been able to rely on anything else.

Like censuses. Remember them? As we’ve moved from millions of people to billions of people, they’ve become harder to do, more expensive, less accurate and very very time-consuming….. as long as the way we did them was the time-honoured way.

But some time-honoured ways are out of time and no longer deserving of honour. At a level of abstraction, the LIBOR scandal is a classic consequence of using a “stocks” process rather than a “flows” process.

There are LIBORs everywhere. The way we value things; the way we count things; the way we sound out opinions; the way we measure things ….. we’ve had to put up with taking samples of information and then extrapolating the samples according to some convention or another. Much of the information we see and use from traditional sources can be characterised as low-frequency snapshots of discrete samples aggregated and represented according to some agreed convention or other.

It doesn’t have to be that way. Now terms like real-time and all and continuous can be used without sounding like hyperbole. No more samples, no more extrapolation, no more conventional representations. Just the facts.

A whole new way of looking at information.  Information that emanates from everything and everyone, in a world where everything and everyone can “publish” and “subscribe”.

Information that comes as a live, “real-time” stream, as a series of streams; streams that get aggregated, filtered, personalised; streams that get aggregated, trended, analysed and projected; streams that help tell the past, the present and the future of  many small yet important things. Things that affect our health, our education, our welfare.

The technology architectures that have emerged over the past decade or so are built for this new paradigm, one of streams and filters and drains; one where there is no longer any difference between telecoms and IT and embedded systems; one where information comes from subscribing to flows and gaining insights from those flows, using personally chosen filters to make every firehose look like a set of comprehensible capillaries.

New paradigms come with new problems. The debates we continue to have about identity and intellectual property and the internet.

Subsidiary debates about privacy and confidentiality and sharing and anonymity and censorship and all the regulation those things bring with them, often creating new forms of trade protectionism as barriers get drawn on political lines. These debates have been going on for some time now. And in a John Lennon kind of way, life has carried on while we’ve been busy making noise about all this.

And life will carry on.

This new interconnected always-on publish-subscribe streams-and-filters-and-drains world is on us.

The silo was the symbol of stocks. The stream is the symbol of flows.

The stream is where some people live, and where more join every day. Tools continue to emerge, tools that help us harness the stream and navigate it. Tools to provide context; tools to filter; tools to visualise; to personalise; to aggregate; to imbue with enhanced meaning; to analyse; to project, forwards and backwards in time; to move, in geography and in culture.

The stream is here to stay. And I for one am excited.

Still deceased after all these years

 

Screen Shot 2013-10-12 at 16.35.57

“I don’t know where that leaves you, but you’re still deceased as far as the law is concerned.”

You couldn’t make it up. A man disappears in 1986; is declared legally dead in 1994; reappears in 2005. And the judge, calling it a “strange, strange situation”, found that death rulings cannot be overturned after three years.   There’s no more raising Lazarus from the dead, not with today’s Pharisees. Especially if you’ve been dead three years or more.

Sometimes the problem is not with returning from the dead, there’s a more fundamental issue at stake. In some places it’s illegal to die. Yup, death is prohibited.

gedc0219-copier-19

A few years ago, we went to Aiguebelle for our summer vacation, far from the madding crowd. And the nearest town was Le Lavandou. Which you could get to on the “road train” pictured above. And so we did. While we were there, I looked up Wikipedia to learn more about the place. It turned out that A Nightingale Sang In Berkeley Square was written in a bar there.

It also turns out that it’s illegal to die there. The mayor called it “absurd… to counter an absurd situation”.

Le Lavandou’s not alone. Apparently the Greeks of Delos beat them to it by about 2500 years, and there have been many others since.

Not being allowed to die. Once dead, not be allowed to come back. Predictable phenomena in a world of letter rather than spirit, a topic I’ve had the opportunity to delve into time and time again. Some decades ago, as part of my investigations, I was pointed towards the works of Michael Polanyi in this regard, particularly in the context of things tacit and things explicit. His “we can know more than we can tell” mantra resonated with me, and helped me understand something about the challenges of formulating workable law in any sphere.

foura

Influenced by my upbringing (Brahmin family, Jesuit schooling and university, all in cosmopolitan Calcutta) and by my teenage reading habits (which included a healthy dose of Asimov, not just the fashionable Foundation series, but, more relevant to this context, the Robot series), I had become a firm believer in spirit-not-letter, that the intent of the law or principle or guidance was what really mattered.

isaac-asimov-escritor

Asimov believed that we all needed to understand why the spirit-versus-letter argument was going to become more and more important, particularly as we moved towards a posthuman society. It takes a lot to laugh, it takes a train to cry. Robots can do “letter” with their eyes closed. But spirit? That’s a whole ‘nother deal. [Incidentally, if you’re interested in this topic, you should go read Sage Leslie-McCarthy’s paper on Asimov’s Posthuman Pharisees; it is excellent].

Polanyi believed that our tacit dimension included tradition, inherited practices, implied values and prejudgments. We live in a time of intense change, and often we throw much of this to the four winds in the name of progress. But we have to be careful. Amidst all the bathwater of tradition there’s a baby that may be worth keeping.

Implied values.

Some of those implied values are part of what makes us us.

That’s what I believe George VI was referring to when he said:

Screen Shot 2013-08-08 at 08.11.24

Sometimes, we behave “in a way which would not do if generally adopted“. It is something that every one of us is capable of, something we need to watch for.

So I get concerned when Stuart Broad doesn’t walk, deciding that the letter of the law had suddenly become more important than the spirit. In cricket, there is nothing more important than spirit.

It’s not just in cricket. Spirit matters.

Throughout the Cold War, there appeared to be a tacit rule that spies did what they did as part of statecraft while carefully avoiding any accusation of spying on commercial grounds alone. That rule is now being declared irrelevant and unworkable.Spy-vs-spy

There used to be an unwritten rule that governments didn’t spy on their own citizens unless they had demonstrable cause.

There used to be an unwritten rule that the role of elected officials was to serve their electors, to “govern”; to debate law while it was formed, then to uphold law once it was passed; where law needed changing, to use the process of law to change the law.

These unwritten, tacit rules were part of the fabric of society. An “understanding of things we knew but could not tell”, formed over millennia.

We are moving headlong into a society where we have to learn to teach robots to be human as they drive cars and operate drones and perform surgery and make our hearts work.

We need to be careful.

Because we could also be moving headlong into a society where we forget how to be human, and become primitive robots instead, unable to tell spirit from letter.

Spirit matters.

 

 

Not singing in the rain

Each year I try and spend a night out underneath the stars to try and experience, in a tiny and vicarious way, what the homeless face. It’s for a charity close to my heart, Action for Children. The event is called Byte Night, and this Friday hundreds of people all over the country will join us in raising money for Action for Children.

Last year was an interesting experience. It rained. And rained. And rained. The non-swimmer in me wasn’t perturbed, but another hour or two and I may have been.

This year, sadly, I won’t be amongst the sleepers-out this Friday, wet or not. I’m unavoidably away on business, and we haven’t yet invented the transport that can get me back from the US on time.

But that doesn’t stop my colleagues from spending Friday night on the banks of the Thames. It doesn’t stop them hoping and praying that the weather is somewhat better than last year.

It doesn’t stop me donating to the cause.

I hope it doesn’t stop you. Please give. And give generously.

Here’s the link.

Thank you.

Coda: I will, in conjunction with some of my colleagues, be running some other events this year in order to augment what we raise for this. Watch this space.

 

 

Musing about coincidences

med_verdoliva_improbable_coincidences-05-jpg

Every now and then something happens and you realise the world’s getting smaller, more connected, somehow overlapping in strange and wonderful ways.

There I was last night, quietly putting my feet up after a satisfyingly hectic week. I’d gone out for a wedding anniversary dinner with my wife (our 29th), and we were enjoying one of those companionable silences that can only be enjoyed by people who’ve spent lives with each other, reading peacefully. And then I remembered something I needed to do before going to sleep. It involved my having to leave the bedroom and go online.

Which I did. And I noticed that a good friend and erstwhile colleague had sent me a FollowFriday recommendation:

Screen Shot 2013-09-28 at 16.00.52

He’s known me for a long time, and he knew that the Gerald Waller 1946 photograph was something that meant a lot to me. No surprise it’s been my Twitter background since Twitter had backgrounds. No surprise I’ve written about it years ago.  I’ve met many people who like that photograph, so the #ff didn’t make me jump into action.

So Stu decided to press home the point:

 

Screen Shot 2013-09-28 at 16.06.23

He didn’t stop there either. He went on to share a youtube link with me, one that featured the musician in question, Nicki Wells (@nickiwellsmusic)

Screen Shot 2013-09-28 at 16.10.49

Then Saturday came along, I went about my chores, settled down for the afternoon. And watched the video. Liked the music, carried on.

Screen Shot 2013-09-28 at 16.15.57

And then stopped. Thought I recognised the drummer. Rewind. Replay. And there he was.

My nephew. Jivraj Singh.

Screen Shot 2013-09-28 at 16.17.20

Coincidence? I guess so. But that world’s getting smaller, the connections are getting more and more wonderful.

That made me call up his mother and speak to her, and I found out more about what Jivraj is up to now. Rolling Stone had already told me he was working on five different album projects.

For some reason, I’d missed learning about the Chaiwalla Sessions with Nischay Parekh. Big miss. Big mistake. Now corrected. Now I’m hooked. To Jivraj Singh, to Nischay Parekh, to Nikki Wells. To Nitin Sawhney.

Don’t make my mistake. Go listen to them. Go listen.

Coincidences. They make the world go round.

 

 

Of right things and right ways

I was born in 1957. A long time ago. And that meant I grew up in the Sixties and early Seventies, in an India that was teenage in its independence. [In fact, when I was born, Goa still belonged to Portugal, albeit only since its annexation over four hundred years earlier.]

They were times of tumult and of transformation, of triumph as well as tragedy. Change everywhere. Change at incredible speed. Change with long-lasting consequences. Women entering parliament and even becoming heads of state; racism being tackled head-on for the first time; valves being replaced by transistors; man going into space; man landing on the moon; computers beginning to be exploited commercially, the mouse and the pointer being invented, personal computing becoming a possibility, the internet beginning to take shape. Mobile phones entering the fray; e-mail rearing its ugly head. Protest movements everywhere, people fighting to be heard, people fighting for their rights. Fighting against colour prejudice, against gender inequality. Fighting for the right to choose and for the right to live. Affordable international travel, leading to greater mobility across the world. Assassinations, hijackings, the beginnings of modern terrorism. Entire countries dying, and new ones coming up, as the last vestiges of five hundred years of European colonialism came to an end. War. And peace. And some of the finest music ever produced.

By the time I was 14 years old change, and rapid transformational change at that, was a constant in my life.

I’ve spent the next 40-odd years observing change, being part of that change, changing myself. And sometimes even trying to change some of the world around me.

For some reason, quite early on, I began perceiving each wave of change, regardless of its locus and coverage, as something separate in itself. Compartmentalising the change in order to make sense of it. Seeing each change as an individual thing, with something I could identify as a start, something I could identify as an end, some people involved in making the change happen, some people involved in leading that change. Starts and ends, reasons and goals, resources and costs and times and outcomes.

Yes. I confess. I viewed much of the change around me as a set of projects, sometimes interwoven, sometimes overlapping, sometimes gloriously isolated.

All projects involve change. All change meets inertia. And there’s always some risk as a result. A lot of the time, changes begin without everyone really knowing what the desired end-state is. Which means you have the skills and knowledge for the start of the project and, in all likelihood, you have to figure things out as you go along. Discover stuff. Learn, usually from mistakes. Adjust, refine. Move on. Sense and learn and respond. Again and again. Iterating until you reach where you need to be. The persistence of Robert the Bruce. The perspiration that lubricates inspiration. If you keep on at it, learning as you go, then, in Yoda fashion, you get to there-is-no-try-only-do.

That’s what projects are. Each project is a try that becomes a do as you iterate and refine and adjust and learn. Most of the time, change takes you to a place where you’ve never been; sometimes, it takes you to a place that nobody’s ever been. So discovery and learning and iteration become critical.

As you scale from one-person projects to larger ones, some sort of governance model becomes necessary. A process for agreeing priorities, for allocating resources to the tasks, for monitoring progress and getting feedback, and for intervention in the event of problem or conflict. [I chose not to use the word “failure” in that set. I have not failed, I have found ten thousand ways that do not work…]

Most projects start with constrained resources. Anecdotally speaking, the projects I’ve enjoyed the most have been those with hard constraints. Such projects tend to attract people-who-want-to-govern-inspect-or-otherwise-give-their-opinion in droves. Droves and droves. Advice and support and help are always useful,  provided you know how to use them. There’s a serenity prayer in there somewhere.

Some people may come to praise the “project”, some to bury it. You can’t do anything about it, it’s a hazard of business. But what you can do is this: you can figure out whether the person in question could help you decide on the right thing to do, or about doing things the right way.

In my own experience, there are many many people who can tell you how to do things the right way.

But a rare few who can help you work out the right thing to do. They’re keepers.

George Gilder used to say that every economic era is characterised by its unique abundances and scarcities, and that a successful business is one that makes use of both the abundances as well as the scarcities.

So that’s what I’ve tried to do in projects. Corral together the people who can help me do things the right way, convert their energy into repeatable process, build bureaucracies around them as and when needed, just to contain the sheer numbers and make things efficient. A QWERTY keyboard approach to prevent logjam.

And then I’ve tried to spend as much time as possible identifying, nurturing, developing the people who can tell me the right thing to do. The rare and the scarce. The keepers.

Which is why, when I look at governance models in companies, sometimes I have to smile wryly.

Startups tend to be hungrily looking for people who can help them ensure the right things are done. And they call them mentors and coaches and advisors.

Established companies tend to be looking, somewhat less hungrily, for people who can help them ensure that things are done the right way. And they call them NEDs.

Abundances and scarcities. It’s important to know which is which.