Musing about silos and streams

 

Silos. They’ve been around for years, millennia even; evidence of silos can be found in Tel Staf, c.5200-4700 BC. Storehouses of valuable produce, protecting and enhancing that value, connected into the supply and demand rituals of local markets.

 

 

They’re still around, but now they’re crumbling and decaying, with rats fighting over the grain that left’s behind. Largely empty, forgotten and forlorn, serving very little purpose other than to signal their obsolescence.

And I’m not just talking about grain silos.

 

There was a time when the post, the telegraph and the telephone formed separate and successful silos. Each to its own. And as they grew up, helped by the protection provided them through monopoly structures, they lived happily together in an even bigger silo called telecommunications.

There was a time when the computer and its peripherals and software and services formed separate and successful silos, each to its own. And they too grew up in holy and proprietary ways, making sure that customers had splendid isolation in the name of choice. And all was well.

 

There was a time when there were even more silos. If your computer wasn’t “general purpose” then you were allowed to build yourself a whole other silo, called embedded systems. And everyone stayed within their silos and honed their skills and built up their markets and never talked to anyone else.

Unlike grain silos, you couldn’t just transport information from one silo to another that easily; it required specialist skills and a lot of time and a lot of expense. Which kept some people very happy and a lot of people very poor. Maybe not that different from grain silos after all.

 

Those fairy tales are over. Now we have convergence. Actually we’ve had it for decades, but it’s all becoming more affordable and more visible and more useful.

We have a number of cats at home. Three, to be precise. They’re wonderful creatures.

Two of them, Mudpie and Midnight, are sisters. They’ve been with us for over a decade; we took them over from friends who were emigrating, so they’re really old.

 

So we wanted to have a younger one as well, to prepare for days to come we’d rather not think about.

Tiger. He was great. But one day he went missing, never to be seen again.

Some time later Lily joined our household.

She’s still in charge. Has us eating out of her hand, and she even keeps Mudpie and Midnight in check. Occasionally bullies them, until they realise they’re wiser than she is.

Convergence has affected our cats’ lives. They’ve been chipped. Now they and only they can use the cat flaps to enter our house. No more late-night raids from the neighbouring marauders, unwelcome visitors for nigh on a decade. And if I wanted to, I could set different rules for each cat. Even track where each cat was, and where it went the previous night. All our cats are outdoor cats, still keeping their mouser instincts honed, often bringing in rabbits and birds as well. Most of the time it’s okay; occasionally, when said mouse or rabbit or bird is alive, unharmed and running/flying/hopping freely around in the house, it makes for an interesting half an hour or so.

Lily, the youngest, is part of the iPad generation:

Yes, as the saying goes, there’s an app for that. You can get apps for kittens to play with.

Actually it’s not just kittens. There’s an app for everything.

And everything is connected.

All the time.

Everywhere.

[Pedants will write in and tell me that everyone is not connected, that there is a digital divide, that there are many parts of the world where this is just not true. And I will agree with them. And yet.  Please just go to Mumbai or Beijing or Nairobi and look around you and realise the future’s already here, it’s just not evenly distributed.]

This interconnectedness and smart-phone-ness and app-ness of everyone and everything is blurring the lines between telecoms and general-purpose IT and embedded systems, bringing us a world of opportunity we could only dream of not that long ago.

Soon we will be able to say “If the doorbell rings and if the only person in the house is my aged hard-of-hearing aunt, then please switch on the light near the flat-screen TV in the living room because she doesn’t like using the computer to watch television, she’s old fashioned that way; but before you do that check if the person ringing the doorbell is that pesky brush salesman; if it is he, then don’t disturb my aunt, just let the Alsatian next door know that his favourite trouser leg, still attached to its human owner, is back in town.”

We live in exciting times. Silos have broken down.

Everything that’s connected is capable of sharing streams of information about state and status, accompanied by rich metadata to provide context. We’ve been able to do all this before, but not for everything and not instantly.

The instant bit changes everything. There are so many things we do that are based on information not being real-time. We rely on historical stocks of information because we’ve not been able to rely on anything else.

Like censuses. Remember them? As we’ve moved from millions of people to billions of people, they’ve become harder to do, more expensive, less accurate and very very time-consuming….. as long as the way we did them was the time-honoured way.

But some time-honoured ways are out of time and no longer deserving of honour. At a level of abstraction, the LIBOR scandal is a classic consequence of using a “stocks” process rather than a “flows” process.

There are LIBORs everywhere. The way we value things; the way we count things; the way we sound out opinions; the way we measure things ….. we’ve had to put up with taking samples of information and then extrapolating the samples according to some convention or another. Much of the information we see and use from traditional sources can be characterised as low-frequency snapshots of discrete samples aggregated and represented according to some agreed convention or other.

It doesn’t have to be that way. Now terms like real-time and all and continuous can be used without sounding like hyperbole. No more samples, no more extrapolation, no more conventional representations. Just the facts.

A whole new way of looking at information.  Information that emanates from everything and everyone, in a world where everything and everyone can “publish” and “subscribe”.

Information that comes as a live, “real-time” stream, as a series of streams; streams that get aggregated, filtered, personalised; streams that get aggregated, trended, analysed and projected; streams that help tell the past, the present and the future of  many small yet important things. Things that affect our health, our education, our welfare.

The technology architectures that have emerged over the past decade or so are built for this new paradigm, one of streams and filters and drains; one where there is no longer any difference between telecoms and IT and embedded systems; one where information comes from subscribing to flows and gaining insights from those flows, using personally chosen filters to make every firehose look like a set of comprehensible capillaries.

New paradigms come with new problems. The debates we continue to have about identity and intellectual property and the internet.

Subsidiary debates about privacy and confidentiality and sharing and anonymity and censorship and all the regulation those things bring with them, often creating new forms of trade protectionism as barriers get drawn on political lines. These debates have been going on for some time now. And in a John Lennon kind of way, life has carried on while we’ve been busy making noise about all this.

And life will carry on.

This new interconnected always-on publish-subscribe streams-and-filters-and-drains world is on us.

The silo was the symbol of stocks. The stream is the symbol of flows.

The stream is where some people live, and where more join every day. Tools continue to emerge, tools that help us harness the stream and navigate it. Tools to provide context; tools to filter; tools to visualise; to personalise; to aggregate; to imbue with enhanced meaning; to analyse; to project, forwards and backwards in time; to move, in geography and in culture.

The stream is here to stay. And I for one am excited.