Big Data: It’s Not How Big It Is, It’s How You Use It

If you haven’t heard about Big Data this year, please tell me your secret. Tell me how you managed to avoid hearing about it. I want to know. Really. There are days when I want to be in a place like that. Desperately.

For some time now we’ve been hearing about device proliferation. A classic 10x market. People tell me that mainframes are numbered in the tens of thousands, minicomputers in the hundreds of thousands, PCs in the millions, smart mobile phones in the billions. Smart devices in the tens of billions.

These tens of billions of thingummybobs are getting busier and busier as they sense and observe and record everything and everyone around them, saluting the things that move and painting the ones that don’t.

In a perfect world, this would mean that there’s a lot of good data being produced, data that should prove useful to improve our lives. So that means there’s a market for software that helps us crunch the data into something useful, find the data we need, see the data in ways that can help us extract meaning and act on the meaning.

Even though this world is far from perfect, I’m glad to see that all that is happening. VCs have been plunging into Big Data for a while now. Data visualisation tools continue to get better, better and better. And search is beginning to do something about its verb-based future.

And yet……

And yet I have this sense of unease.

You can have lots and lots and lots of data, Big Data. You can have wonderful Big Data Crunchers, tools to help you do something with the data. You can have the world’s best visualisation and search tools.

But they all mean nothing unless you can act on what you see.

Innovation takes place through adoption into practice, and not just through invention and disruption. Until something is being used, nothing actually changes.

Big Data becomes useful when it leads to action.

That action takes place across a wide spectrum. At one end the actors are all machines. And we run the risks that Kevin Slavin alluded to in his TED talk, How Algorithms Shape the World.

At the other end all the actors are human. Shouting from the rooftops about overload, seeking to convert firehoses into capillaries. With limited success.

As Clay Shirky so memorably pointed out, there is no such thing as information overload, it’s all about filter failure. We need better filters, something I’ve been writing about for a while now.

Between the two extremes there’s a universe of space where we have a lot to learn. The more I read about the Air France tragedy the more I feel the need to look deeply into this, how human beings cope with decision making amidst such complexity. Just reading the IEEE blog coverage here and the Popular Mechanics coverage here gives us pause for thought.

My concerns get heightened when I realise major segments of the industry I’m part of can’t stop rubbing their hands with glee as they intone “big data, big big data, big on-premise servers, big on-premise storage, big processor licences, big profits…. Christmas is every day”.

That causes new problems related to the quality and reliability of the big data, as multiple sources take snapshots at different times. When I worked in capital markets, working on out-of-date data could cost millions in seconds flat; it became very very important to know how “live” the data was, and to keep checking that the data was real and up to the second (or even millisecond).

So we have the risk that there’s a lot of Big Wrong Data.

But you know something? These things can be solved. We can learn how to process the data more effectively, present it for visualisation more elegantly, check for its validity more ruthlessly, use the best software and services to do all this, use machines to filter and humans to curate. We can do it all.

And find ourselves in a situation where all we do is “wait faster”.

Because there’s one more thing we have to change. And that is this: the way we make decisions. And then do something.

Some years ago, I remember a story that was used to illustrate the importance of a particular book. [Sadly, I can’t remember the book any more, other than it may have had something to do with Price Waterhouse and may have had a yellow cover]. The story asked a simple question. Five frogs on a log [Aha, that was the title. Five frogs on a log!] Where was I? Five frogs on a log. Four decide to jump off. How many are left?

And the answer was… five. Because, as the authors point out, deciding isn’t doing.

The way we’ve allowed email and calendar to become part of our work lives, decision-making seems to be perennially poor in large, often still hierarchical, organisations. Email chains fragment and break and become divergent infinite loops. “Snooze, you lose” gets stated sometimes but rarely acted on. Getting the right people together has become harder and harder, process delays begin with scheduling time. Everything is at priority one in the scheduling process. If compromises are sought on the quorum front, they are temporary, evanescent. Infinite loop problems recur.

Big Data can have Big Effects.

That excites me.

Big Data can create Big Problems, sometimes with tragic consequences; we need to take extreme care.

Big Data can lead to Big Waste in on-premise activity, and we need to take care here as well.

Big Data can have a Big Positive Impact on industry, on education, on healthcare, on jobs (the subject of a separate post I will try and write this week) and even on government. People like Tim O’Reilly, Tim Berners-Lee, Nigel Shadbolt, Wendy Hall et all have worked hard to get everyone to understand what is possible here, and I am a convert.

But the cultural changes that have to take place in institutions should not be underestimated. Institutional immune systems can render all the big data useless by rising up to slow the processes down, make decisions harder to take, make action harder to initiate, make outcomes harder to achieve.

Big Data means Big Changes.

It’s Not How Big It Is, It’s How You Use It.