Big Data’s Big Problem

info and energyA few months ago I had the opportunity to attend a panel discussion at MIT about the powerful changes Big Data could make possible in our lives.  A little over an hour later, frustrated and angry, I left – vowing never to buy a blue shirt again.

Big Data does not create anything new. It just enables us to see more of the information that constantly swirls around, through, and inside us. As with energy, how productively we use all this information is entirely dependent upon us.”

On the panel were two professors from MIT, two company CEOs, and a journalist/blogger who served as moderator. It all seemed very promising.

Or so I thought…

Unfortunately, what had been billed as a stimulating discussion about the possibilities of harnessing extremely large data sets in ways that could enable us as a society to do radically new and exciting things quickly devolved into a conversation between the two businessmen and the moderator about the future of men’s clothing sales.

Big Data, they breathlessly opined, would finally make it possible for retailers to know their customers so intimately that they could take data showing that Jim liked blue shirts, cross that with data that indicated Bob and Jim were friends and made roughly the same income, and determine that Bob probably liked blue shirts, too – and should be shown them as soon as he walked in the store so that Bob, his every need already anticipated, would simply snatch up the blue shirts and head straight for the cash register.

Ah, what an amazing future awaits us: Big Data will fling open the doors to the best of all possible worlds, where we live out our days comfortably attired and content in our beautiful new blue shirts!

The folly of all this was not lost on the two MIT professors. The younger one gamely tried to inject himself into the conversation at first, but then spent most of the remaining time slouched in his chair, a pained smile on his face.  The other simply slipped off into his own world, staring at his feet and probably trying to see how many prime numbers he could count in his head before he could finally leave Kresge Hall and get back to his research.

Meanwhile I sat there, steaming.  There were no new ideas here – just a rehash of the same old ideas I’d heard from marketing when I was an executive at CompuServe some 20 years ago.  The only difference was the amount of data accessible today had grown.

Note that I said “accessible” and not “available” or “created.”

One of the tenets of information theory is that the amount of information in the universe is a constant. It neither grows nor shrinks.  Conservation of Information means the world is a very “noisy” place, and that the signals, or “meanings,” we extract from all of the available information are simply different ways of harnessing the bits (literally) and aggregating them in ways that we discover are productive for us, much in the same way that we harvest energy to produce and distribute goods, heat our homes, and light our way. Some people even argue that the two – information and energy – are ultimately one and the same, or at least that information can be converted into energy (which, applying the principle of conservation of energy, equates to pretty much the same thing…)

Big Data does not create anything new. It  just enables us to see more of the information that constantly swirls around, through, and inside us. As with energy, how productively we use all this information is entirely dependent upon us.

And therein lies Big Data’s Big Problem: Small Minds.

Small minds say, “With more data, we can do more of the same – only better!”

Bigger minds – like those at MIT’s Computer Science and Artificial Intelligence Lab –  realize that the exponential increase in the amount of data we can capture requires new ways of thinking if we are to fully understand it, let alone act upon it.

As with energy, information – while constant – is also constantly in flux. The power of our computers to capture and process this flow of information is rapidly exceeding the limits of our brains to hold let alone comprehend the vastness of what is out there. We are forced to develop heuristics – rules of thumb – that help us truncate the information into discernible patterns that make sense to us.

Plato’s Allegory of The Cave remains relevant today – perhaps even more so as we grapple with the implications of computers that can not only gather and process more information than we can hold in our tiny brains, but also reach “conclusions” about the information in ways we don’t fully comprehend.

The limited computing capabilities of our brains means we can only see the shadows of what is possible flickering on the wall of the cave. Similarly, however, our human ability to reason and make judgements ultimately shapes everything we do with the information around us. The challenge is to develop new heuristics and algorithms that manage to bridge the gap between computer power and human reason and enable us to think and act differently – both as individuals and as societies.

One thing, though, is certain: If all we do is harness this power to help sell more blue shirts, we will have failed miserably as a species.


cv3Originally posted at Chris Varley is a member of External Science & Technology Program at The Goodyear Tire & Rubber Company and serves on the Prosperity Team of Scotland’s New Media Partners. The views in this blog are his own and do not necessarily represent the views of either The Goodyear Tire & Rubber Company or the Prosperity Team of Scotland’s New Media Partners. You can follow him on Twitter @Foresights or connect with him via LinkedIn.



Add a Comment