Where did Big Data go anyway?
Just a few years ago the hype was almost unprecedented. Big Data. Our world would never be the same. Utopia for businesses and society was jut around the corner. Kind of the same refrain we heard in 2009 as social media slammed into the world like an asteroid on steroids. A new wave of democracy would sweep the world. The opposite has happened and democracy is in decline globally.
Then after 2018, Big Data, the saviour it was to be, seemed to falter, fade and finally, fizzle. Except it hasn’t really. Quite the contrary, Big Data is being used everywhere today. If you have a smartphone upon which you may very well be reading this, then Big Data is in the background.
Big Data Had a Marketing Problem
The thing is, it was always rather hard to get ones head around the concept of Big Data and what it really meant. I helped market two Big Data conferences in Halifax in 2015 and then 2017. We drew an audience from around the world, especially when we partnered with the little known, but extremely influential World Productivity Congress. We attracted Ray Kurzweil as a keynote and the late, esteemed management guru, Clayton Christensen. Big Data was Big Buzz. Until it wasn’t.
So what took it’s place? Nothing. What was easier in marketing terms was two words we hear a lot about today; Artificial Intelligence, or AI if you will. It’s far easier to wrap our minds around this term. It has become an umbrella term for all the various branches and disciplines than comprise AI; Machine Learning, Neural Networks, Natural Language Processing and so on. All are subsets of AI.
While there were some great examples of Big Data applications, the words themselves don’t tend to point to any one solution. The reality is, Big Data is the oil that makes AI and advanced analytics platforms work. The Economist and others have called data the new oil and well, Big Data is that oil.
Big Data has Become a Digital Lubricant
You can’t run Siri, Alexa or Hey Google (could Google just please come up with a name like Apple, Amazon and Microsoft and stop using “Hey Google”? They’d get better consumer adoption) without massive amounts of data, like petabytes worth of data. Want to develop an AI tool that detects breast cancer? You need Big Data. Want to teach a self-driving car? You need Big Data. Want to build an AI that fights parking tickets? You need Big Data. You get the picture.
Big Data is Alive and Thriving
The reality is, Big Data never went away. For Hadoop and MongoDB, they’re still in use, but they’ve just become tools that feed a bigger machine — Artificial Intelligence and Advanced Analytics platforms. No longer does MongoDB even really use Big Data as a key marketing point. Big Data as a concept has been relegated to a more behind-the-scenes world. Amazon RedShift, Microsoft Azure, Tableau, Google Data Studio, they all need Big Data to be effective. But they’re the names that come to the forefront.
Big Data is all about variety, velocity and volume. You need a huge volume (massive amounts of data), then you need a lot of variety in data sets and then you need to process it at speed, velocity. These are the 3 core “V’s” of Big Data.
If a company is going to build some kind of AI solution or deliver some kind of AI driven SaaS business, behind them all is Big Data. In order to conduct machine learning or natural language processing you need lots and lots and lots of data. That’s Big Data, that’s the oil that helps make a solution work.
So Big Data isn’t dead at all. Far from it. It’s just that AI is the engine and Big Data is the oil that makes it work as well as a number of other applications.
