The end of Big Data

With the rising interest in artificial intelligence (AI), Big Data is increasingly thought of as a must for training AI.

This is detrimental to what AI is supposed to achieve due to biases and limits inherent in Big Data. We believe an AI technique known as reinforcement learning holds the key to making AI truly disruptive. Reinforcement learning algorithms are differentiated in that they do not require the injection of Big Data. Instead, they learn the same way as a child does, via trial and error. The biggest recent advances in AI have come via this methodology, which puts algorithms, instead of data, at the centre of the AI revolution.

From when the first hominids started making use of fire, technology has always been a tool for enhancing human productivity. We can broadly divide much of human endeavour into those activities driven by fact and those driven by opinion. History tells us that while opinion-based productivity gains are cyclical (eg efficiency in political structures), fact-based productivity gains are largely directional (eg transport efficiency) and, hence, of greater importance to humanity.

While conventional wisdom holds to data-driven productivity gains being the only option, when it comes to fact-based activities, we argue that technology will begin to rely less and less on historical data to drive productivity. Many an article has been written about how data is the oil of the current industrial revolution; we think this view overplays the importance of data and underplays the importance of algorithms. We predict an end to the hype around Big Data’s broad-based impact given the rate of advancement in alternative AI techniques such as reinforcement learning. This replicates the paradigm of learning by trial and error, leveraging rewards or punishment to strengthen or weaken future behaviours. It is not reliant on external knowledge sets (the raison d’être of Big Data), and like a child, constructs its own knowledge.

Click here for the full note.