Ovum: Big questions for those looking for answers in big data
"Some complex systems, such as the weather, are oblivious to our predictions. The process of human development, in contrast, reacts to them. Indeed, the better our forecasts, the more reactions they engender.
These observations of Professor Yuval Noah Harari, one of the outgoing American president's favoured authors (Sapiens: A Brief History of Humankind), are particularly apposite right now as pollsters, experts, and analysts in various parts of the world examine their election forecasts and models.
At a time when organisations and nation states have more data and more compute power than ever before, the notion of "the more we know, the less we can predict" should provoke "big questions" for those working with "big data.
The future is disobedient: the more we know, the less we can predict
Analytics, as part of the broader topic of exploiting data for business value, remains one of the most widely discussed topics in the enterprise applications software market.
Given this continued surge in demand, vendors are eagerly looking for new ways to offer business analytics and market insights through a broader range of channels, be it self-service analytics, embedded business intelligence, cloud analytics, big data analytics, or the infusion of predictive analytics into real-time applications.
And as if this were not enough, collaboration, productivity, and business application vendors are introducing rules engines, machine learning technologies, natural language processing systems, bots, and cognitive capabilities to their products too.
Power users and early adopters are already being drawn to these new capabilities, combining them in various ways to make sense of big data, but the "law of unintended consequences" may yet present users with some unexpected outcomes, especially where human-centric data is concerned.
The notion of "setting information free" and providing analytics to the entire workforce is of course laudable, and one that this analyst espouses. But business leaders and CIOs must be mindful of its potential impact.
In the words of Professor Harari from his latest book, Homo Deus: "Knowledge that does not change behaviour is useless. But knowledge that changes behaviour quickly loses its relevance. The more data we have and the better we understand history, the faster history alters its course, and the faster our knowledge becomes outdated.
This paradox is one that few vendors and data scientists seem to openly acknowledge, and even fewer CIOs have considered. However, one thing is apparent: the future is still "disobedient." No amount of data and compute power can accurately forecast the outcome of events that are dictated by human emotions, sentiments, and feelings. But for how long?
