The ancients had their prophets and sibyls and modern man has looked to data scientists, futurologists and the astrology columns of tabloid newspapers for guidance on his destiny. Yet all of these seers may now be outdone by an algorithm.
That is the implication of a paper by two data scientists at Massachusetts Institute of Technology in the United States. In the paper, which will be presented to a conference in Paris this week, Max Kanter and Kalyan Veeramachaneni claim to have created an algorithm that predicts human behaviour faster and more reliably than the majority of its human competitors.
The algorithm, based on work in Mr Kanter’s master’s thesis, was used to create a program that they called the data science machine. Not only could the machine process vast quantities of data and trillions of possible outcomes, it was also able to seek out patterns and variables and use these to build a model to predict how people would behave.
The task of choosing which data points were of most relevance was considered a job for the human mind, involving intuition and judgment. In data science competitions, teams of scientists typically spent many months on the task. The data science machine was said to have completed the same work in between two and 12 hours.
The two MIT computer scientists tested the machine’s performance in three competitions. In one, contestants had to predict the likelihood of a student dropping out of an online course; in another, they were required to predict the excitement that would be generated by a crowd-funded project.
Advertisement
A third contest required teams to predict whether customers who bought from a merchant during a promotion were likely to return as repeat customers.
The machine beat 615 of the 906 entrants, the scientists wrote. “In two of the three competitions we beat a majority of competitors, and in the third we achieved 94 per cent of the best competitor’s score.”
Mr Kanter and Mr Veeramachaneni hope that the machine and its algorithm, which they call Deep Feature Synthesis, will prove a huge labour- saving aide for data scientists.
It has obvious applications in commerce, in analysing consumer behaviour, and could also be harnessed by product designers.
“We view the data science machine as a natural complement to human intelligence,” Mr Kanter told MIT’s newspaper.
Advertisement
“There’s so much data out there to be analysed. And right now it’s just sitting there not doing anything. So maybe we can come up with a solution that will at least get us started on it.”