Researchers at MIT say they've found an algorithm that can tell what's going to be trending on Twitter hours before it happens, opening the door to modeling other kinds of data. Twitter's trending topics — the ones shown on its home page — are selected automatically, based on total number of tweets and a jump in how often a topic is mentioned. But Associate Professor Devavrat Shah and student Stanislav Nikolov say that by building a statistical model based on a sample set of 400 topics, they can tell with 95 percent accuracy whether or not a topic will end up trending. In tests, topics can be predicted an average of an hour and a half before they trend, with some predicted up to four or five hours beforehand.
95 percent accuracy, an average of an hour and a half beforehandWhile this development could be used to optimize advertising linked to topics on Twitter, Shah also thinks it could also be used to predict other things that update at regular intervals. Unlike machine learning that looks for different parameters that might factor into a trend, this system essentially recognizes statistical patterns. If the pattern of a new topic resembles ones found in its library of past data, the topic is seen as more likely to trend.
As long as there's enough computing power available and a consistent relationship between past and present data, this method should produce solid results, even if researchers don't know what factors are in play. Of course, since the team is essentially reverse-engineering an algorithm written by Twitter, it's likely much easier to find trends in it than in less predictable real-world data. More details will be presented at the Interdisciplinary Workshop on Information and Decision in Social Networks in November.