Literary pursuits are beset on all sides, it seems. Not only do computers rank student essays about the same as human reviewers do, a working paper led by Harvard Business School professor Michael Luca has found that top critics, in aggregate, give roughly the same overall scores when reviewing books as Amazon consumers. The paper, which was published last month, attempted to determine what factors (besides pure "objective" book quality) affected professional reviewers. What he found was that in most cases, both consumers and critics gave similar scores, but with a couple of important differences. Professional reviewers, for instance, tended to give higher ratings to authors who had received widespread attention or won book awards than Amazon reviewers, while Amazon reviewers were more favorable to first-time authors than critics were.

One of the reasons for this, the study found, was that critics were 25 percent more likely to review books whose authors wrote for their publication, and the reviews for the books were an average of 5 percent higher. This doesn't mean, however, that they're explicitly favoring these authors. After controlling for other factors, the researchers concluded that this was both because critics could get more information about these books and because a given paper or magazine tends to write to the tastes of a certain kind of audience. It seems rather obvious, but the basic conclusion seems to be that media outlets share a collective house style, and that the aggregate of these outlets roughly matches the opinions of the general public (or at least the general public on Amazon.)

As with book reviews, people are sure to take out of this study what they bring into it. Luca's preliminary research has been used to justify the work of critics by pointing out that they give higher ratings to award-winning books, and therefore to "works of great merit that puzzle or turn off a significant minority." Conversely, this working paper, which seems to show similar results, has been used to deride them as elitist or "closed-minded." If you'd like to draw your own conclusions, the entire working paper is available online here as a PDF.