Embedded in Academia

Reviewers Get Worse Over Time

This is a depressing paper. The study shows that:

92% of peer reviewers deteriorated during 14 years of study in the quality and usefulness of their reviews (as judged by editors at the time of decision)

The quality of reviews that I write has definitely decreased over the 12 or so years that I’ve been doing it. The main factor is time: I’m far busier than I used to be and I’m asked to review many more papers, grant proposals, etc. There is just not enough time anymore to write a pages-long discussion of the pros and cons of a paper. Ennui is also a factor: there are certain classes of bad paper that I’ve seen so many times that I can’t be bothered to write a detailed description of why it’s bad. Additionally, senior people often write bad reviews because they farm them out to very inexperienced students, postdocs, etc. and then fail to rewrite the review themselves (for some reason, students often write the harshest reviews). Another factor is that as people become more senior, they can start to lose touch with the technical details.

A reasonable response to this trend is to ensure that younger researchers make up a significant fraction of review committees. I deliberately did this the last time I formed a program committee. This has the additional benefit of giving younger researchers experience and visibility that they require at early career stages.

regehr

,

3 responses to “Reviewers Get Worse Over Time”

  1. On the other hand, my ability to write a quality review has gone up over time (that is, if there was an absolute number you could assign to rank the quality of a review I am capable of writing given enough time, that number is larger now than it was even 5 years ago, I’d say).

  2. Hi Robby- you’re absolutely right. However, in my case at least, I’m pretty sure that this increasing ability has been trumped by time constraints.