The conference system that we use in computer science guarantees that several times a year, each of us will need to review a lot of papers, sometimes more than 20, in a fairly short amount of time. In order to focus reviewing energy where it matters most, it helps to review efficiently. Here are some ideas on how to do that.
Significant efficiency can come from recognizing papers that are not deserving of a full review. A paper might fall into this category if it is:
- way too long
- obviously outside the scope of the conference
- significantly incomplete, such as an experimental paper that lacks results
- a duplicate of a paper submitted or published by the same or different authors
- an aged resubmission of a many-times-rejected paper that, for example, has not been updated to reference any work done in the last 15 years
These papers can be marked as “reject” and the review then contains a brief, friendly explanation of the problem. If there is controversy about the paper it will be discussed, but the most common outcome is for each reviewer to independently reach the same conclusion, causing the paper to be dropped from consideration early. Certain program committee members actively bid on these papers in order to minimize their amount of reviewing work.
Every paper that passes the quick smoke test has to be read in its entirety. Or perhaps not… I usually skip the abstract of a paper while reviewing it (you would read the abstract when deciding whether or not to read the paper — but here that decision has already been made). Rather, I start out by reading the conclusion. This is helpful for a couple of reasons. First, the conclusion generally lacks the motivational part of the paper which can be superfluous when one is closely familiar with the research area. Second — and there’s no nice way to say this — I’ve found that authors are more truthful when writing conclusions than they are when writing introductions. Perhaps the problem is that the introduction is often written early on, in the hopeful phase of a research project. The conclusion, on the other hand, is generally written during the grim final days — or hours — of paper preparation when the machines have wound down to an idle and the graphs are all plotted. Also, I appreciate the tone of a conclusion, which usually includes some text like: “it has been shown that 41% of hoovulators can be subsumed by frambulators.” This gives us something specific to look for while reading the rest of the paper: evidence supporting that claim. In contrast, the introduction probably spends about a page waxing eloquent on the number of lives that are put at risk every day by the ad hoc and perhaps unsound nature of the hoovulator.
Alas, other than the abstract trick, there aren’t really any good shortcuts during the “reading the paper” phase of reviewing a paper. The next place to save time is on writing the review. The first way to do this is to keep good notes while reading, either in ink on the paper or in a text file. Generally, each such comment will turn into a sentence or two in the final review. Therefore, once you finish reading the paper, your main jobs are (1) to make up your mind about the recommendation and (2) to massage the notes into a legible and useful form. The second way to save time is to decide what kind of review you are writing. If the paper is strong then your review is a persuasive essay with the goal of getting the rest of the committee to accept it. In this case it is also useful to give detailed comments on the presentation: which graphs need to be tweaked, which sentences are awkward, etc. If the paper needs to be rejected, then the purpose of the review is to convince the committee of this and also to help the authors understand where they took a wrong turn. In this case, detailed feedback about the presentation is probably not that useful. Alternatively, many papers at top conferences seem to be a bit borderline, and in this case the job of the reviewer is to provide as much actionable advice as possible to the authors about how to improve the work — this will be useful regardless of whether the paper is accepted or rejected.
I hope it is clear that I am not trying to help reviewers spend less total time reviewing. Rather, by adopting efficient reviewing practices, we can spend our time where it matters most. My observation is that the amount of time that computer scientists spend writing paper reviews varies tremendously. Some people spend almost no time at all whereas others produce reviews that resemble novellas. The amazing people who produce these reviews should embarrass all of us into doing a better job.
Update: Also see Shriram’s excellent notes about reviewing papers.
9 responses to “Reviewing Research Papers Efficiently”
I too skip the abstract, for exactly the same reason. I’m committed to reviewing the paper anyway, so it doesn’t really matter what the abstract says.
I find it easier for me to do a few passes over the paper, form an initial judgement with comments, and then iterate over “write review” and “refer to paper”. I find it easier to break the initial logjam of starting to write. Interestingly I will often change my initial rating as I write (either up or down).
John, then you’re in for unpleasant surprises from my papers, because (a) I often don’t have a “conclusion” section, and (b) I write my introductions at the very end, after the work is done, because that’s how I can say what needs to be said and not prognosticate about the lethality of unsound hoovulators (which, despite your levity, I hope your readers realize is a very, very serious concern with significant social implications, especially if those readers are NSF PMs).
However, there is a perfectly good strategy for those remaining papers (which is the bulk of the load) that I use. I begin my reading _with the related work_. This is remarkably effective for decent papers (and infuriating for ones with bad authors, because they don’t know how to _relate_ work). Because I have a pretty good fund of work in my head, by reading the related work I can see how the authors position their work relative to what has gone before.
So: I first read the “wrapper” matter and make a list in my head of what I expect to see discussed. Then I read related work. This way I find out that the work is closest to this prior work, pretty far from that one (though the wrapper-matter led me to believe otherwise), and in particular the key contribution is going to be 10% more soundness over Regehr, et al while achieving the same performance as Venkatasubramanian, et al. I also note what I’m surprised _didn’t_ get said in this section, because these could be places where the authors are trying to pull wool over my eyes.
In short, I TRIANGULATE. Now I have a space that the paper is trying to occupy, with its claims enumerated. With this I can commence reading the paper proper, checking off what I expect to see and not see and noting what I didn’t expect to see.
I made notes on the rest of my review process here: http://cs.brown.edu/~sk/Memos/Paper-Reviews/
I don’t mention this triangulation issue because anyone who’s reading the above (typically a young grad student) doesn’t have the related work databank so this method wouldn’t apply.
I really like the suggestion to read the conclusion first! A related tack that Jan Vitek suggested to me once was to judge the claimed contributions without worrying about the evidence for them, and if the contributions are insufficient you don’t need to spend lots of time reading the paper.
Another time saver that your initial list hints at is to use tiered reviewing. Why have three or even four people review a paper that is clearly a D? Instead, have two people review it first. If they both say ‘D’ then it’s done. Only papers with two non-D reviews get out of the first round, and get a third review in the second round. Then, at the end of the second round, only papers whose outcome is still contentious get a fourth review, or even a fifth. That way you end up getting the same number of reviews total, but you put them where they are needed. The drawback is that there are more internal deadlines that reviewers have to adhere to.
For some reason, this makes me think of writing interview feedback.
Shriram, I’m pretty sure I’ve never been assigned one of your papers to review — your style is pretty distinctive! I try to write my introductions last but in practice those are sort of fun to write so I end up writing them earlier usually. I almost always write the conclusion first but it ends up getting rewritten so many times that this doesn’t end up mattering too much.
I like the triangulation idea, and will try out your related-work-first strategy. As you say, though, this will end up being infuriating pretty often, since in some areas these sections are almost uniformly poor. I’ll update my post to include a link to your notes which include a lot of good material that I left out here.
Mike, definitely.
Suresh, I try to avoid a multi-pass back and forth process of reviewing when I have a big pile of papers to work on. And this works just fine most of the time. But yeah, difficult papers require multiple passes.
I have yet another approach, but it only applies to some papers. In papers with a substantial and important “Experimental Results” section, I often start reading by going there first, having perhaps skimmed conclusion/intro to get some general idea of the paper’s point. The most common reason I reject papers with a reasonably large experimental section is that the experiments are pointless/measuring the wrong thing/incomprehensible/too weak to show anything. If I know that, I can read the rest with a goal of seeing if the idea is so interesting (or supported by proof of some useful property) that experimental weakness can be overcome. If the experiments are strong, it usually tells me what the paper is really showing, even more strongly than a conclusion, and is a good candidate for a very careful reading and perhaps championing.
Doesn’t work for papers where experiments are not present or not really the interesting part, of course.
My first filter for reviewing manuscripts is to see if it’s going to be ‘open access’. It’s an automatic refusal for having me as a reviewer otherwise.
When I do review a manuscript, the first thing I do is look for what I consider a personal insult, as a reviewer: barely readable English. My native tongue isn’t English so when I submit papers I make sure it has been reviewed thoroughly for the language, as much as it is reviewed for its technical merit. Submitting an unreadable paper is insulting because it shows complete disregard for my time as a reviewer, and is a first indication for a loose approach to detail.
This isn’t an automatic reject, of course, but I might not do a thorough review if I need to struggle to read the manuscript. I can’t accept that I’d be required to put additional time into this just because the authors were lazy with their grammatical review.