Peer Review Poker

Peer review is a bureaucratic, consensus-based approach to making decisions. Thus, it is not inherently entertaining and authors like myself need to amuse ourselves as best we can. One of the games I like to play is peer review poker, where certain combinations of review scores are more desirable than others.

Straight: Review scores form a continuous sequence. A straight, particularly if it is anchored at the high end of the spectrum, is extremely desirable because provoking a diversity of reactions means that there is something interesting about the work. The best kind of straight — which I have accomplished only one time, that I can remember — is when each possible score is received exactly once.

Flush: All reviews are the same score. This is not particularly desirable as it indicates a predictable, uncontroversial paper. Of course, if all scores are “strong accept” then predictability doesn’t seem so bad — but this hardly ever happens. The other extreme, all “strong reject” scores, is certainly achievable.

Obviously, two of a kind, etc. are very common peer review outcomes and are not particularly interesting. By taking reviewer expertise into account it should be possible to define a royal flush but probably that’s pushing the analogy too far.

4 replies on “Peer Review Poker”

  1. A great practical joke is to wait until about the day before the accept/reject, then send out a fake acceptance email. Actually, this has happened “by accident” at least twice that I know of, and once to me. (Later, I got another email that the paper had actually been rejected.) I recall how I felt at the time, and thought this would be a terribly fun thing for everyone to experience. No?

Comments are closed.