Computer science — at least in the areas that I work in — is conference-driven. Since journals go unread, it’s important that conference program committees make good decisions about which papers to accept. The established way to do this is to hold a program committee (PC) meeting: the entire committee holes up in a highly air conditioned meeting room for a day and argues about every paper that’s not an obvious reject (occasionally we also manage to avoid arguing about papers that are obvious accepts).

The cost of in-person PC meetings is substantial in terms of money and time. Just as a random anecdote, last January I attended the EuroSys PC meeting in Paris; this trip used up enough time and money that I couldn’t easily justify traveling to France again a few months later to go to the conference. This was a bummer since I’d have loved to go (the volcano in Iceland, however, would have stopped me from getting home in a timely fashion).

The alternative to the in-person meeting is to have a remote meeting. This post is not about comparing the merits of in-person vs. remote meetings, but rather about how to run a remote meeting: a topic that is fresh on my mind since I just ran a remote PC meeting for the Sensor Networks track at the Real Time Systems Symposium, where we evaluated 30 submissions. I’ve previously run a couple of in-person meetings so I have some basis for comparison.

The remote PC meeting could be designed to emulate an in-person meeting, for example using Skype or a conference call. I didn’t choose this option for my RTSS track for several reasons. First, it would waste people’s time: each PC member reviewed only 20% of the papers (normally this percentage would be higher, but since I had been warned to expect a large number of submissions, I invited a lot of people to be on the committee). Second, 20 different voices on a conference call is difficult; generally I’d consider five or six to be a reasonable maximum. Third, spending all day on a conference call just sounds unspeakably boring; I doubted that anyone (including me) could keep paying attention to disembodied voices for that long. I think a conference call would work fine for a small committee evaluating a small number of submissions. I’ve never tried a Skype session with anywhere close to 20 people and would appreciate stories or advice from people who have done this.

If the entire PC is not going to get together (actually or virtually) the alternative is for the group of people who reviewed each paper to discuss it and make a decision. One possibility would be to emulate the journal review process: the program chair simply reads all reviews for each paper, and makes the decision. My sense is that this would be unsatisfying for people, who generally enjoy the back and forth of arguing over papers. People like to come to a consensus. This discussion could be done in email, on the phone or Skype, in a text channel like IRC, or on a message board.

Since the conference software RTSS uses supported a decent message board, we used that. The process that I put to the PC members was that the reviewers for a paper should read all the reviews and then someone should propose a decision. From there, the discussion could proceed with others either agreeing or disagreeing. If nobody chimed in about a particular paper, I started asking leading questions: “Joe you didn’t like this paper but your review is marked low-confidence. Do you stand by the ‘reject’ recommendation?” I only had to do this in a few cases.

The only misgiving I had about this process was that it might give too much weight to the first comment about a paper. But, as it turned out, people felt free to argue with the first commenter and I think the decisions reached ended up being generally good. Of course, in-person PC meetings suffer from substantially more severe versions of this kind of unfairness where the loudest, most persuasive people can have an inordinate effect on the decisions that are made. An alternative would have been to ask a specific person (for example the most positive or most negative reviewer for a paper) to propose a course of action, but I didn’t think of this soon enough.

In principle, a conference accepts all acceptable papers and rejects the rest. In practice, however, the number of available presentation slots at the conference has a strong influence on the number of accepted papers. When the PC meeting is in-person, global constraints like this are easy to account for: once we notice that too few papers are being accepted, we can try to start making more generous decisions. When the PC meeting is fragmented into a lot of individual meetings, there is less global information and we risk accepting too few or too many papers. For whatever reasons, this didn’t happen at my track meeting and we accepted just slightly more than the expected 20%-25% of submitted papers.

I’m not claiming that a remote meeting is preferable or saying that I don’t like in-person PC meetings. It’s just that looking forward, I’m probably going to attend at most a couple of in-person meetings per year and turn down any invitations beyond that. I’d expect other rational academics to do the same. If oil prices rise significantly, the in-person meeting will die completely and the entire conference system will be in jeopardy.

In summary, I was happy with the remote PC meeting process. It puts a significantly larger burden on the program chair to help things run smoothly, and it also (potentially) gives the chair much more control over the decisions that are made. The best thing about the remote meeting is that it saved a lot of time and money for PC members, several of whom told me they wouldn’t have agreed to be on the PC if that meant attending an in-person PC meeting. I doubt that the resulting decisions about papers were worse than, or even much different than, the decisions that would have been made in person.

Michael Ernst’s advice on running a PC meeting is great, though I don’t buy his claim that the low bandwidth of the phone or a message board is a limiting factor in arriving at good decisions. A typical in-person PC meeting is actually a fairly low-bandwidth activity with quite a bit of irrelevant discussion and frittering away of time, unless the chair is very much on top of things. Also, by using a message board people can take time to compose well thought-out arguments as opposed to saying the first thing that comes to mind.