Although I work with a great collection of students, I’ve come to realize that my selection of students is based too much on luck and not enough on my skill in evaluating them. Therefore, I have a serious question for the professors out there, and particularly for the computer science professors in systems-related areas: Given a student who is interested in working with your group (particularly at the PhD level), how do you decide whether the student is the right one?
To get things started, these are the basic skills I need in a research student:
- Ability to clearly explain their thinking and their results. If someone cannot, the rest of the group (and I) become stuck in a protracted struggle to reverse engineer their ideas and results. Also, people who cannot explain themselves clearly seem less receptive to explanations and instructions that I think are totally clear.
- Willingness to work hard, and remain interested in a problem and motivated to work on it, over a period of years. Luckily, failures in this area usually manifest early. However, when someone loses focus late it creates problems that are tough to solve since both sides have invested so much time and energy by then.
- Capacity to be self-monitoring: there has to be a tight mental feedback loop where they notice errors in thinking, math, and code, and correct them spontaneously. A failure of self-monitoring greatly reduces a student’s productivity because so much time is spent recursively exploring unpromising approaches and just being stuck. My productivity also takes a big hit because the student requires micromanagement and extra oversight: I have to double-check results and end up spending quite a bit of time worrying that I haven’t asked the crucial question that would have uncovered a faulty line of thought. Since most students are doing the right thing 98% of the time, the probing questions I ask at group meetings are without a doubt quite annoying. However, since a low-level error can easily invalidate a high-level result, I have to err on the side of paranoia.
- Ability to develop a second, much looser feedback loop where they think about things learned by experience, by conversations, and by reading, and apply the results to their work. They have to be able to learn good taste in problems and solutions and to be willing to try new things. Failure at this level is perhaps the easiest to deal with out of the problems I’ve listed here because I need to supply high-level course corrections anyway, as part of the advising process.
- Competence in computer science, in program development, and in math. Ideally, people not fulfilling these requirements are filtered out by our graduate admissions process, and in fact most people we admit are smart enough and good enough programmers to earn a PhD.
Reading back over this, it looks like the most important thing I need to do is develop a collection of interview questions that do a better job evaluating potential in areas 1-3 above. I’d appreciate any advice.
14 responses to “Interviewing PhD Students”
What percentage of the people satisfying (5) that you even consider then fail to come up to scratch wrt (1), (2) or (3)?
I’ve seen several sorts of interviews, but the best IMO asks the student to make a presentation on their last piece of “research” — final year project for example — in front of faculty /and/ the other candidates. How students respond to comments / criticism is very enlightening, and tells you a lot about their own thought process and how they might interact with a group. Also, if they ask good questions of others, that’s a very positive sign.
I’m also a big fan of asking candidates to submit code, either in an “exam” setting (viva or otherwise) during the interview process, or by submitting links to open source projects they contribute to.
For evaluating #1: I generally ask the student to describe the largest and/or
most interesting project they’ve worked on, and to tell me *why* it was cool:
in school, in a previous job, on their own, or whatever. I try to make it
really clear that they are welcome to describe their own projects; some seem to
think they have to describe something somebody else told them to do, and the
students that can give the best explanations of cool stuff they’ve done on
their own initiative often are often the ones I really want. And the ability to
pick out why something was cool, even if it was an assigned project, tells me a
lot about whether they’re going to have the skill of explaining why their
research is valuable.
This can also give you a bit of an idea on #2: while being passionate in their
explanation doesn’t guarantee that they can do #2, you can often tell whether
they got stuck, got sick of the project, gave up, etc. I should say, though,
that I’ve seen some students who started out jumping around from interest to
interest and eventually settled on the *right* problem and worked hard and long
on it. So the fact that they haven’t done #2 in the past doesn’t necessarily
mean they won’t in the future. The only way I know of to assess #2 if it’s not
obvious from their history is to do an independent study or something before
bringing them on as a full student.
In the past, I used to ask prospective students how they would approach solving
some problem related to the area I expected to have them work in. I’ve
basically given up on this: I never got good enough at asking a question with
sufficient detail but that didn’t take me too long to set up, and trying to ask
quicker questions usually resulted in vague questions and vague answers that
weren’t very valuable. Asking someone to understand and comment on research on
the spot in an interview just doesn’t work well. And I’m not sure how a well a
generic set of questions will work for this purpose – what I’ll have them work
on is in the intersection of what I have funding to do and what the students’
strengths are, and that intersection will be different with different students.
Lately, I’ve been asking the student to read a couple of papers (usually mine)
related to the area, or having them go off and spend a little time using the
system I’m going to ask them to improve. Then, we meet again later to discuss
a specific initial project for them based on this ‘homework’. I’ve had a few
students come up with some pretty solid ideas on their own this way. This does
require two discussions, but you don’t have to give everyone homework if you
don’t think they’re likely to work out.
Also, I highly recommend doing at least some interviews over lunch. 🙂 It’s a
lot more pleasant for all involved, and lets you spend a long time with some of
them without eating more into your day.
Hi Michael- I don’t have a good estimate of the percentages, but basically each item in the list is motivated by one or more negative experiences…
Thanks Sarah and Rob!
While the criteria you list are very useful, I don’t know if it’s reasonable to expect them to be present upfront. I think especially with 1-3, training can go a long way. Now my experience is with theory or theory-ish students, so YMMV, but usually I just look for signs of deliberative thinking and a certain level of energy (so hints of 2, and 1), and then work with them from there.
I like the idea of the group interview, may have to try it out next fall. This last fall, we interviewed around 15 students, so this would be a big timesaver.
I have no experience here, but I feel most/all students admitted to a PhD program have sufficient amounts of those five traits. Those traits are necessary to succeed as an undergrad.
I would add another trait: humility, teachability, or a willingness to learn. This is hard to determine in an interview, since the interviewee is attempting to show how awesome he or she is.
I’ve seen students who looked pretty good on those scales at the beginning, but refused to improve and therefore didn’t, and other students who looked pretty bad on those scales at the beginning, but worked very hard to improve and therefore did. I’d MUCH rather deal with the latter.
But that’s criterion number 3, isn’t it?
I’d also add a dual to #2: Willingness to drop a problem-solving approach (or at least explore other options) when it doesn’t look likely to pan out.
Hi John, I’ve talked to my own supervisor for a bit about this issue. His key statement with respect to #2 was that when he started as a professor, he thought motivation was important, and it took him several years to discover that it’s all about the applicant’s personality. Most excelling PhD students that I have made were ambitioned and had fun solving difficult problems with not so obvious solutions, whereas motivation is typically a short-term property. The issue of personality also dovetails with number #6 that I would add: trustworthiness.
However, these are probably the most difficult characteristics to get reliable answers for in a job interview. If you find a way to extract them from an applicant, I’d be glad to read about it.
Hi John,
My personal experience from the other side (aspiring PhD student) is that 1 and 4 are “trainable” attributes picked up in graduate school (or elsewhere, but probably not undergrad), 2 and 3 are personality traits, and 5 is direct from admissions criteria and other filters.
To find out about a students work ethic and ability to self-monitor, you can probably craft some open-ended questions that ask about term-length or longer projects where students faced trouble and how they resolved setbacks. This should give a clue about their drive and ability to critique their own work.
I advise all of my PhD students entirely through Twitter and SMS text messaging. I use an automated system to reply to most student questions. I have what you might call a “Single Instructor Multiple Student” approach, giving advice and direction on a massively-parallel scale. Of course I don’t reveal to each student that they are working on exactly the same project, and receiving the same advice, as 511 other students in my student vector.
If your conscience compels you to take a more traditional approach, then I recommend accepting only robots as students. Their level of commitment cannot be matched by the human students.
If your university has a minimum quota for human students, then I recommend using functional MRI brain scans to evaluate their Ph.D. potential.
In the olden days, I used to drop a single Ph.D. diploma in to a room of grad students and lock the door, and let the “qualifier” begin, Battle Royale style. But this practice might not be lawful in your country.
0x1337c0de- Thanks, this is very helpful. I am also working on a SIMS-based approach to advising. Details will appear in a future post. Feedback would be very useful as I’m almost certain that my heuristics will not (initially) be optimal.
Regarding fMRI, I’m not convinced of the applicability here. But if you have more details I’d love to hear them.
Hi all- I got behind in replying to comments on this post, but the advice has been really helpful. Thanks everyone! Perhaps I’ll post a followup in a few years.