In Hanoi, as the story goes, the French placed a bounty on rat pelts. The locals responded by farming rats. A child who gets candy for cleaning up a big mess is likely to create another mess the next day. These are perverse incentives: incentives that have unintended and often undesirable side effects. As a particularly stupid example, I recently decided to start putting only one sugar cube in my morning coffee and then caught myself pouring small cups of coffee and having two or three.
Once we see the pattern it should be easy to predict when happens when you reward professors, postdocs, and grad students for producing many publications. The rewards are significant: a long CV can get a candidate in the door for a job interview or permit an assistant professor to keep her job during a tenure evaluation. Obviously numbers aren’t everything, but they matter a lot.
It’s true: there is a large number of low-quality publications being produced. I end up reviewing maybe 100 papers per year, and quite a few of them are just bad (I won’t try to define “bad” here but I took a stab at this earlier). I make an effort to be selective about the things that I review, turning down many requests to review journal papers and a few invitations to be on program committees each year.
The recent Chronicle article We Must Stop the Avalanche of Low-Quality Research says that the main costs of the avalanche are increasing the reviewing load, raising the bar for younger researchers, encouraging shallow inquiry, creating a difficult-to-surf ocean of results, and hurting the environment. There’s some truth to all of these, but I’m not sure they represent the main costs. For one thing, the peer review system is fairly effective at weeding out the bad stuff, and it is continuously improving as we adapt to the increasing paper load. For example, many conferences now have multiple rounds of reviewing to avoid wasting reviewer time on the worst submissions. As for the ocean of results, it is effectively tamed by Google. Suresh makes some similar points in his recent blog post where he equates the paper flood with spam email. It’s not a terrible analogy but it makes it easy to overlook some the true costs of the perverse inventive to maximize publications.
It isn’t the flood of bad research that’s the problem, it’s the secondary effects of this flood. First, promising young academics learn and propagate a misleading view of the research process, reducing the likelihood that the high-value, high-impact, once-a-decade results will be produced. To become and remain competitive, talented researchers waste time on short-term results and on spreading results across multiple papers in order to avoid exceeding the LPU by much. Bad research isn’t free: it is produced using grant money, which comes from taxes paid by hardworking citizens. Not only is it unethical to waste this resource, but waste has a high opportunity cost because it prevents useful work from being funded.
Fixing the problem is not so easy because the incentives are hard to change. The fixes offered by the Chronicle article are quite unconvincing when applied to my field. My read of the situation in CS is that little fixing is needed at the top-tier schools: a prolific publisher of worthless results is unlikely to be hired or tenured there. (This is one of the many reasons why the top schools tend to stay there.) On the other hand, the incidence of highly quantitative evaluation of faculty in the next few tiers — schools ranked 5-75, maybe — is significant and troubling.
One of the lessons from Michael Lewis’s fantastic book Moneyball is that if almost everyone is using broken metrics, there’s a tremendous opportunity to scoop up players who are undervalued by these metrics. This is exactly what forward-thinking, non-top-tier research departments should be doing at hiring time. The problem is that identifying these candidates and pushing them through internal hiring barriers is hard work. My guess is that departments who do this will win big in the long run; it is simply inconceivable that a focused practice of hiring paper machines will be a long-term advantage. You can fool the university, the tenure committee, and the awards committee at your favorite conference, but you cannot fool the larger ecosystem where significant research results are those that change how people think about the world, or spawn billion-dollar industries.
Extra credit question: What happens if the amount of grant money a professor earns becomes as important as her publishing record for purposes of tenure, raises, and internal prestige?
13 responses to “An Epidemic of Rat Farming”
“What happens if the amount of grant money a professor earns becomes as important as her publishing record for purposes of tenure, raises, and internal prestige?”
Of course this has already happened, especially at non-top-ranked schools. It’s pushed by administrators who LOVE their overhead. and yes, we know what happens then. I work in theoretical computer science 🙂 – ’nuff said.
“Extra credit question: What happens if the amount of grant money a professor earns becomes as important as her publishing record for purposes of tenure, raises, and internal prestige?”
In every one of my job interviews (at research universities), the major focus of conversation was grant money: where do I intend to apply for the grant money, what projects has been funded already in my field and how much money was it, etc. I was told to include my grant application plans as a major part of my job talk. Many fewer people asked about the research itself, independently of its role as a vehicle for grant money. The search committee member who was closest to my area and wanted me hired explained that I should really focus on how I will get money whenever anyone asked me about research, since they were basing their decision almost exclusively on who had the maximum expected future amount of grant money. Of course I understand this is how administrators at research universities think, and in fact if they didn’t spend time thinking about these issues, probably fewer schools would be able to support research programs (and the associated reduced teaching load and other perks that come with it). But it opened my eyes with how direct they were about grant money in a job interview for assistant professor, and how much they downplayed the scholarly aspect of research.
Suresh and Dave, thanks for the data points.
Dave: Would you mind characterizing the schools where you interviewed, for example in terms of top-tier, second-tier, etc.? In what year did you interview?
To add a bit more data: in Spring 2003 I had 10 interviews, all at schools in the 15-50 ranked range. As far as I can remember, money figured prominently in only one of those interviews. Maybe things have changed since then?
The discussion of money at the one school was truly memorable. The department chair sat me down and explained that exceeding $1M in grant money was a necessary and sufficient condition for tenure.
It is understandable that the administration worries about grant money. But it is very sad when this attitude percolates down into the professor ranks to the extent that money displaces quality of research as the first-order method for evaluating a (potential) colleague.
“What happens if the amount of grant money a professor earns becomes as important as her publishing record for purposes of tenure, raises, and internal prestige?”
Gosh, I hope it never comes to that. 😉
Extra credit question: What happens if the amount of grant money a professor earns becomes as important as her publishing record for purposes of tenure, raises, and internal prestige?
“If”? Whaddya mean “if”?!
“Would you mind characterizing the schools where you interviewed, for example in terms of top-tier, second-tier, etc.?”
Are you using an official designation for tiers? None was ranked in the last National Research Council ranking, so they are ranked somewhere below 100. All have a CS Ph.D. program and are trying to climb in rank. They must have decided this will happen as a side effect of pulling in lots of grant money, and they are probably correct. It just struck me as odd that they were so up front about it in a job interview. But I’m not complaining: my impression is that this directness was a favor, and there are probably lots of schools (even in the top 50) giving or denying tenure based mostly on amount of grant money without communicating this as a clear expectation. I’m not saying this is how they should make the decision, but however the decision is made, it should be explained transparently, rather than talking about 1/3 research and 1/3 teaching and 1/3 service and then firing someone who does a great job at all three but comes in a hundred thousand below the grant threshold for tenure.
“In what year did you interview?”
I interviewed this past year.
Thanks Dave. For “tiers” I sort of use USNews as an informal guide for assigning schools to brackets (not a great way to do it but at a coarse level it’s OK).
I agree with everything you said.
BTW my analysis (based on anecdotes from ~10 friends, friends-of-friends, etc.) is that the absolute worst place to be an assistant professor is the “teaching school with R1 aspirations.” You get a high teaching load and not a lot of support for doing research, but are still expected to pull in money and publish.
“the absolute worst place to be an assistant professor is the “teaching school with R1 aspirations.””
The places I interviewed had a 2+1 teaching load if you have a grant, 2+2 if you don’t, and ability to buy your way to 1+1 with 1/3 salary paid from a grant. The chairs seemed to know what it takes to run a successful department (even if they don’t have some of those things, like a huge endowment). But I can imagine how with a dumb chair it could be much worse, if they decide that you should be able to pull in money like the R1 schools while teaching 3+3 (or 2+2 with 100 students in each class) and with no Ph.D. students and no TAs. My friend was hired at a “metropolitan” (Master’s teaching) university where she taught courses whose syllabi read like an ad for a short-term web developer; she quit after a year. My advisor’s advice was: figure out if the people are happy who have similar aspirations to me (do research and teach smart students for a living), and no matter what the school is ranked, don’t go there if the people seem miserable.
But for anyone who wants a faculty job this year, being picky may not be an option. It’s ugly out there.
Dave: 2+1 or 2+2 doesn’t seem so bad. As you say, it comes down to having a reasonable chair/colleagues, transparent expectations, etc.
I interviewed (Spring 2010) at mainly top 25 liberal arts colleges where teaching/education is supposed to be focus. I had multiple offers and would certainly have had more had I expressed the appropriate amount of groveling/ohyouareabsolutelymytopchoice. I was asked about how I planned to bring in money at a liberal arts school, NFS, private, etc at every single one of these places. One even went so far as to tell me that if a NFS Career award wasn’t recieved the chances of tenure were slim to none.
The smallest teaching load was 2+2 and some wanted as much as 4+4. These weren’t grad student seminars either, but real courses of 20ish students with no TA’s. For those intersted (private liberal arts rarely publish these stats), the salary ranges offered were 40-65k and one outlier being at 80k.
I took an industry job instead.
Hi Dollar- Do you mind sharing the salary you actually started at in industry? (This isn’t germane to this thread but will help with a post I want to write in the future which looks at salary differences between academia and industry in CS. My existing data points are from mid-level to senior people.)
I took an offer of 110k in a non-bay area location with some potentially lucrative options (think buy a upper-middle class house, not retire) if the company does well in the next 2-3 years. on the downside there is no 401k match, but a 10ish% cash bonus seeming to be the expectation. Overall the offer was more than competitive with what bay area companies (intel/ms/hp/msr/amd/oracle) appear to be offering, with the major downside being risk (small companies can and do fold).
Having talked to many many grad students – my contemporaries have gone both industry and academics – and now being on the hiring committee for mostly undergrad folks here is a initial rough lay of the land assuming you are a “good to great” candidate at each perspective job / hiring level in CS at big tech. My experience is that startups don’t like to pay equal to this offering more in equity, but will if you have a competing offer…
Ugrad – 55-60k + 5-10%ish perks
Masters – 75-80k + 5-10%ish perks
PhD – 100-120k + 10-20%ish perks
PhD Academics R1 – 90k + 20-25% perks (kids college free can be another 10%*20 years) + summer salary potential
PhD Academics non-R1 – 45-75k + 25% perks (same college free bit) + summer salary unlikely
Overall I don’t think R1 vs Industry is nearly the gap people think it is… however non-R1 to industry is a substantial amount. The difference between being mediocre and good makes a big difference in industry at all levels. if you don’t have competing offers prepare to take at least 25% hit at any level. Your PhD isn’t worth much/anything if 2 good undergrads can get the same job done… I know of at least one PhD who took a job making 60k in the bay area because it was the best they could find – they also could’t write code to save their life nor describe how a computer “works”. Over-specialization will kill you if you miss the academic boat!
[…] An Epidemic of Rat Farming This entry was posted in Higher Education. Bookmark the permalink. ← Quote by Alfred North Whitehead Find my keys: the Promise of the Internet of Things → […]