As the REF draws closer, there are ongoing debates about how we assess scientists and their merits. Such assessments are critical for hiring and promotion and we are all in agreement that the current system, by which scientists are assessed by the impact factor of the journals they publish in, is not suitable, for reasons outlined by Stephen Curry and picked up again recently by Athene Donald.
Since I’ve been involved quite a bit recently in promotions and hiring, I’ve thought about this a lot. The problem is that we are trying to measure something intangible, and impact factor is the only number we have available – so naturally we use it, even though we all know it’s rubbish.
OK so here’s an alternative idea. It’s totally off the top of my head and thus doubtless deeply flawed but any ideas are better than none, right? And the impact factor discussion, at least in the last few months, has been remarkably devoid of new ideas.
The reason we don’t like impact factors is that we feel they are missing the essence of what makes a good scientist. We all know people who we think are excellent scientists but who, for whatever reason, don’t publish in high impact journals, or don’t publish frequently, or aren’t very highly cited, or whatever. But we trust our judgement because we believe (rightly or wrongly) that our complex minds can assimilate far more information, including factors like how careful is the person in their experiments, how novel are their ideas, how elegant are their experimental designs, how technically challenging their methods etc etc etc – none of which is reflected in the no. or impact factor of their publications.
My idea is that we crowdsource this wisdom and engage is a mass rating exercise where scientists simply judge each other based on their own subjective assessment of merit. Let’s say every two years, or every five years, every scientist in the land is sent the name of (say) 30 scientists in a related area, and are simply asked to rank them, in two lists: (1) How good a scientist do you think this person is, and (2) How important is the contribution you think they have made to the field to date. Each scientist is thus ranked 30 times by their peers, and they get a score equal to the sum or average of those 30 ranks. A scientist who made the top of all 30 lists would be truly outstanding, and one who was at the bottom of all 30 probably unemployable. Then institutions could choose to decide where to draw the boundaries for hiring, promotion etc.
This would all be done in confidence, of course. And a scientist’s own rank wouldn’t be released until they had submitted their ranking of the others. It would be relatively low cost in terms of time, and because specialists would be ranking people in a related area, they would be better placed to make judgements than, say, a hiring committee. In a way it is like turning every scientist into a mini REF panel.
Comments on this idea, or indeed better ideas, welcome.