Groupthink, bean-counting and the REF

When I was a medical student we were encouraged to conduct vaginal examinations on anaesthetized gynaecological patients, so that we could learn how to examine the reproductive system in a relaxed setting. The women did not know this was going to happen to them during their surgery, and did not sign any consent forms. Probably they would not have minded anyway (they were asleep after all, and educating the next generation of doctors is undoubtedly a good cause) but the possibility that they should be asked if they did mind did not occur to anybody, until my ultra-feminist friend kicked up a stink and organised a rebellion. The surgeons were genuinely surprised and mystified. Being well-meaning, it had not occurred to them that some people might see what they were doing as wrong.

Fast-forward a few years to the Alder Hey scandal, in which it emerged that doctors at a children’s hospital had been retaining the organs of dead children for post mortem examination, without the parents’ consent (or, in some cases, despite the parents’ withholding of consent). While the scale of the Alder Hey operations was beyond anything anywhere else, the practice of retaining postmortem issue for examination was widespread at the time and I have no doubt that the doctors involved were very surprised by the intensity of the public reaction. Being well-meaning, it had not occurred to them that some people might see what they were doing as wrong.

The fact is, professions are not very good at self-regulation, and when left to themselves they can develop bizarre practices that seem quite fine to them. This is because like all social groups, they fall foul of what Orwell termed groupthink – the tendency of groups of people to start to think more like each other than like society at large. Groupthink promotes into social cohesiveness, but if a group is not grounded to reality it can drift off into the wild blue yonder. As time passes we as a society have become more aware of this, and increasingly reluctant to allow professions to completely self-regulate. In just a few days’ time, for example, we will learn the outcome of the Leveson enquiry, in which press self-regulation has come under intense scrutiny due to the groupthink practice of phone-hacking that spread like a cancer through some sections of the press.

Which brings us of course to science. Science is very much a self-regulating profession, both in the way research is conducted and in the way the products of that research are taught to the next generation. We have been pretty much left to ourselves to get on with it, and have by and large been very successful, having invented the scientific method and peer review – both profoundly important contributions to knowledge-gathering. Recently, however, scientists have received a number of challenges to their internally-widely-accepted practices. The taxpayer has started to wonder  for example, why it has been considered OK for them to fund research but then not be able to access the results of that research, hence the “academic spring” and the open-access movement, which is rapidly (and unsettlingly, to some) transforming scientific publishing. The taxpayer has also started to wonder if the science they are paying for is any good. How would we know? “Don’t worry”, scientists have been saying “we have internal checks and balances and although these might seem opaque from the outside, you can trust us. We are scientists after all, and well-meaning”.

The taxpayer is not so easily fooled any more, however, and has demanded accountability. Hence the REF – the “Research Excellence Framework” – in which academics (all academics – not just scientists) have to come out of their lairs and present their best work to be judged. The REF is a gruelling administrative process that has been derided as a bean-counting exercise by some (e.g., Jenny Rohn in the Guardian) but the fact is, someone has to count the beans, otherwise we don’t know that beans are actually being produced.

I could write another whole post, an probably will sometime, about what a flawed process the REF is. We are trying to measure quality, which is something almost impossible to define and measure, but define and measure it we must. Hopefully as the years go by, our measures of academic output will become better, just as our climate models have slowly become better as we figure out what to measure. But I’m afraid the REF, or its descendants, will never go away. And nor should it. Being well-meaning, we academics need to acknowledge groupthink and recognise that we aren’t the best people to judge the quality of our own practices.

This entry was posted in Science and society. Bookmark the permalink.

2 Responses to Groupthink, bean-counting and the REF

  1. I see where you are coming from but I disagree. As someone involved in trying to get things ready for REF, I see the amount of time and money that is being wasted on this pointless exercise. If you reward people based on metrics, you get people who are good at playing games to deliver those metrics. A measure of quality as flawed as REF is worse than no measure at all as it rewards and punishes all the wrong things and wastes resources to do so.

    You are quite right about self-regulation but this – and your examples – is about best practice and acceptable behaviour, not quality. Academic integrity etc. should all be open to external regulation. Even what we research should have political and public input. Academic quality, on the other hand, can only be judged by academics. The REF panels themselves are made up of academics. All REF does is replace the current multitude of different metrics, surveys and measures of esteem that – whilst slightly vague and opaque at times – are currently used to judge academics in different scenarios with a single, flawed and equally vague and opaque one.

  2. To the left of centre says:

    Reblogged this on To the left of centre and commented:
    I wanted to reblog this partly because it’s good – in my view – to see more people writing about REF. I also think the post makes some interesting points. I think REF is horribly flawed, but maybe we have to also realise that self-regulation also has its problems. I would suggest that self-regulation might be the wrong term to use. Not all of our research money comes through REF. A big fraction comes through research grants that are competitive and are assessed by a reasonably knowledgeable panel (although I would suggest that the allocation of research council grants in the UK has its own problems). REF is essentially allocating QR money that goes – in the first instance – to the university, not to individuals. The university then decides how to divide this money up. It’s not clear to me that a simpler assessment exercise that used up less time, was less easy to game, and in which maybe the allocation was less non-linear (i.e., a weaker dependence on how well you do and how much money you get per person submitted) would be just as effective and less damaging. Part of me thinks that maybe the criteria should be secret until the process is finished, but I suspect many would feel that they then wouldn’t trust it at all and everyone would complain afterwards if they didn’t do as well as they thought they should have done.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s