Hard lessons of assessment

The accusation from disgruntled dons is serious. An assessment system devised to lift the quality of research in UK universities and post-graduate schools has distorted research output, created a cut-throat hire-and-fire labour market among academics and imposed intolerable pressures on institutions that should be concentrating their efforts on producing excellence rather than demonstrating it to government inspections.

But is this fair? In the case of business schools, for example, many prominent academics agree with the funding bodies: this accountability system, since its introduction in the 1980s, has, along with increased international competition and a global market for talent, helped produce British institutions that can hold their own against the best.

“We would probably have had some of the effect simply from the competitive international market,” says Dame Sandra Dawson, former director of the Judge Business School at the University of Cambridge. She argues there has been a significant increase in the rigour and reputation of UK-produced research in business and management as a result of pushing it into the peer-review academic journals surveyed under the assessment system.

“But measurement and evaluation systems do influence behaviour. That’s why we have them.”

The latest Research Assessment Exercise, RAE 2008, will be the last of six such processes, to be replaced with a system of light-touch annual measurement. The new metrics have still to be decided but are likely to be based on citations, completed post-graduate qualifications and research income, possibly with some version of a peer review for some disciplines.

By 2013-14, assessment (and the public funding on which it depends) will be based on the new system. In anticipation of the changes, some academics are already predicting a move towards more book publication – since books tend to receive more citations over time.

Perceptions

But, as Eric Thomas, vice-chancellor of Bristol University, says, RAE 2008 will set the baseline in terms of perceptions about an institution or its departments.

In the next few months, panels across 15 broad subject areas will read submitted research and draw up quality profiles of departments. Their work will this time be graded on a scale from unclassified to 4*, to signify work of world-leading significance. There will be no overall grade for a department and its staff.

Cary Cooper led the panels evaluating business and management research in 1996 and 2001. Now David Otley, his colleague at Lancaster University Management School, has taken the chair, leaving Prof Cooper free to air his views on the pros and cons of the process.

Prof Cooper believes the stringent demands of US publications have led to improved methodologies among British researchers and a flowering of more and better UK-based journals. But he says that, as these journals have become more specialised, then more accessible, inter-disciplinary research has been squeezed out. The RAE, he adds, has also contributed to a bias against longitudinal studies, which cannot be assessed until completed.

Innovative

But, overall, he thinks the effects are positive. “In the 12 disciplines that make up the management sciences, the RAE stimulated a more research-intensive environment and this was real, empirically based research on real-life problems.”

The Association of MBAs says schools have been recruiting fewer faculty from industry and commerce in favour of academics. But Prof Cooper claims schools are good at keeping their research relevant and innovative, whatever the pressures of the RAE, because of the nature of their MBA-focused activities and their attraction towards practitioners as an audience.

As for the “premier league problem” – a transfer market comparable with the world of football, whereby staff are poached on high salaries to try to ensure their research output shows up in the new institution’s submission – Prof Cooper blames the practice for preventing several schools with excellent MBA programmes from developing their research side.

“We [academics] just learnt how to play the game. It was not very nice what went on and 12 or 15 of our business schools could now be world-class but a lot of their people got poached.”