UK higher-education research institutions were recently critiqued by the Research Excellence Framework (REF). The REF aggregates individual researcher performance by host institution to produce an overall score of institutional prowess. This allows qualitative comparison between research institutions as a product staff output.
Atop this process is £2 billion in research capital. The aim of the REF is to “distribute these funds selectively on the basis of quality”. That is: do well in the REF and you get more of the £2B. This meritocratic financial outcome means REF results are hugely important for research institutions.
Recently, the pressure to obtain a high REF score has led to accusations of systematic ‘gaming’. That is, finding loopholes in the REF process to obtain better scores. One well-publicised ‘gaming’ technique is ‘selective submission’.
The technique is very simple: as institutional REF scores are an aggregate of all submitted individuals, some institutions knowingly submit discerning individuals and fail to submit those who might drag the score down. This falsely biases the intuitional score up. For example, Cardiff University only submitted 62% of REF-eligible staff. They selectively picked their best staff to get the best institutional score. Fortunately, as we have a record of how many REF-eligible individuals each institution should be submitting, it’s possible to quantify the ‘selective submission’ gaming process and correct for it. This recently happened with a revised REF scoring. (Cardiff dropped from 7th to 50th.)
Correcting for REF gaming is crucial to obtain an accurate quantification of institutional prowess. It ensures funds are fairly distributed and allows future employees to empirically select the best places to work.
There is another form of REF gaming that receives less attention that I think we should also be correcting for....
I call it ‘output imitation’.
Here's the premise: REF submitted research does not have to be performed at the institution which it represents. For example, all 3 REF papers I submitted for my current institution (ICR) were performed in my Ph.D. lab (Cambridge). That means the ICR gets judged for work done in Cambridge.
If the REF aims to measure individuals – that is fine. But given the REF purports to measure institutional research quality, this opaque allowance is extremely deceiving.
Output imitation ensures REF scores do not represent the quality of work done at a host institution. Instead, REF scores represent a combination of work that was done at a host institution and work that was imported from elsewhere.
I believe the REF should resolve between these two research types.
Why?
Because it's much easier to import research papers from elsewhere – than to provide the resources, time, money and stability so that equivalent research can be performed in-house.
It's widely accepted that junior researchers are selectively recruited because they can transfer sexy papers from other labs. Meritocratic hiring is great – but only if the host institution is sincere in what they offer junior researchers.
The prevailing (anecdotal) zeitgeist implies successful junior researchers are recruited to bump the institutional REF score – and then left under-supported when it comes to producing new research. Why would an institution spend time and money supporting existing talent when it can easily import a new wave of successful juniors for the next REF?
Output imitation allows research institutions to overcome poor internal investment by importing external merit. It encourages insincere short-term hiring and discourages long-term internal investment.
I propose a single checkbox next to every future REF paper submission: 'Was this research performed at the host institution?'
Yes or No.
If the REF could institutionally resolved between those supporting internal researchers and those recruiting external prowess, host institutions will have an incentive to actually support the talent they recruit.
Disclaimer: The ICR came 1st in the REF. I've got no personal reason to complain about the current system. I just think it's deceiving.