EU's Agreement on Reforming Research Assessment
This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators. Among other purposes, this is fundamental for: deciding which researchers to recruit, promote or reward, selecting which research proposals to fund, and identifying which research units and organisations to support.
The Agreement on Reforming Research Assessment was published on the 27th of July, 2022.
Aalto University has signed the agreement and is therefore a member of the Coalition for Advancing Research Assessment, CoARA. Below is a short review of the Agreement but we recommend to read the whole Agreement. Please read more about Aalto University's research assessments in this page.
In the context of the Agreement, research assessment encompasses:
• The assessment of research performing organisations and research units, by assessment authorities, research funding and performing organisations
• The assessment of research projects by assessment authorities, research funding and performing organisations, and prize awarding organisations.
• The assessment of individual researchers and research teams by research funding and performing organisations and prize awarding organisations.
The Agreement focuses on the specific challenges involved in reforming the research assessment of researchers, research projects, research units and research organisations.
The signatories of the agreement agree:
I. Base our actions on the following Principles
Principles for overarching conditions
• Comply with ethics and integrity rules and practices, and ensure that ethics and integrity are the highest priority, never compromised by any counter-incentives
• Safeguard freedom of scientific research.
• Respect the autonomy of research organisations.
• Ensure independence and transparency of the data, infrastructure and criteria necessary for research assessment and for determining research impacts.
Principles for assessment criteria and processes
• Focus research assessment criteria on quality. Openness of research, and results that are verifiable and reproducible where applicable, strongly contribute to quality. Openness corresponds to early knowledge and data sharing, as well as open collaboration including societal engagement where appropriate. Assessment should rely on qualitative judgement for which peer review is central, supported by responsibly used quantitative indicators where appropriate.
• Recognise the contributions that advance knowledge and the (potential) impact of research results.
• Recognise the diversity of research activities and practices, with a diversity of outputs, and reward early sharing and open collaboration.
• Use assessment criteria and processes that respect the variety of scientific disciplines, research types, as well as research career stages, and that acknowledge multi-, inter-, and trans-disciplinary as well as inter-sectoral approaches, when applicable.
• Acknowledge and valorise the diversity in research roles and careers, including roles outside academia.
• Ensure gender equality, equal opportunities and inclusiveness.
II. Implement the following Commitments
The are two main types of commitments; core commitments (1 – 4) and supporting commitments (5 – 10).
The core commitments include commitments to enable better recognition of the diverse practices and activities that maximise the quality of research, and commitments to enable a move away from inappropriate uses of metrics. The supporting commitments include commitments to enable the move towards new research assessment criteria, tools and processes, and commitments to facilitate mutual learning, communicate progress and ensure that new approaches are evidence informed.
1. Recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research
Changes in assessment practices should enable recognition of the broad diversity of: valuable contributions to science and society; practices that contribute to robust, open, transparent and inclusive research; diverse activities (e.g. teaching, leadership, supervision etc.) and diverse roles (e.g. data stewards, software engineers etc.).
2. Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators
Research assessment should rely primarily on qualitative assessment for which peer review is central, supported by responsibly used quantitative indicators where appropriate.
3. Abandon inappropriate uses in research assessment of journal- and publication based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index
Inappropriate uses of journal- and publication-based metrics in research assessment should be abandoned. This means moving away from using metrics like the Journal Impact Factor (JIF), Article Influence Score (AIS) and h-index as proxies for quality and impact.
4. Avoid the use of rankings of research organisations in research assessment
Recognising that the international rankings most often referred to by research organisations are currently not ‘fair and responsible’ , the criteria these rankings use should not trickle down to the evaluation of individual researchers, research teams and research units. Research organisations should regain the autonomy to shape assessment practices, rather than having to abide by criteria and methodologies set by external commercial companies.
5. Commit resources to reforming research assessment as is needed to achieve the organisational changes.
6. Review and develop research assessment criteria, tools and processes.
7. Raise awareness of research assessment reform and provide transparent communication, guidance, and training on assessment criteria and processes as well as their use.
8. Exchange practices and experiences to enable mutual learning within and beyond the Coalition.
9. Communicate progress made on adherence to the Principles and implementation of the Commitments.
10. Evaluate practices, criteria and tools based on solid evidence and the state-of-the-art in research on research, and make data openly available for evidence gathering and research.