Research Evaluation: Talking about your outputs

At the University of Glasgow we want to create an environment that produces research of the highest quality. Good-quality research happens within a community that supports, incentivises and rewards a positive research culture.  

The fair and inclusive assessment of research activities is central to this ambition. For this reason, in 2020 the University of Glasgow formally declared its commitment to the following international initiatives, to demonstrate our commitment to good practice in research assessment: 

These commitments sit alongside the University’s previous adoption, in 2017, of the Leiden Manifesto for Research Metrics.

We have developed the following resources to help researchers in thinking about how they talk about research outputs (e.g. in a job, promotion or grant application) and the sorts of questions to ask about publication strategy during mentoring or career development conversations.  

Below we provide: 

  • Conversation starters for both self-reflection and mentorship discussions relating to DORA-inspired publication strategies and tactics. This aims to ensure that research outcomes reach the desired audiences promptly and openly, thereby maximising both the impact of the research and recognition of the researchers. 
  • The policies and initiatives that we have introduced since 2015 to assess the range of contributions to research in a fair and transparent manner, and that are sensitive to disciplinary context. 

The principles of DORA/Leiden/Hong Kong Manifesto support the University’s own principles for research, as outlined in the 2020-25 Research Strategyi.e. that: 

  1. We value the quality of our research over its quantity.
  2. The University succeeds when our individual researchers succeed.
  3. How research is done is as important as what is done.

The University’s progress in this area is overseen and reviewed annually by Research Planning & Strategy Committee (RPSC) and the Research Culture and Careers Group. 

Conversation starters about individual research and publication strategy

Questions researchers might want to ask or explore:

The Declaration on Research Assessment (DORA) has a fundamental aim to avoid the use of the journal impact factor as a proxy for article quality. However, discussion around the quality of our published outputs and publication strategies remain important, and we encourage supervisors, line managers, mentors, and indeed peers, to support others to reflect on the quality of their work and on how to enable the work to reach the widest relevant audience.

Below, we list possible questions to aid conversations centred around a researcher’s own situation and career needs. The guidance relates not only to the outputs themselves but the research and plans for research that underpin these outputs. 

  1. What are your plans for developing research outputs (e.g. publications, software, performances) over the next 12 months?
  2. What strategy will give your work the highest international profile: a) with your immediate peers? and b) within the wider discipline?
  3. What is your approach to sharing your research (before, during, and after formal publication) and getting critical input to it? How has advice from critical friends been incorporated into your research and resulting outputs?
  4. How would you succinctly describe your strongest research outputs, in terms of the originality of the research, its significance to the field, your contribution to the work?
  5. What is your approach to open research, e.g. the  sharing of data, code or software, and how might you enhance it?
  6. In terms of your international peers, what has been the most significant output(s) published in the last 12 months, and why? What can you learn about the research direction or how it was shared or presented?
  7. In the next couple of years, what is the right balance between your outputs targeted at a specialist audience and those aimed at a wider readership?
  8. (where appropriate) Is this the right time in your career to consider writing a monograph or review paper; if so, what would be the best route to publication?
  9. In considering your recent publications, who is citing or commenting on your work? What are they saying about it? Of your influential or highly cited outputs, do you know why they have attracted attention?
  10. (where appropriate) How are you supporting colleagues at an earlier career stage to develop an outputs strategy that is complementary to yours?

Strategic context

The support of the University of Glasgow for DORA, Hong Kong Principles and the Leiden Manifesto is consistent with and embedded within our institutional strategies, policies, and action plans.

Strategy 

University Research Strategy 2020–2025 

  1. We value the quality of our research over its quantity 
  2. The University succeeds when our individual researchers succeed 
  3. How research is done is as important as what is done 

Action plans 

Externally the University of Glasgow is a member of the UK Reproducibility Network, the European Guild of Research Intensive Universities and has established a Lab for Academic Culture to enhance academic research and teaching culture both by implementing local initiatives and shaping sector policies. 


Our policies and practices

Our policies and practices include:

 Policy/PracticeLinksWhy
2020 Updated our institutional KPIs for research to include transparent and normalised bibliographic metrics, including open access, open data, author contribution statements, and uptake of ORCID. Research KPIs

ORCID
Institutional success requires targets and goals. These apply to large groupings, not individuals, and are aligned to the environment we wish to encourage.
2012 Introduced academic promotion criteria that follow a preponderance approach. Promotion is determined through meeting transparent criteria in at least 4 of 7 performance dimensions: outputs, income, supervision, teaching, engagement, and esteem. Promotion criteria are detailed and openly available. Targets are benchmarked and subject-normalised. Academic Promotion Criteria Recognition that staff contribute to the success of a team/organisation in a variety of equally important ways. No individual is required to meet the criteria for all seven dimensions, but as a team we do.
2012 Promotions consider the previous six- year period when assessing the contributions made across the 7 performance dimensions. Academic Promotion Policies Research success requires planning and execution over many years and cannot be judged on a single review cycle.
2019 Revised professorial promotion criteria to require evidence of collegiality (i.e., support for the careers of others). Making Collegiality a Core Element of Research Culture at UofG Government, funders, and the sector more widely are rightly insisting on an improved research culture.
2019 Updated professorial promotions to bring parity of reward for impact and outputs, a commitment to open research and minimum standard applied to learning and teaching. MyGlasgow News: Academic Promotion Criteria Funders and government require academia to make a more rounded contribution to society.
2017 Launched policy on “responsible use of metrics”, committing to fair and transparent mechanisms for evaluating and reporting on research performance. Responsible Metrics Quantitative indicators have a vital function in unbiased assessment of research, but must be normalised (by subject and career stage), contexualised, and used in tandem with peer review.
2016 Open research (including open access, wide and early sharing of research). Library supports open data, data availability statements, open research; all tracked as part of Research Culture Action Plan. How do I make my Publication Open Access

Open Research
Research is accelerated and integrity is maintained by early, wide and complete reporting.
2016 Introduced new recruitment practices within our Fellowship programme, e.g. requiring short narrative on small number of best outputs, and explicitly referring to different output types, and requiring a commitment to open research. LKAS Fellowships Output quality cannot be judged on metrics alone and requires context to assess significance, contribution, and contribution to strategy.
2019 Ensure that impact factor, or wording that could be perceived as referring to journal-based metrics, is not used as an indicator in research assessment processes. Statement on Use of Quantitative Indicators in Assessment of Research Quality The Journal Impact Factor was introduced to compare journals and has no role in the assessment of individual researchers or outputs.

Development and support

In implementing the policies above we seek to both Develop and Support our staff.

 Development and SupportLinksWhy
2015-2018 Panel-based external reviews of research (2015, 2018) were aimed at identifying and sharing what makes a good quality article in each discipline. Interim Research Reviews (VPN required) 

Interim Impact Reviews (VPN required)
Establishing what research excellence means and how it might be recognised at the discipline level.
2015 Established a network of research integrity advisors, to record informal queries or concerns received from staff and students on good research practice. Research Integrity Advisers 

Research Integrity Areas of Conduct
Research integrity is not simply about compliance, but about supporting and advising on best practice.
2019 Research culture awards to recognises the contribution of all grades and job families of our staff. Research Culture Awards Supporting and celebrating the work that our colleagues undertake, often previously unrecognised, in support of the careers of others
2017 Embedded the CRediT taxonomy into our Code of Good Research Practice to encourage researchers to put their author contribution on the public record; we have also enabled researchers to record their author contribution(s) in our institutional outputs repository. CRediT taxonomy

How do I make my Publications Open Access

Before Submitting Your Manuscript
Research is often a team activity and traditional author listings fail to capture individual roles and contributions that are central to researchers’ assessment and their careers.
2019

Advice to individuals on managing publication profiles:

  • Information on bibliometrics indicators
  • Tracking tools for article-level analysis
  • Sources of citation data used at UofG.
Managing Your Publications The communication of our research to the right audiences is as important as the research itself, both in support of the wider community and the progression of individual careers.
2021 Conversation starters in support of the careers of others and publication strategies. Conversation Starters Career development depends upon local peer group support. This support requires formal and informal conversations, the initiation of which is critical.
2020 Developed resources / campaign for PI career conversations (summer 2020) with follow-on survey of researchers in October 2020; articulated what ’10 days of CPD looks like’ for researchers. PI career conversation with researchers Career advice depends upon local support and advice. Dialogue, not form filling, is central to the support we seek to encourage.
2021 Produced a toolkit for hiring managers to ensure we recruit researchers and academic staff who will embody the values of our new UofG research strategy — to support creativity, collaboration and career development — and ensure a positive research culture. SCOPE

UofG SCOPE case study
This resource emerged from a SCOPE workshop we hosted to identify what is important in supporting the careers of others and how we can evaluate this (e.g. in grants, recruitment, appraisals or future REF exercises.

Additional resources