College of Medical, Veterinary & Life Sciences

Evidencing and evaluating impact is the process used to determine whether or not your research has led to change or benefit. At one level it is the process you undertake to identify, capture and document evidence across all stages of your research impact journey. This process can involve recording evidence of your own key activities and engagements that contributed to the change or benefit. It may include the collection of documentary evidence published by others (e.g., clinical guidelines, drug approvals, policy documents that reference your work), or requesting testimonials from stakeholders which describe the research contribution.

However, evidencing may also involve more formal evaluation or assessment of your outcomes through additional research, evaluation or audit process to help understand the impacts, substantiate the causal links, and quantitate the precise 'significance' and 'reach' of the impacts. Briefly, 'significance' describes the magnitude or intensity of the effect of impacts arising from your research; whereas 'reach' can describe the numbers, extent or scale of the impact. Reach might be achieved through social or geographical spread, penetration into a particular community of interest, or scaling beyond the original stakeholder groups (e.g., from grassroots to policymakers or vice versa). Knowing more about the outcomes and impact of your research can add value to future research development—useful collateral for funding applications, and to enhance the impactfulness of your research. For example, knowing the health economic benefits of an intervention you've trialled can improve its uptake. Evaluations may reveal ways to optimise core components of your research, or make it more generally applicable.

The main question we often faced with is: Did the research make a material and distinct contribution to the impact, without which the impact would not have occurred (or would be much diminished)? While also recognising that impact may result from a body of research, rather than a single study, to which other institutions may have similarly contributed.

Knowing what you need to evidence comes partly from your planning for impact exercise. Knowing the change or benefit you hope to see, and who might deliver or benefit from it, will clarify what you need your evidence to show. From there, you can try to identify the types of evidence and indicators that corroborate it. The guide below provides an non-exhaustive range of examples. 

TIP: Impact doesn't have to be global. Remember that impact isn't always global and wide-ranging. Impact can be achieved even at comparatively small scales, if that is the aim and intent. For example, many policies or interventions are only relevant for specific populations, at specific times and in specific places. Understanding this should factor into your plan for impact—consider your end goal, but plan for the first steps.

There are many different evidence sources, a useful guide is: ‘Types of impact and examples of evidence required to corroborate the impact’. Another useful resource includes 'Guidance on the types of evidence that could be collected', which lists different types of evidence types and their respective advantages and disadvantages.

How do we use evidence?

There are two principle uses of evidence to substantiate impact: (1) providing a LINK between a piece of UofG research and the impact/research users; and (2) quantifying the significance of the impact. The former is required to establish the relationship between your research and what you might claim as impact. It is essential that you collect linking evidence as early as possible. You may have a direct relationship through a contract or something less formal. Alternatively, more indirect evidence might be secured where research users cite a published piece of work; however, clarifying in-person how they have used it is always beneficial. Evidence on the scale and extent of an impact is also important to consider from the outset of your research. Impacts tend to happen beyond the academic sphere, so sometimes evidence of scale and extent may be created by third-party stakeholders.

• Documentary evidence—materials describing exactly what research was considered and how it led to change—is extremely useful, but often rare. Early stage documentary evidence might include published synopses from workshops held by stakeholders. More mature examples include policy documents, clinical practice guidelines or drug approval documentation from health regulatory bodies that cite your work. However, not all citations are equally valid. Consider the context of the citation: is it just background research or rationale that sets the scene? This is a harder case to make unless your research clearly identified the gap that mandated the policy or guideline recommendation. The strongest evidence is where a policy proposal, clinical or practice recommendation, or drug approval review specifically cites your research study (either by trial name or by research paper) to support that recommendation.
  
  
• Testimony evidence—in the absence of published documentation, you can seek testimony from a person or organisation. This should describe what research was received and how it has been used. Testimony evidence is particularly useful at the outset of a research engagement, to help establish the nature of the engagement, the research communicated, and how they plan to use it. This can always be followed up at a later date, to identify some of the outcomes, but the initial record provides the LINK. Similarly, for more mature potential impacts (e.g., if an impact on a clinical guideline isn’t explicit), you may need to ask the guideline chair or another clinical body to identify how your research has contributed—sometimes this is very clear to professionals within a field, but not apparent to those outside that field.

London School of Economics blog post: 'Guidance on testimonials and statements to corroborate impact'.

NOTE about 'activities': Many researchers misconstrue activities (the things they do) with evidence of impact (substantiating the changes that happen as a result). Evidence of activity is not the same as evidence of impact. Recording your activities, be it speaking at a policy event/workshop or academic conference attended by industry stakeholders, has useful narrative value only if you have a plan for how you will follow up on those activities, or know the change that resulted from them. Who was it you were hoping to reach? Did you speak to them or share a document with them? Do you have their contact details? Do they have yours? What did they think of your research? How might they use it? Without such evaluation plans or opportunity to identify potential outcomes, a list of activities is good for esteem, but it is not evidence of impact. 

Altmetric Institutional Explorer: It is useful to know which stakeholders might be using your research, so that you can search for (or ask them for) documentary evidence they have published that cites your work. However, you may not always know who is using your research. Altmetric Explorer is a useful tool to investigate this: UofG staff have access—just follow the login link. Most of what Altmetrics provide is 'buzz' in the form of media interest, which can be interesting to note, but is not impact in and of itself (the impact would be if you can link the media piece to a change or benefit as a result of someone reading it—no easy task). However, Altmetrics also includes policy documents, such as some national and international guidelines, policy documents and other grey literature. The Research Impact Officers also have some limited access to Overton.io, a newer platform that indexes policy and business documents.

Audit and evaluation evidence

Audit and evaluation evidence is particularly important in interventional research, where you are developing products, services or other interventions that require more evidence than just functionality or efficacy. Building evaluation into the design of your project, ideally involving a relevant non-academic research partner, will add considerable value to the outcomes of the research. It can also identify gaps to be addressed with further research, both opportunities for additional research funding and the scope for increased impact.

The Research Impact Support Team can prospectively support research programmes in identifying opportunities to evaluate their impacts, especially where such evidence would enhance the potential impact. Examples might include:

• health economics analysis of your intervention;
• surveys of professional practitioners to identify uptake and use of recommendations; and
• analysis of prescription rate data studies to show change in medical treatment.

We hope to be able to connect you to College facilities that may be able to support this, or researchers who have practice in these types of investigations. We anticipate that some of these audit and evaluation activities may be funded through internal funds, such as the Glasgow Knowledge Exchange Fund. However, each opportunity is reviewed on a case-by-case basis.

The MVLS Research Impact Support team can help you to review your plans for evidencing impact. In addition, we can offer our own expertise on where to find sources of information, and what such investigations entail. For impact opportunities that we have identified, we can help you to secure testimonies from key stakeholders if you feel that we, as a third party College service, are better placed to make such a request. For more substantial investigations, including evaluation and audits, we are happy to participate in these discussions, and to either bring together or signpost to College professionals able to help with design of such activities.

Where to store your evidence? All researchers who are engaged with external stakeholders should make efforts to store securely any evidence to substantiate the enaggement and any outcomes from it. The Research Impact Team are happy to advise, and will also help store materials related to those impact opportunities that have been added to the MVLS Impact Opportunity Pipeline. However, UofG also hosts a Knowledge Exchange and Impact Repository within the Englighten platform. This allows you to record and document non-academic collaborations, consultations, public or community engagement, knowledge exchange or impact-generating activity and any other external engagement related to or stemming from your research.