Stage 3 - Methods and evidence

Stage 3 - Methods and evidence


By now you should have carried out the necessary preparation in relation to current literature in the field, and considered resource implications, etc. Now you need to consider what would be the most appropriate methods for your project.  As you can imagine there are numerous methods available and which one you use will depend very much on the type of project undertaken. 

Glassick, Huber & Maeroff (1997, page 3) state that scholars must use appropriate methods, a yardstick that can and should be used in all aspects of academic work. 

In particular, in evaluating methods used in scholarship, Glassick, Huber & Maeroff (1997, page 6) propose the following questions: 

  • Does the scholar use methods appropriate to the goals?
  • Does the scholar apply effectively the methods used?
  • Does the scholar modify procedures in response to changing circumstances?

The method chosen for your project should fit the 'Clear question or statement of intention' being investigated.

A suitable starting point would be to look at you project overall and identify what type it is.

  • Is it a) evaluating current practice or is it b) implementing a change to current practice and/or developing new teaching sessions or resources that you wish to investigate?
    • If it is an evaluation of current practice (a), will this involve gathering feedback from students and/or staff?
      • If so what methods could you adopt to gather this feedback?  E.g. questionnaire, interview, focus group, etc.
      • What type of data would actually be most useful to you? See below for ideas.
    • If you are implementing a change to current practice and/or developing new teaching sessions or resources (b), how will you go about this?
      • Will you be comparing new to old with students doing either/or?  If so, how would the students be split?  Are there any ethical issues associated with this, e.g. is the new intervention expected to be better than the old, in which case would you be disadvantaging some students in their learning?  Might there be any differences between groups of students that could confound the results, e.g. if students are able to choose whether they are exposed to the new intervention or not, might a certain style of teaching suit a particular group of students?
      • Are you going to gather information prior to the intervention and then again after the intervention? If so, would you collect data from the same set of students before and after exposure to the intervention, or from different cohorts of students exposed to different learning conditions?  If you are using two groups, are there any potentially confounding variables (e.g. differences between cohorts) that would need to be taken into account, and could these be controlled for or minimised?  If the same group is participating twice, might there be any confounding variables (e.g. if the students are filling out the same questionnaires before and after, might they have an idea of what kind of difference you are looking for)?
      • What type of method is most appropriate for this?
      • What type of data would best answer your question? See below for ideas.

In choosing an appropriate method, one of the big decisions is whether to opt for a quantitative method or qualitative procedures or a combination of both - giving you the opportunity to cross-validate results on the same research question by using multiple methods (Webb, Campbell, Schwartz & Sechrest, 1966).  Both quantitative methods and qualitative procedure can be useful for investigating teaching/learning questions.

Quantitative methods

Test objective theories by examining the relationship among variables.  These variables in turn can be measured, typically on instruments, so that numbered data can be analysed using statistical procedures. 

Qualitative procedures

Explore and make attempt to understand the meaning individuals or groups ascribe to a social or human problem.  Qualitative procedures commonly used include observations, in-depth interviews and focus groups.  Hypotheses are generated during data collection and analysis, the data tend to be based on discourse, and measurement tends to be subjective. 

Mixed Methods

Simply employ aspects of both quantitative methods and qualitative procedures.   (Extracts from Creswell, 2008)

The advantage of using qualitative procedures is that they generate rich, detailed data that leave the participants' perspectives intact and provide a context for behaviour.  The focus upon processes and "reasons why" differs from that of quantitative methods, which addresses correlations between variables, or cause-and-effect relationships.  For example, say you were looking to find out the efficacy of a new intervention.  You could ask the students to complete a questionnaire in which they rate elements of the intervention by means of a numerical scale (e.g. Lickert), this data can then be analysed – quantitative.  Alternatively you could ask the student to take part in a focus group which is audio recorded, transcribed and themes extracted – qualitative.  Doing both (mixed method) would give greater strength to the study.

What type of data count towards evidence?

  • The views of your students, either quantitative, qualitative, or both would be the most obvious place to start.
  • Some ideas for the type of data to collect - student work, assessment scores, opinions on how much information was learned in relation to learning objectives, what skills were developed, how confidence (self efficacy) levels were affected, what the retention levels were, what attendance was like, the extent to which students participate in activities, the extent to which students make use of feedback, how well were staff able to implement changes, how much resistance/uncertainty there was (both from staff and students) etc.?
  • Your reflections before, during and after the project would be a useful addition, and perhaps also your peers reflections, if they are involved with grading assessments etc.

A Question of Ethics
Do you require ethical approval to carry out your project?  You will normally require ethical approval whenever you are investigating human behaviour, and you wish to make the results of your investigation public.  Every institution/faculty has specific guidelines detailing ethical approval processes/procedures and this should be your starting point.   PLEASE do not collect any human data prior to ensuring the ethical stance of your faculty.  It is highly unlikely that ethical approval will be granted retrospectively.
Any act of scholarship ultimately must be judged by the significance of its results Glassick, Huber & Maeroff (1997, page 6).

When evaluating the results of a project, they propose the following questions:

  • Does the scholar achieve the goals?
  • Does the scholar’s work add consequentially to the field?
  • Does the scholar’s work open additional areas for further exploration?

What type of data count towards evidence?

  • The views of your students either quantitative, qualitative, or both would be the most obvious place to start.
  • Student work, assessment scores, etc.
  • Your reflections before, during and after the project would be a useful addition, and perhaps also your peers reflections, if they are involved with grading assessments etc.

Please note that significant results does not necessarily mean ‘statistically significant’, but significant in relation to the field of work.  What this means in effect is that the results of your project may turn out to be non-significant but the overall message remains important and worthy of dissemination and therefore still significant.  At the planning stage, the work should be designed to build upon previous work and thus contribute to the area. 

If you have registered for the online workshops then this will be the topic covered in workshop number 3 'Methods and evidence’.  This workshop will provide the opportunity to discuss various methods of investigation while offering the opportunity to give and receive peer-feedback.

Useful Resources
Quantitative and Qualitative Methods

Kairuz T., Crump K. and O’Brien A. have produced a two-paper series on qualitative research.  The first part is: An overview of qualitative research.  This paper provides an overview of qualitative practice research in pharmacy and the second part is: Tools and data collection and analysis.  In this paper the authors discuss two methods of data collection – focus groups and in-depth interviews commonly used in qualitative research.  Both papers can be accessed here.Morgan D. L., 1998.  Practical Strategies for Combining Qualitative and Quantitative Methods:  Applications to Health Research.  This paper describes a series of research designs for combining qualitative and quantitative methods.Thomas D. R., 2003.  A general inductive approach for qualitative data analysis.  This paper explains a simple approach to qualitative data analysis and is available here. Jick T. D., 1979.  Mixing Qualitative and Quantitative Methods: Triangulation in Action.  In this paper Todd provides an explanation of triangulation and provides an illustration of how it works. Ethics

Burman M. E. & Kleinsasser A., 2004.  Ethical guidelines for the use of student work: moving from teaching’s invisibility to inquiry’s visibility in the scholarship of teaching and learning.  This paper details nine principles to guide the use of student work in classroom inquiry.Chang R., Gray K., Polus B., Radloff A., 2005.  Scholarly teaching practice: Ethics issues and responses in research into teaching in tertiary education.  This paper is an excellent resource covering ethical concerns.  Available here.Case Studies

Available here.

Cresswell J. W. 2008.  Research Design: qualitative, quantitative, and mixed methods approaches.  3rd Edition Sage, London.

Glassick, C.E., Huber, M. T., and Maeroff, G. I. (1997). Scholarship assessed: A special report on faculty evaluation.  Presentation to Fifth AAHE Conference on Faculty Roles and Rewards, San Diego, California, January 18, 1997.  Available here.

Webb E. J., Campbell D. T., Schwartz R. D., & Sechrest L.1966.  Unobtrusive measures: Nonreactive research in the social sciences.  Chicago: Rand McNally.