Statistics Tutorial

Experimental Design

Tutorial Navigation

Getting StartedGeneral Instructions | Introduction to Your Study | Experimental Design | Stating a Hypothesis
Descriptive StatisticsHistograms | Central Tendency | Standard Deviation | Confidence Intervals
Comparing Two SamplesSamples and Populations | Choosing a T-Test | Paired T-Test | P-Values and T-Tables
Important ConceptsThe Normal Distribution | Z Scores | Probability Distributions
LevelsYou are currently on Experimental Design at level 3. Level 1 | Level 2 | Level 3
Next Topic Introduction to Your Study | Stating a Hypothesis

Explanation

Why Specify Experimental Design?
At level one of this topic, we introduced the two types of experimental design and at level two we explained the advantages of each. At this stage, we will explain why it matters which type of experimental design you choose.

The key thing to remember is that the sample values collected in one sample of a paired design can paired with corresponding values in the other condition's sample. This fact allows us to be much more specific when we make comparisons between the two conditions. In an independent experimental design, no pairings can be made so the best we can do is compare the average (the mean) of each group.

Sampling Error and Variance
You can understand the improved accuracy that a paired design provides by thinking about sampling error. Sampling error is likely to be far smaller when the same subject is measured twice.

Here is an example. Imagine you measured your height on a monday morning, spent a week stretching, and measured your height again. Even the smallest change in height would suggest that the stretches had made you grow. Now imagine you spent a week stretching, and then measured your height and the height of another person who had not spent the week stretching. You would expect there to be a difference in your heights because of natural variation.

It is obviously important to take into account these differences in variation when comparing samples. You will see how this is done when we move on to how to choose a t-test.

Confounding Variables
At level two, we introduced the concept of confounding variables. These are variables other than your independent variable, which might have an effect on your dependent variable. Confounding variables increase the variance of the measurements of the dependent variable.

Imagine an example where you want to test whether men or women are taller. The independent variable is gender and the dependent variable is height. Anything else that effects height is a confounding variable in this study. An obvious example would be age. The more different age groups you measure, the more variation you will see in heights. Nationality would be another. If you are lucky, confounding variables will only increase variance but if you are unlucky, they could introduce bias. In our current example, bias might be introduced if the men you measured were all jockeys (and so quite short) and the women were all basketball players. In this case, the confounding variable (sport) would eclipse the dependent variable (gender).

It is important to consider what confounding variables there might be when you carry out an experiment so that you can avoid their ill effects. Here are the most common ways to reduce the effects of confounding variables once you have identified them:

  • Within subjects designs have a naturally lower risk of confounding variables introducing bias as the same subject is tested in each case. Of course, anything could happen to the subject between measurements, which could influence the second measurement, so such experiments are not immune to confounding variables;
  • Counter Balancing can be used to reduce fatigue or practice effects for paired designs. Counter balancing controls the order in which subjects are placed in each of the two conditions. Half are measured first under condition 1 and then under condition 2, while the other half are placed under condition 2 for their first measurement and then measured in condition 1 afterwards. This is not always possible, particularly in 'before and after' type experiments that test to see whether or not an intervention (a medical treatment, or a diet, for example) have an effect.
  • Random Sampling reduces the risk of counfounding variables introducing bias. If samples are picked completely at random, then any confounding variables that are present should be spread evenly across both groups.
  • Controlling confounding variables can reduce their effect greatly. In our example on height above, you could control for age by only measuring 25 year olds. You could control for nationality by only measuring people from your own country (or race). Another way of controlling for confounding variables is to make sure that each group contains the same spread of values of each confounding variable (the same number of people of each age, for example). This obviously gets difficult if there are more than two confounding variables.

Exploration

Consider this web based tutorial. We hope that it makes understanding statistics easier for students, but we would like to carry out an experiment to find out properly. Imagine we have a group of students and a test for them to perform that tells us how well they understand statistics. Our dependent variable would be score on the test. The independent variable would be whether or not the student had used the tutorial.

We have several options for the experimental design. Imagine firstly, that we tested each student, then gave them the tutorial, and then tested them again to see if they had improved.
What type of experimental design would this be?   Help
Which of these is a confounding variable:
  • The fact that they see the test twice and so gain from the practice;
  • The fact that we do not know how good the tutorial is.
  Help
Counter balancing can be used to control for practice effects.
Would it be possible to balance the study so that some students were tested
in the pre-tutorial condition first and some in the post-tutorial condition first?
  Help
Alternatively, we could choose two groups of students but only give the tutorial to one group. We would then give the test to both groups and see which did best.
What type of experimental design would this be?   Help
By denying some students access to the tutorial, what kind of problem do we introduce?   Help
Could we solve the ethical problem by letting the students choose whether or not they took the tutorial?   Help

Application

Your experimental design is within subjects.
Would you expect more variation due to sampling error for your experimental design or for a between subjects design?   Help
Is the risk of confounding variables greater for your type of experiment, or for a between subjects design?   Help
Consider your experiment for a moment and see if you can identify any confounding variables. How might you minimise their effect?
Introduction to Your Study | Stating a Hypothesis