How much difference does it make? Notes on understanding, using, and calculating effect sizes for schools

Publication Details

A good way of presenting differences between groups or changes over time in test scores or other measures is by ‘effect sizes’, which allow us to compare things happening in different classes, schools or subjects regardless of how they are measured. This booklet is designed to help school staff to understand and use effect sizes, and includes handy tips and warnings as well as useful tables to calculate effect size values from change scores on standardised tests.

Author(s): Ian Schagen, Research Division, Ministry of Education and Edith Hodgen, New Zealand Council for Educational Research.

Date Published: March 2009

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  To view the individual chapters please refer to the 'Sections' inset box.

Section 3: Possible comparisons using effect sizes

The following broad types of comparisons are possible using effect sizes:

  • differences in scores between two different groups (e.g., boys and girls)
  • changes in scores for the same group of students measured twice
  • relationships between different factors and scores, all considered together.

In the first type of comparison (between-group differences) we calculate an effect size by taking the difference in mean scores between the two groups and dividing that by the nominal standard deviation.

The second type of comparison (change scores) is what we have considered in the preceding section. The effect size is simply computed as the average change in scores divided by the nominal or assumed standard deviation.

Basically, the calculation is the same whatever we're doing: take a difference in mean scores and divide by a standard deviation.

The third type of comparison is more complex, but can be important. For example, suppose we have a difference between boys and girls, and also a difference between those who have done some homework and those who have not. There may be a relationship between these two groups, so that when we consider the data all together the effect size we get for each factor controlling for the other is smaller than it would be otherwise. Using a statistical technique such as regression we can estimate such "joint effect sizes" and compare the magnitude of the boy/girl difference with that of the homework/no homework distinction, each taking account of the other.

Footnote

  1. There are other, more appropriate, ways of measuring effect sizes in regression and related models, but these are outside the scope of this discussion.