Outcome 1. Shifts in student learning and achievement

Indicators for outcome 1

Students will:

• be clear about what they are learning and why, how they will achieve this, what the learning might look like and how well the learning has been achieved (self assessment)
• self evaluate and self regulate their learning by reflecting on their current achievement and progress to determine their next learning goals.

Results

Although the concepts of learning and achievement are sometimes used interchangeably, there is an important distinction between the two terms. In simple terms, learning is the process or experience of gaining knowledge or skill. It can be likened to the journey towards a destination. For deep learning to occur, the learner needs to be aware of his or her learning, for example, by reflecting on the processes used, by questioning how the learning can be applied to other contexts, by having an openness to new ways of learning, and by considering the views of others and the ways in which their learning may need to be modified. Deep learning is ongoing and is integrated with assessment. By contrast, achievement refers more to the successful completion of something (especially by means of exertion, skill, practice or perseverance). It can be likened to arrival at a destination.

A simple numeracy example makes the distinction between the terms clear: achievement is when a child can correctly answer that 3+5=8. Learning can be shown when the child is able to demonstrate or explain several strategies for finding the answer (including manipulating concrete materials in several ways, explaining verbally or in writing), and then use this knowledge in a variety of different contexts.

Students in classes where teachers had participated in the AtoL project became more confident in understanding what they were learning and why it was important. They were able to articulate learning intentions and success criteria. In many classes, students were also becoming more aware of self and peer assessment.

Typical responses given by students when asked about their learning included:

• We are learning to write persuasively. You know how some reasons are stronger than others.
• If you try to persuade someone and you didn’t have good reasons you wouldn’t be convincing.
• We are learning how to take notes so that we can write an effective report for our class newsletter.

As well as determining the impact on student learning, the evaluators explored the impact on student achievement. AsTTle V4 achievement data were collected from a sample of schools at the beginning and end of each year. The data were analysed in order to determine the amount of shift and the effect size of the impact of the AtoL programme. Only students who were present for both tests were included in the final data set. Schools collected data in reading and writing. While it must be acknowledged that it is almost impossible to attribute student gains solely to one intervention, the results achieved provide evidence that AtoL does impact positively on student achievement.

In the following graphs and charts, national AsTTle V4 mean data and effect size using Cohen’s d¹ were used to compare AtoL mean results with the expected differences in student achievement within a teaching environment without professional development intervention.

Effect size

In professional development interventions it is considered that an effect size (defined below) of between 0.3 (Wiliam, Lee, Harrison and Black, 2004) and 0.4 (Hattie, 1999) demonstrates changes beyond natural maturation or chance.

For the purpose of this evaluation the definition of “effect size” is an index that measures the strength of the association between one variable and another. These indices take different forms depending on the measure being used and are commonly reported where the main interest is in the difference between group means.

The effect size tool used in this report is Cohen’s d, which measures the difference between means. In the case of AtoL, the effect size measured was the shift between beginning of year (BOY) and end of year (EOY) scores relative to the standard deviation. The theoretical distribution of the national asTTle V4 scale has a standard deviation of 100. Therefore the mean differences reported in the national asTTle V4 scores for reading and writing, when divided by 100, can be used to indicate Cohen’s d effect size.

Interpretation of Cohen’s d effect size scores:

    < 0.20     small effect
    ~ 0.50     medium effect
    > 0.80    large effect

”Shift” data, effect size (Cohen’s d) and national asTTle V4 data are all used in this report to provide a more comprehensive basis for analysis and comparison of results than is feasible from using only one of these measures.
 

Writing

Most schools in their first year of AtoL focus on writing for developing formative assessment principles. They use both national curriculum exemplars and asTTle V4, but due to space restrictions, only asTTle V4 data are reported here.

Graph 1 compares the shift of the mean writing scores of AtoL students with national asTTle V4 shifts. The end of year (EOY) scores are the sum of the beginning of year (BOY) scores and the shift scores. The shift in the national data in each year represents the expected shift from the previous year.

Graph 1: AtoL and national primary writing shifts (years 4-8)

Commentary

In years 4-8, the mean shift in AtoL scores (blue line) is higher than the mean national shift (red line). In other words, data for each year group show greater shifts for schools involved in AtoL compared with the national data. 

Graph 2 indicates the effect size of students in the AtoL project compared with the national AsTTle V4 norms. The line at Cohen’s d 0.50 represents a medium effect.

Graph 2: AtoL and national primary writing effect sizes

Commentary

Graph 2 shows effect sizes for students from years 4-8. The effect sizes in AtoL are much greater than is evident nationally. This provides strong evidence that the AtoL results have a medium (year 8) to large effect (year 6) between beginning of year data (BOY) and end of year data (EOY).

Reading

STAR and asTTle V4 were the main tools used for assessing progress in reading. In order to make comparisons between reading and writing, only asTTle V4 data are used to make comparisons with AtoL data.

Graph 3 shows the shift in the mean reading scores of students from the BOY (pretest) and compares it with the mean national shift from the previous year.

Graph 3: Reading shifts Years 4-11

Comments

In Graph 3 we see that the AtoL shifts in years four to eight are above the national mean in terms of both ‘shift’ and EOY scores. The secondary patterns are not as clear cut.  One factor is that the national shifts for year nine (117) and ten (94) are much greater than the average for other years.  One school involved at year nine reportedly did not engage in the AtoL process as fully as intended and this helped bring down the overall shift.  At year 11 the AtoL shift (92) is more than twice the national shift (40).

Graph 4 indicates the effect size of students in the AtoL project compared with the mean national asTTle V4 scores. The red line represents a Cohen’s d of 0.50 which is a medium effect.

 
Graph 4: Reading: AtoL and national effect sizes (Cohen's d)
 

Commentary

Graph 4 demonstrates that Cohen’s d in isolation does not tell the whole story. In this graph, the lowest Cohen’s d score is 0.19 (shift = 16). However from Graph 3 we can see that for year 8, the mean beginning of the year AtoL score (554) is much higher than the beginning of the year national score (508) and by the end of the year, the AtoL score had shifted even higher. Graph 4 in isolation does not convey this impact. If one puts aside the high asTTle V4 Cohen’s d scores for year 9 (117) and year 10 (94), the AtoL effect size scores for year 10 and year 11 are impressive.

Summary of findings for student achievement

These graphs indicate the very positive impact of AtoL on student achievement. Student achievement increased at a rate of up to twice that of the expected benchmarks for educational interventions of 0.3 (set by Hattie, 1999), 0.4 (set by Wiliam et al, 2004), and 0.5 (set by Cohen’s d).

Footnote

1. Cohen’s d: a measure of effect size. It measures the difference between means relative to a standard deviation.