National Education Findings of Assess to Learn (AtoL) Report
This report describes the impact that the Ministry of Education’s professional development Assess to Learn Project has had on teachers, students and schools in New Zealand.
Author: Dr Jenny Poskitt (Massey University) & Kerry Taylor (Education Group Limited).Date Published: July 2008
Skip to:
- Introduction
- Executive Summary
- Literature Review
- Methodology
- Key Findings
- Outcome 1: Shifts in student learning and achievement
- Outcome 2: Shifts in student learning and achievement
- Outcome 3: Developing coherent school practice to promote better learning
- Outcome 4: Demonstrate a culture of continuous school
- Conclusions
- References
Methodology
Quality teaching and assessment is at the heart of increased student engagement, learning and achievement. Therefore, the challenge in evaluation of professional development is to not only measure the impact of teacher learning on student achievement but to identify significant elements that enhance teaching practice.
Approach
Evaluation of the AtoL project occurred over three years, however, the 2007 data form the focus of this report. The evaluation was primarily formative although it provided some summative data to inform policy development in the Ministry of Education. A multi-method design, involving both qualitative and quantitative methods of data collection, was used. Triangulation of methods, data and people strengthened the reliability and validity of this study. The evaluators used a collaborative and responsive approach to ensure their work would be meaningful and formative. To achieve this, the evaluation was planned cooperatively with AtoL directors and the ministry, based on AtoL’s key outcomes and a need to give project directors formative feedback. The evaluation also aimed to gain a national picture about the effectiveness of the AtoL project. An iterative process was used where the evaluators and the AtoL providers worked together to develop, refine and apply evaluation tools to gather meaningful data nationally.
Data consisted of:
- student achievement data collected from schools in term one 2007 and term four 2007. AsTTle V41 data were collected only from schools that had sufficient knowledge and experience to use asTTle V4 effectively.
- data on teacher learning (classroom observations) collected by facilitators from new and continuing schools in term one and term four
- planning analysis collected by facilitators and sent to the evaluators at the end of the year
- responses to questionnaires developed for this evaluation. The questionnaires were sent independently from the national evaluators in May and November to principals, lead teachers and one other teacher in all participating schools.
Evaluation framework
The effectiveness of professional learning opportunities, according to Guskey (2002, 2005), can be identified through five levels of evaluation. These are:
- participants’ reactions to the experience
- participants’ learning
- the organisation’s support and change
- participants’ use of new knowledge and skills
- student learning outcomes.
Table one summarizes the evaluation frame used throughout the evaluation of AtoL.
Table 1: Guskey’s levels of evaluation of professional development programmes
| Level of evaluation | What is measured or assessed? | What questions are asked? | How is information gathered? |
| One Participants’ reactions | Initial satisfaction with experience | Did they like it? Was their time well spent? Was the facilitator knowledgeable? Did the material make sense? Will it be useful? | Questionnaires or surveys |
| Two Participants’ learning | New knowledge and skills of participants | Did participants acquire the intended knowledge and skills? | Simulations (scenarios) Demonstration Participants’ reflections |
| Three Organisation’s support and change | The organisation’s advocacy, support, accommodation, facilitation and recognition | Were sufficient resources made available? Were problems addressed quickly and efficiently? Was implementation advocated, facilitated and supported? What was the impact on the organisation? Did it affect organisational climate and procedures? | Minutes of follow up meetings Questionnaires Structured interviews School records |
| Four Participants’ use of new knowledge and skills | Degree and quality of implementation | Did participants effectively apply the new knowledge and skills? | Questionnaires Structured interviews Participants’ reflections Observations Video/audiotapes |
| Five Student learning outcomes | Student learning outcomes Student achievement Attitudes and dispositions | What was the impact on students? Did the professional learning affect students’ achievement? Are students more confident as learners? Is attendance improving? Are dropouts decreasing? | Student records School records Questionnaires Structured interviews |
Adapted from Guskey (2005).
Guskey’s model provided an appropriate framework for evaluating the effectiveness of the AtoL professional development project. It addressed the need to take into consideration the effectiveness of professional development for participants as well as its impact on student achievement and learning. Guskey’s model was used to inform the evaluation tools, criteria and analysis and interpretation of findings. A summary of the analysis of data from these tools forms the basis of this report. The evaluation tools included:
- national questionnaires (principals, lead teachers and classroom teachers)
- interviews (facilitators, teachers)
- classroom observation schedules (teachers and students) and associated matrices (see appendix)
- planning analysis
- student achievement data.
Footnote
- AsTTle V4: Assessment Tools for Teaching and Learning, version four