Incredible Years-Teacher Evaluation Summary Publications
Publication Details
The fourth report is a summary of the evaluation findings focusing on the short term outcomes for teachers and their students.
Author(s): Cathy Wylie and Rachel Felgate [NZCER]
Date Published: August 2016
Introduction
The Incredible Years-Teacher programme
The Incredible Years Teacher (IYT) is a research-based teacher development programme that is part of the suite of linked Incredible Years (IY) programmes for teachers, parents and children developed by Carolyn Webster-Stratton and her team in the United States. The goal of the interlinked Incredible Years programmes is "to prevent and treat young children's behaviour problems and promote their social, emotional, and academic competence".1 Incredible Years Teacher (IYT) is delivered by a pair of IYT-trained group leaders through full-day workshops for groups of around 16–20 teachers, once a month over six months. The group leaders also provide coaching for individual teachers in between workshops. The coaching involves discussion and classroom observation and feedback. Each workshop's content is detailed in the IYT manual. This manual also provides process guidelines for facilitators, and checklists for planning and review. The programme is intended to be tailored to the specific needs of each participating group of teachers.
IYT is part of the Ministry of Education's Positive Behaviour for Learning (PB4L) strategy. It was first offered in 2011. It is provided by Resource Teachers of Learning and Behaviour (RTLB), non-governmental organisations (NGOs), such as kindergarten associations, and Ministry of Education staff. More than 7,000 early childhood education (ECE) and primary teachers in New Zealand have now completed IYT.
The evaluation
The evaluation focused on the short-term outcomes for teachers and students, the factors enabling positive change, and whether the IYT programme in New Zealand was being delivered as intended. It used material from group leaders and participants in IYT programmes that ran over the first eight months of 2014. We based our data collection and analysis on the theory of change developed with the Ministry of Education and IYT practice leaders. Table 1 shows the data we have used.
Outcomes for students | 428 teacher responses to a NZCER paper survey early in their IYT programme and an email survey in late 2014 about the 'target' child they focused on during the programme |
---|---|
Outcomes for teachers | 1,103 teacher responses to the NZCER paper surveys completed at their first and final IYT workshops |
Delivery of IYT in | 97 IYT group leaders' responses to an email survey in mid-2014 |
Factors supporting positive change | All the data sources above |
Analysis of responses showed that they were generally representative of the known characteristics of IYT group leaders and participants in New Zealand, so we are confident that the evaluation findings are sound.
Key findings
"My classroom is a much more positive place, and there is so much more time for learning."
(Participating primary teacher, May 2015)
Outcomes for students
At the end of their programme, the majority of teachers reported gains for their students that they attributed to their IYT learning. These gains were that ...
- there was less disruptive behaviour in their class or group: reported by 90 percent of the ECE teachers and 75 percent of the primary teachers
- students were more focused on their learning work: reported by 85 percent ECE and 70 percent of primary teachers
- students had better self-regulation: 82 percent ECE, 75 percent primary
- students showed more problem-solving skills: 78 percent ECE 69 percent primary
- students could ignore negative behaviours of others more: 74 percent ECE, 74 percent primary.
Comments made by teachers about the IYT programme as a whole were generally very positive, with illustrations given of gains made for student behaviour and attention to learning in their classes overall.
The majority of the 'target' students who were a particular focus for teachers in their IYT work were also reported to have made gains. Target students were those whose behaviour was challenging for their teacher. Working individually with their IYT group leader, teachers devised a behaviour plan for their particular target student using the IYT strategies. During the six months of the IYT programme the teacher and group leader then generally discussed the target student's progress together in relation to the behaviour plan and altered it if needed.
At the end of their IYT programme, 85 percent of the ECE teachers and 78 percent of the primary teachers noted improvements in their target student's behaviour. Some months later, at the end of 2014, 66 percent of ECE teachers and 47 percent of primary teachers reported that this improvement represented a large positive difference in target students' behaviour, while almost all the rest felt that small positive differences had occurred.
Marked increases in positive interactions with their peers and teachers, self-management and attention to learning were reported for the target students over the course of 2014. For example:
- 57 percent of target ECE students often or always persisted with solving a problem at the end of 2014 compared with 21 percent at the start of the IYT programme
- 45 percent of target primary students often or always followed all class routines without having to be reminded at the end of 2014, compared with nine percent at the start of the IYT programme.
Target student achievement also increased. By the end of 2014, 55 percent of the primary target students were described by their teachers as achieving at a medium or high level for their class compared with 35 percent at the start of 2014.
Parental support for ECE target students also improved.
The gains they saw with their target students encouraged teachers to develop and use behaviour plans with other students in their class. By the end of their IYT programme, 81 percent of ECE teachers and 80 percent of primary teachers reported improvement in the behaviour of these other students also.
Outcomes for teachers
Teachers felt their confidence in managing student behaviour increased over the course of the IYT programme.
At the start of the programme: 46 percent of ECE and 55 percent of primary teachers rated themselves as confident or very confident that they could manage their current students' behaviour.
At the end of the programme: 88 percent of ECE and 74 percent of primary teachers now considered themselves as confident or very confident that they could manage their current students' behaviour.
Confidence levels in managing student behaviour were maintained 8–9 months after the end of teachers' IYT programmes (May 2015).
At the end of their programme, 90 percent or more of the ECE teachers and 85 percent or more of the primary teachers thought that what they had gained from IYT included:
- more awareness of the value of being proactive in relation to children's behaviour
- a much deeper understanding of how to teach social and emotional skills
- useful strategies to work with children in ways that encouraged their class engagement.
Teachers made real changes in their practice that meant they were more consciously managing behaviour, defining and recognising positive behaviour from students and building children's social and emotional skills, rather than acting reactively after challenging behaviour and paying counter-productive attention to misbehaviour. There was a marked increase, for example, in positive coaching, problem-solving and anger management strategies, and having a clear classroom discipline plan.
Teachers made more use of IYT strategies by the end of the programme, and they also became more effective in the way they used these strategies.
A marked theme in comments made by the majority of teachers at the end of their programme was that IYT had renewed their sense of agency. IYT gave them the understanding and tools they needed to reframe how their classes operated, with testimonies of new calm for both students and teachers.
Teachers' comments about the value of their IYT learning remained very positive when surveyed 8–9 months later, in 2015. There were some changes in the frequency of their use of IYT strategies. Half of the IYT teaching techniques we asked about continued to be used by primary teachers often or very often and ECE teachers continued to use a third of the IYT techniques often or very often.Some IYT teaching techniques appeared to be easier to sustain than others. By May 2015, frequency of use of some of the teaching techniques had dropped, particularly those that involved coaching, modelling and some of the deeper aspects of the IYT approach. Declines in use were most apparent for ECE teachers. It was also noted that there was evidence of an increased use of some of the negative strategies for managing student behaviour among primary teachers.
Delivery of IYT in New Zealand
IYT was chosen as a professional development programme to support positive behaviour in New Zealand because of both its content and coherence. Evaluations overseas had shown that it was effective. It provides a structured programme, using video vignettes to illustrate strategies in use by teachers with small groups of students, but also emphasises 'flexible implementation' based on participants' needs, which is consistent with research about effective teacher learning. This programme is not one that can be taken off the shelf and used by any professional development provider. It can only be provided by trained group leaders, who review their own work using IYT tools, and are given feedback from peer coaches who are further along the IYT group leader pathway. Our evaluation therefore looked at the support for group leaders, as well group leader and teacher practice in relation to IYT delivery and learning, as reported in the various surveys used for the evaluation.
Teachers were largely positive about their experiences in the IYT workshops. The IYT group leaders were reported as modelling positive interaction, reflection and inquiry. The workshops engaged teachers and underlined their sense of agency about student behaviour by providing variation in activity, and ensuring that teachers shared their experiences with one another, so that the strategies were able to be seen as 'live', and not as something outside their own scope. This interaction with other teachers was highly valued. Teachers' views about the IYT video vignettes however suggest that they are not a straightforward resource for supporting change. On the one hand, they were valued as demonstrations and for stimulating discussions; on the other hand, they sometimes also caused scepticism given the differences in group size and adult resources between the American and New Zealand contexts.
Some variability around what IYT group leaders were doing in their work with individual teachers between workshops was evident. A substantial minority of teachers did not 'usually' discuss their target child's progress or the self-report inventory with their group leader. Teachers who usually had such discussions reported more gains in practice, and therefore it is important that these regular discussions occur for all teachers undertaking the IYT programme.
In mid-2014, IYT group leaders themselves were not all getting the ongoing support envisaged in the IYT model. The time available for IYT work was less than needed for 62 percent of the group leaders, particularly time to work individually with teachers between workshops.
Almost half the group leaders were not able to keep improving their practice through coaching with accredited peer coaches. Such access was associated with group leaders' use of other reflective processes in the IYT model that support ongoing fidelity, and confidence with presenting the IYT approach, such as the use of checklists in video review. Those who were most recently trained had less of this kind of support to ensure fidelity and grow them as group leaders. There were also some issues evident around ensuring that newer group leaders had sufficient experience of leading groups to keep developing, and that effective practice (positive framings of the video vignettes where teachers react negatively to them, for example) can be shared. Responsibility for these aspects fell to different organisations, making it hard at times to ensure coherence.Since the evaluation survey data were collected, the Ministry of Education has undertaken IYT workforce planning, and has recently contracted Massey and Canterbury Universities to provide group leader development and support. It will be important for the ongoing quality of IYT delivery in New Zealand that the issues identified in this section can be addressed.
Factors supporting positive change
Gains from IYT and positive views of it were across the board: the programme appears to be effective regardless of teaching experience or ethnicity.IYT programme delivery and the teachers' own contexts both played a part in changes made by teachers and gains for students.
IYT programme delivery
The quality of the IYT group leaders — their teaching, preparation and interest in teachers (eg, by modelling what teachers can do in their own classrooms) — matters. It also matters that they work well with individual teachers between workshops, to both share their knowledge about strategy choice and use, and model ongoing review and inquiry.
The behaviour plan work for 'target' students was particularly important for putting the IYT approach into practice. It showed the value of a systematic approach, and related teacher strategies to student behaviour. It gave teachers a heightened awareness of the centrality of their relationship with the student. It also gave tools to identify (and defuse) triggers for negative behaviour, and build positive behaviour in manageable steps.
It was great and very useful, focusing on one aspect of behaviour and building it up slowly. Very specific and reachable goals. Intend to use it in the future. I am a much better teacher and the strategies have helped me grow personally and professionally. Thank you IYT.
When things start to go wrong in class you realise it's because you are not sticking to the plan or you are neglecting the foundation stages of the pyramid. The behaviour plan is important as you need to think about specific behaviours, specific opposite behaviours and the possible cause — attention/aversion. The behaviour plan has such a positive impact on the classroom atmosphere.
Teachers' ECE or school contexts
Most of the challenges teachers expressed in putting their IYT learning into practice came from their own school or ECE context. Such challenges were expressed by 33 percent of ECE and 29 percent of primary teachers at the end of their IYT programme, and 29 percent of ECE and 20 percent of primary teachers in 2015. Lack of consistency among their colleagues was a particular challenge for ECE teachers, who sometimes were the only teacher in their ECE service to have done IYT. Lack of time was the main challenge for primary teachers.
Our analysis found that gains for teacher practice came when they had active discussion of IYT work with their colleagues, shared that work with others, provided mutual support to make change, and felt supported to make those changes. Simply having done IYT with others in their school or ECE centre was not enough on its own. Teacher aide and RTLB support for IYT approaches was also shown to be useful for making and sustaining change in practice.
In 2015, most teachers reported that they had some support for making or sustaining their IYT-related work within their school or ECE service, though not as much as when they were on the IYT programme. We also found that teacher practice changed more when IYT was seen to be compatible with the curriculum, as enacted in particular ECE centres and schools. For example, where theNew Zealand Curriculum's (NZC) key competencies were fully included in a school's approach to curriculum and its timetabling, IYT was seen and valued as fully compatible with their enacted curriculum.
Our full findings are in these reports:
- Wylie, C., & Felgate, R. (2016). 2014 IYT group leaders' views and experiences. NZCER report for the Ministry of Education.
- Wylie, C., & Felgate, R. (2016). IYT in New Zealand: Participants' reports of learning and change. NZCER report for the Ministry of Education.
- Wylie, C., & Felgate, R. (2016). Use of IYT learning in New Zealand. NZCER report for the Ministry of Education.
Footnote
- The Incredible Years website.
Navigation
Contact Us
Education Data Requests
If you have any questions about education data please contact us:
Email: Requests Data and Insights
Phone: +64 4 463 8065