2013 Annual Evaluation Report for the Teach First NZ programme pilot: Delivered in partnership with the University of Auckland Publications
This evaluation’s main purpose is to understand how well the programme has been implemented and to what extent it has achieved its objectives. The key evaluation questions are:
- How well (effectively and efficiently) has the programme been implemented?
- To what extent has the programme achieved its overall outcomes and objectives?
To answer the evaluation questions the evaluators developed a set of evaluation criteria in consultation with the Ministry and the Teach First NZ partnership. Our key findings are provided for each of the evaluation questions in relation to these criteria.
Author(s): Jenny Whatman, Marie Cameron, Liesje Stevens and Lorraine Spiller, New Zealand Council for Educational Research.
Date Published: June 2014
The evaluation data for the first year of the Teach First NZ pilot programme were collected through document analysis, an online survey (Me and My Class) administered to students, interviews, and site visits to the nine schools hosting the 16 participants that make up the 2013 cohort. The interviews were with key personnel in schools (including participants, mentors, co-ordinators, Heads of Department/Heads of Faculty/Heads of Learning Area (HoDs), principals, and other teachers in the school including New Zealand Post Primary Teachers Association (PPTA) office holders) and with the Teach First NZ partnership and Faculty employees (including VCSs, and staff who teach on the university’s established Graduate Diploma in Teaching (Secondary)). Two members of the evaluation team also spent a day observing an Assessment Centre (selection day) which involved interviews, a group exercise, a sample teaching lesson, and self-evaluation.
How well (effectively and efficiently) has the programme been implemented?
The programme was, in general, successfully implemented, with the Teach First NZ partnership being seen to provide timely and responsive support to participants and schools. The rigorous selection process and the high calibre of the 2013 cohort were seen by the evaluation team (based on the views of the different groups of people we talked to) to be major strengths of the programme. School principals and co-ordinators spoke highly of the participants and the Teach First NZ partnership staff.
Where there were any issues, reported by a minority of schools during the year, these appeared to relate, for the most part, to in-school issues (including inadequate communication between some personnel). To a large extent, variations in implementation depended on the clarity of internal school communications about the Teach First NZ programme, the suitability of organisational arrangements (including timetabling), and the placement of the participant (for example, whether located in the same department as their mentor), the learning culture within their department, and the quality of the mentor.
Who the programme attracted
The quality of the programme was judged by the extent that it attracted high-quality applicants, some of whom may not otherwise have undertaken teaching at this time (particularly in schools serving lower decile communities).
The programme was successful in attracting high-quality applicants. Sixteen participants (6 percent of those who applied) were selected for the 2013 cohort from a pool of 261 who completed the online application, although the pilot allowed for 20. The programme aimed to attract high-calibre participants and targeted males, Māori, and Pasifika graduates. The participants were perceived by the evaluation team (based on the views of the different groups of people we talked to) to be high calibre and a quarter (n = 4) are Māori or Pasifika and a quarter (n = 4) are male. Nine were English majors, three te reo Māori, three mathematics, and one chemistry and science. Over half the participants had worked and/or travelled since leaving university, some of them in teaching or tutoring roles. Ten participants had already decided to become teachers (either now or at some point in the future), and half of this group had already been accepted for a traditional ITE programme. Two thirds chose the Teach First NZ programme in preference to 1-year graduate diplomas because this programme was presented as high challenge and because of its mission to help redress inequality by deliberately working in schools that serve low-decile communities with high populations of Māori and Pasifika students. The Teach First NZ recruitment and selection team has made adjustments for 2014 to target and recruit higher numbers of males, Māori and Pasifika participants, and more science majors.
The selection process
The selection process for the 2013 cohort was thorough and exacting. All those who had been involved in selection saw it as a key strength of the programme. Participants reported that they had found the selection process “thorough”, “rigorous”, and “intense”, and were impressed with the professionalism of staff and the effort to seek out capable candidates with the best potential. Selectors’ judgements appeared to be considered and sound. While the selection process was highly regarded, two people suggested improvements to the process, particularly for Māori and Pasifika candidates, and to assess cultural competence.
Summer Initial Intensive
We collected data to evaluate: how well the School Observation and Reflection (SOAR) tool prepared participants for their SII; to what extent the SII strengthened participants’ motivation to teach in low-decile schools and strengthened their understandings of the cultures of their students and how to incorporate this understanding into their teaching; how well prepared participants felt to begin teaching; and to what degree the SII built a sense of connectedness within the cohort.
Participants reported that the SII had helped them to develop the mind-set they needed to teach in schools that serve low-decile communities. An emphasis at the SII was on developing cultural competence, and the te reo Māori VCS took a lead role in managing this aspect of the programme and the participants’ involvement in it. The SII was seen to be intellectually stimulating by almost all participants and to have contributed to building strong relationships within the cohort and with the Teach First NZ partnership. Improvements participants suggested were more practical and pedagogical input regarding teaching their subject, more time in their host school before starting teaching, and more time practising teaching.
A successful retention rate for each 2-year cohort was set at 90 percent.
All of the 2013 cohort were still in the programme in November 2013, and all expected to complete their 2 years with a limited authority to teach (LAT). At this stage the majority of participants expect to stay in teaching beyond their 2-year placement, as they work towards full teacher registration. Three participants gave indications that they may not stay in teaching beyond the 2 years.
For the programme to be successful, it needs to: have a common clear vision of effective teaching which permeates course work and practice in schools; have a strong core curriculum; apply case methods and teacher research; allow participants to confront and rethink assumptions about learners; be judged by the participants as intellectually challenging and practically relevant.
The programme was seen by participants and the Teach First NZ partnership to be very responsive to participant needs. The clinics and intensives during the year, along with subject meetings, provided stimulus and input that reportedly revitalised participants. The SII established the parameters for a strong core curriculum and focused on challenging assumptions about learners. The university assignments were integrated into the school-based teaching components and applied case methods and teacher research. These approaches were maintained throughout the year with ongoing course work and assignments. A small number of participants commented on a lack of clarity in some assignments and course booklets. A requirement of the programme is for all participants to build a portfolio of evidence against the GTS and Registered Teacher Criteria (RTC). A central e-portfolio system had been established by the university in July; however, most participants chose to use established systems in schools (which vary widely) for evidence collection. School teaching staff would have liked more advance information about the year’s programme schedule/timelines so they could support participants with what those teachers regarded as a heavy workload.
Support to participants from the Teach First NZ partnership
We evaluated the extent to which participants felt that their wellbeing has been important to Teach First NZ and that they are part of the Teach First NZ community. Overall, participants reported they felt very well supported by the Teach First NZ partnership team and by VCSs. The kaihapai (pastoral care) role was appreciated by participants. Opportunities for ongoing communication within the cohort were well established and well utilised by almost all participants. Participants in Northland felt very well supported by Teach First NZ, with liaison visits to Northland viewed as useful and timely. The Northland participants often had to travel long distances to take part in workshops and meetings, although Teach First NZ compensated for the financial cost and sometimes arranged meetings in North Auckland so that participants could more easily take part.
Support to participants from host schools
We assessed to what degree the quality of mentoring and school support enabled participants to be successful beginning teachers. Overall, mentoring and school support was enabling participants to become successful beginning teachers. Schools, for the most part, welcomed and supported the participants as new members of staff. For the very few exceptions, participants had found ways to overcome any initial problems by the time we visited and were well integrated into the school and accepted and appreciated by staff.
The quality of the subject department in which the participant was placed was critical, both to the wellbeing of the participant and to the support they received in learning to teach. Host schools provided support to participants in different ways. Where a participant was in a strong department, as they were in the majority of cases, that department provided “wrap-around” support alongside the mentor. Co-ordinators in some schools provided pastoral care or subject-specific information to strengthen the mentoring that participants required. Participants in very small or less organised departments had less support in learning to teach, even with a good mentor, than participants placed in larger or well-organised departments.
Host schools preparation for, and support in, their roles
We looked at the extent to which host schools thought the preparation and support for their roles by the partnership was high quality and how well supported they felt.
Principals reported that the Teach First NZ partnership had prepared them well for their roles in the programme. Within a small number of schools inadequate communication meant that a minority of departments were not well prepared for having a new staff member. Most co-ordinators felt well prepared, although in some schools the co-ordinator role was not established until after the participants had begun in the schools. Once-a-term co-ordinator meetings were appreciated, although difficult for Northland co-ordinators to attend. The majority of HoDs who were not also mentors felt well informed and were happy with the level of information and communication with the Teach First NZ partnership.
We reviewed to what degree mentor teachers were: well supported by the partnership; provided regular high-quality observation, mentoring and feedback to participants; and helped participants to become part of the wider school community.
Levels of mentor satisfaction with their support from the Teach First NZ partnership were mixed: half were very satisfied and the other half, while also generally satisfied, said they would have liked more direction about observations and expectations for participants. The mentor meetings twice a term provided valuable opportunities to learn what others were doing and to learn more about mentoring. A few mentors considered that mentor training should take more account of their previous experiences as mentors, and of other approaches to mentoring already operating in schools. This was particularly the case for some mentors who had pivotal roles in their school’s Te Kotahitanga programme.
he quality of mentoring and the relationships between participant and mentor was variable. All mentors conducted observations of participants and provided feedback to them. Most mentors provided weekly observations, although in a very few cases observations were irregular or the feedback did not meet participants’ needs. Sometimes this was a consequence of timetabling that we were told made it difficult for the mentor to meet with the participant or to conduct observations, or because the mentor was not in the same subject area as the participant. Where mentors were not in the same subject area, participants required subject-area support from additional sources. As this programme is reliant on school personnel to support participants to learn to teach the New Zealand curriculum, participants require strong mentoring in their subject area.
The mentor was pivotal in helping the participant become part of the wider school community. In a very few instances participants relied on the co-ordinator or another staff member to help with this integration into the school.
Visiting Curriculum Specialists
For the programme to work well, VCSs need to provide regular high-quality observation, mentoring, and feedback to participants. There should also be explicit links between the curriculum papers in the qualification and the feedback provided by the specialists in schools.
In some instances, the VCS feedback clearly showed the links to curriculum papers; in others, attention to immediate classroom practice was foregrounded, especially earlier in the year. Participants appreciated ongoing relationships with VCSs, as did most mentors. VCSs were seen to be very responsive to participants’ needs. Co-ordinators appreciated receiving the summary statements from VCSs after observations. A small number of HoDs who were not mentors wanted more involvement with the VCS. A small number of participants found that VCSs provided different, and sometimes conflicting, advice to that provided by mentors, which left them unsure about whose advice to follow. A few participants said the VCS visits did not allow sufficient time for discussion. All VCSs really enjoyed their roles and thought the amount of time allocated to visiting was adequate. A number of participants commented positively on the learning for them during the formal observations conducted by the mentor and the VCS together.
We looked for evidence that the programme was responsive to feedback from participants and participating schools.
Overall, the small size of the cohort and the considerable investment of staffing in the programme has made it possible for the programme to be very responsive to participant and school needs. Evaluations conducted regularly by the partnership and made available to the researchers provided ample opportunity for participants to have input into the programme and for programme staff to respond to these suggestions. Teach First NZ has been able to quickly accommodate requests for information and to arrange resources and information to suit individuals and groups. Participants reported that they had been very well supported by the Teach First NZ partnership when difficulties arose.
To what extent has the programme achieved its overall outcomes and objectives?
The second evaluation question looks at how well and to what extent the programme achieved its overall outcomes and objectives. To answer this question, the evaluation focused on: the effectiveness of participants’ teaching, their level of support for the pastoral life of the school, the leadership development strand of the programme, the ongoing involvement and/or retention of participants, programme impact on quality of teaching and learning in participating schools, status of teaching, enablers and barriers to success, and alumni engagement. On the basis of the data we have collected so far, we can make some preliminary judgements about the 2013 cohort’s teaching, engagement in the pastoral life of the schools, and leadership development.
Effectiveness of participants’ teaching
To gauge the effectiveness of participants’ teaching we evaluated the extent to which participants’ students (particularly their Māori and Pasifika students) are engaged in learning, how well participants’ teaching aligns with the Teach First NZ model, and student achievement.
Most of the cohort were perceived to be teaching as well or better than a ‘typical’ first-year provisionally registered teacher (PRT). In September, two of the 16 participants were reported by VCSs and school mentors to be finding the transition into teaching more difficult than the other participants. Extra support was being provided to them by the partnership and by their schools. All participants were enjoying positive relationships with some or all of their classes.
Participants were reported to be extremely hard working and resilient and in most cases to have ‘won over’ difficult classes. The NZCER survey Me and My Class was used to provide an indication of the extent to which students in participants’ and similar classes experienced teaching strategies and reported their engagement. There was no statistically significant difference between participant and comparison classes at Year 9 or 10. Although we would caution against placing too much emphasis on the Me and My School survey results, there were indications that participants’ Year 9 students may have been more engaged than students in comparison classes and more engaged than participants’ Year 10 classes. Students in Year 10 comparison classes indicated greater engagement than students in participants’ Year 10 classes. Students in participants’ classes responded more positively to some items about teaching strategies. These tentative findings suggest that students report their engagement in participants’ classes as similar to student engagement in classes taught by more experienced colleagues. There was evidence shown to us by some participants that NCEA or standardised (for example, e-asTTle) assessment results for participants’ classes compared favourably with other classes, and this was endorsed by other staff members.
Support by participants for pastoral life of school
This evaluative criterion relates to the expectation that participants will contribute positively to wider school activities.
In their first year, participants have been cautious about becoming involved in too many activities outside of teaching. At the same time, they are seen by school staff to be very well integrated into the schools and to be supporting sports and cultural events and activities, staff professional learning and development (PLD), and homework and NCEA revision centres.
Leadership development strand
The leadership development strand is a key element of the programme. The aim is to produce teachers who will, in the long term, provide “a network of leaders in education and across all fields, who are committed to addressing educational inequality”. In the first year participants are expected to demonstrate effective leadership of students. We assessed the extent to which participants were demonstrating this leadership.
Almost all participants were showing leadership in the classroom, with some also taking strong leadership roles within their department. Participants in two schools were expected to take responsibility for a tutor/form class, while others shared the responsibility. For the most part, participants reported enjoying this role, although a few reported they were left to cope on their own, without additional support and guidance. Principals and co-ordinators reported that all participants were having a positive impact on the school, particularly in their departments. Participants’ content knowledge, the resources they developed and shared, their facility with technology, and their attitude towards wanting to do the best for all students were all seen to make a valued contribution to the school.
Where to find out more
Education Data Requests
If you have any questions about education data then please contact us at:
Email: Requests EDK
Phone: +64 4 463 8065