Evaluation of Te Kotahitanga: 2004-2008
In 2007, Victoria University was contracted by the Ministry to produce an external evaluation of the effectiveness of Te Kotahitanga. It is the first external evaluation of Te Kotahitanga.
This is the full technical report of the evaluation of Te Kotahitanga in 22 schools from phase three and four of the programme, from 2004-2008. Substantive findings from the evaluation report concluded that Te Kotahitanga is a sound and effective process for improving classroom teaching and learning for Māori students.
Also available on Education Counts is the Te Kotahitanga Summary Report, which outlines the key findings.
Author(s): Luanna Meyer, Wally Penetito, Anne Hynds, Catherine Savage, Rawiri Hindle, and Christine Sleeter. Report for the Ministry of Education.
Date Published: August 2010
This report is available as a download (please refer to the 'Downloads' inset box). To view the individual chapters please refer to the 'Sections' inset box. For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
Chapter 1: Te Kotahitanga and the Focus of the Evaluation
The main research question for this evaluation is How well and in what ways does Te Kotahitanga work towards the goal of improving Māori student achievement? To address this question, the evaluation was designed specifically to answer the following sub-questions:
- What is the quality of the overall design, content and implementation of Te Kotahitanga?
- How valuable are the outcomes for the teachers who participate—what new knowledge, understandings and skills do they develop, and how valuable are these learnings?
- How valuable are the outcomes for Māori students, and what is the impact on other classmates/peers?
- How valuable are the outcomes for whānau?
- How beneficial (or detrimental) are the effects of Te Kotahitanga on school culture (covering any changes in formal systems and policies; informal practices, or "the way we do things around here"; and underlying beliefs, values, assumptions and attitudes)?
- What are the enablers and barriers for getting Te Kotahitanga to work most effectively?
- To what extent is Te Kotahitanga likely to work effectively in other settings and contexts? How sustainable is the initiative likely to be when ministry investment of resources is scaled back?
- What are the most critical factors in improving teacher efficacy?
It is important to qualify that this evaluation report and the findings reported here are focused on Phases 3-4 of Te Kotahitanga led by the University of Waikato research team. Prior to the preparation of this synthesis report across these two phases, the VUW evaluation research team submitted two separate interim evaluation reports in March 2009 and June 2009 that were focused on the findings from data collection at Phase 3 and Phase 4 schools, respectively. The two interim reports were prepared for the Ministry of Education and for the University of Waikato Te Kotahitanga research team, hence were available to inform the design of Te Kotahitanga for Phase 5. Thus, aspects of the Te Kotahitanga design and approach for Phase 5 differ from those used in Phases 3-4, including modifications made by the project in response to issues arising from our interim findings. A comprehensive, authoritative description of the current Te Kotahitanga model as it is being implemented in Phase 5 schools from 2010 can be found in Bishop, O'Sullivan, and Berryman (2010).
This chapter, therefore, focuses on description of Te Kotahitanga as it was implemented in Phase 3-4 schools and as evaluated at those schools and reported here.
Overview of Te Kotahitanga
Te Kotahitanga is a professional development programme designed for secondary school teachers with a focus on Years 9-10. Its purpose is to help teachers improve achievement of Māori students by focusing on relationships between themselves and the students within a cultural pedagogy of relations. It does this by implementing strategies and processes that recognise the importance of culture as found in every classroom. These include the ways in which participants relate to one another, the context within which the participants interact, the content of what is taught and learned, and the actual pedagogical act itself. Te Kotahitanga is one of a series of mainstream initiatives designed and implemented to enhance Māori student educational achievements; Māori students attending schools participating in Te Kotahitanga will have transitioned to secondary from other mainstream and/or immersion or bilingual school programmes. Hence, Te Kotahitanga is designed for mainstream secondary schools that include Māori students but are also delivering educational services to the wide range of students enrolled in New Zealand schools. The model, therefore, involves school leaders, other school personnel, and especially teachers who are themselves representative of the wide range of cultural identities found in schools, and where there may be relatively small percentages of Māori among professionals who interact with and teach Māori students in those mainstream schools.
Of 330 state and state-integrated secondary schools in New Zealand at the time, the Ministry of Education selected the first 12 Phase 3 schools based on their participation in one of its schooling improvement programmes that provided the funding source for the project. Selection of the Phase 4 schools was done collaboratively by the Waikato research team and the Ministry of Education. This selection started with an advertisement in the Gazette calling for expressions of interest from schools that required information regarding school staff, board of trustees and principal support; indication that the school's student management system could accommodate the project's needs for data; and other criteria. More than 50 schools responded, and a joint selection panel comprising project leaders along with Ministry of Education personnel then identified 21 schools invited for Phase 4 based on both the percentage of Māori students on the roll (generally higher than 20%) and geographic region (to allow the project to extend beyond the Waikato and Auckland regions where school rolls showed the highest proportion of Māori students).
Table 1 indicates the timeframe for implementation of Te Kotahitanga at the Phase 3 and Phase 4 schools. Phase 3 schools began participation in late 2003 with initial programme preparation followed by the first full year of participation and training the first teacher cohort in 2004. Phase 4 schools began participation in late 2006 with initial programme preparation followed by up to three teacher cohorts experiencing their first full training years in 2007, 2008 and 2009.
|Project Year||Phase 3 Schools (N = 12)||Phase 4 Schools (N = 21)|
2004 (years 9-10)|
2007 (years 9-10)|
2005 (years 9-11)|
2008 (years 9-11)|
2006 (years 9-12)|
2009 (years 9-12)|
2007 (years 9-13)|
2010 (years 9-13)|
Training years are those in which cohorts of teachers are using the Te Kotahitanga model and being observed for the first time. Full implementation years signify that all teachers participating in Te Kotahitanga have been trained in the model, so that all students across Years 9-10 have exposure to Te Kotahitanga trained teachers. Year 9 students in Phase 3 schools in 2006—the first year of full implementation—will not be in the Year 11-13 student cohort until 2008-2010. Year 9 students in Phase 4 schools in 2010—the first year of full implementation—will not be in the Year 11-13 student cohort until 2012-2014. These timelines are important to acknowledge in any evaluation anticipating effects on student outcomes. Because secondary school students in Years 9-10 are enrolled in different subjects across the curriculum, on any given school day a particular Māori student will be exposed to 5-6 different teachers; in the "training years", some of those teachers will be participating in Te Kotahitanga and others not. Even in the "full implementation" years, students will be exposed to various subjects where the teacher is new to the project or may have chosen not to participate at all. Hence, student exposure to the Te Kotahitanga model is difficult to quantify nor would it be possible to quantify the quality of that exposure. These are the kinds of complications that make it challenging to track student outcomes as a function of teacher professional development initiatives and which must be kept in mind in reading this report.
Te Kotahitanga as professional development
During Phases 3-4, Te Kotahitanga was implemented primarily as a professional development model designed to enhance teaching and learning towards enhancing Māori student achievement in mainstream secondary schools1.
In contrast to most professional development (PD) initiatives, the origins of and approaches taken by the project were not based primarily on existing PD theory nor were they driven by professional and adult stakeholder perspectives or conceptions of how to promote Māori student achievement and culturally responsive pedagogies in mainstream schools. Instead, the Te Kotahitanga professional development model is grounded in the voices of Māori students as they articulated what does and does not work for them in school, and how they have been victimised by teacher deficit theorising coupled with a transmission approach to teaching. The "Effective Teaching Profile" (ETP) of Te Kotahitanga came directly from the Māori student narratives; as Bishop, Berryman, Cavanaugh and Teddy (2009) explain, "the narratives [of the students] were used in the professional development part of the project to provide teachers with a vicarious means of understanding how students experience schooling in ways that they might not otherwise have access to" (p. 736).
The Te Kotahitanga professional development model links culturally relevant/relationship-based classroom pedagogy with a site-based process for working with teachers in the classroom. Implementing the Effective Teaching Profile operationalises the project's "culturally responsive pedagogy of relations" to establish "a learning context that is responsive to the culture of the child and means that learners can bring who they are to the classroom in complete safety and where their knowledge is acceptable and legitimate" (Bishop et al., 2009, p. 741).
The model reflects research on the most effective forms of professional development for teachers. Researchers have found that professional development that is most likely to have an impact on teaching is sustained over time, focuses on specific instructional strategies or content areas, involves teachers collectively rather than individually, is coherent, and uses active learning (Garet et al., 2001; Snow-Runner, 2005). Peer coaching in the classroom is emerging as an important facet of teacher professional development that is linked with improved student learning (Joyce & Showers, 2002; Neufield & Roper, 2003). The Te Kotahitanga professional model reflects this research. The Te Kotahitanga programme takes on particular significance given the growing international interest in effective professional development approaches for teachers of indigenous and other minoritised student populations in mainstream schools (Castagno & Brayboy, 2008).
Project schools begin by participating in Te Kotahitanga training delivered by the University of Waikato Te Kotahitanga research team. Over the years of the project, this training has evolved and become highly structured as well as being driven by ongoing data collection reported to the research team by the schools. The school principal, the lead facilitator and, over time, additional school personnel including the facilitation team and school leaders are expected to participate in training opportunities at both the national and regional level. During Phase 3, there was regional facilitation expertise and support provided to schools from the Waikato team, but this regional support has not been part of the Phase 4 model and was discontinued at Phase 3 schools in 2008.
The facilitation team
At each school, Te Kotahitanga activities are coordinated by a lead facilitator supported by one or more additional facilitators with the total percentage of FTE for the team determined based on student and teacher numbers. At a typical medium size-secondary school, the lead facilitator is a full-time appointment and there will be additional facilitators who are usually working part-time as Te Kotahitanga facilitators and spend their remaining time in a variety of roles; these include working with the school advisory services, as a Resource Teacher of Learning and Behaviour (RTLBs), and teaching within the school. Unless the remainder of the facilitator's role is as a teacher in the school, part-time facilitators will be spending set days within the school in that role and travelling to other schools for different work on the other days. These additional facilitators will also have varied reporting lines, thus not necessarily reporting to the principal as employees at that school: school support services advisors are employed by the university in that region running the advisory service, whereas RTLBs are likely to be employed by a school cluster.
The teams comprise a mix of expertise, experience and credibility as facilitators for Te Kotahitanga. Lead facilitators are generally appointed for their strengths relative to key aspects of Te Kotahitanga and nearly all will have themselves been master teachers respected by their peers. The majority of the lead facilitators are Māori but a few are New Zealand European in ethnic origin but genuinely bicultural as professional educators. Their background as teachers will also dictate their curriculum areas, so that any given lead facilitator is likely to have taught in only 2-3 subjects at secondary level. Remaining members of the facilitation team are in theory appointed with a view towards expanding subject area coverage across the curriculum as well as for their personal and professional understandings of Māori education and Māori culture. The majority of facilitators who are not lead facilitators are not themselves Māori nor bicultural, but they are all selected for their commitment to Te Kotahitanga and based on their credibility and experiences relevant to the role.
Teacher cohorts in the school
Teachers begin the programme in groups of 30 from the same school, and throughout, they have regular meetings with other teachers to analyse their own progress in teaching Māori students effectively. The effective teaching profile and classroom coaching focus specifically on teaching strategies and ways teachers position themselves in relationship to their students. Facilitators who are or have been excellent classroom teachers work directly with teachers in the classroom to improve their practice.
The professional development model
Bishop, Berryman, Cavanagh, and Teddy (2007) describe the separate components of the professional development programme as experienced by teachers:
- The initial induction workshop (hui) introducing Te Kotahitanga and the model of a culturally responsive pedagogy of relations
- Structured classroom observations followed by feedback sessions with teachers
- Co-construction meetings where teacher teams problem-solve collaboratively based on observational and student outcomes data
- Specific shadow-coaching sessions for individualised teacher professional development.
More detail regarding these components is provided in the next section.
The Te Kotahitanga professional development model begins with a three-day hui on the marae. At that time, teachers meet with the Waikato team; read parts of Culture Speaks (the narratives of experience that ground the programme); discuss the relationship between deficit theorising, pedagogy, and Māori student achievement; and learn about the process of the professional development programme itself.
Classroom observations with feedback and shadow-coaching sessions
Observations are carried out in the schools once per term for each teacher in the classroom, involving a facilitator who observes a teacher's classroom using an observation tool that is designed to capture various aspects of the effective teaching profile. Following the observation, the facilitator and teacher meet to discuss the lesson and findings recorded on the observation tool. In the post-observation conferences, teachers learn to analyse the observation data for themselves, and to reflect on it with the facilitator; both then establish goals for future growth that will form the basis for subsequent instructional planning, observation, and feedback.
The professional development model also includes shadow coaching, which follows a similar process except that the focus of the observation is on something specific that the teacher would like help with and the facilitator gives the teacher input throughout the lesson rather than providing feedback, privately, after the students have finished the lesson.
In addition to the individual teacher observations, co-construction meetings are also central to the Te Kotahitanga model. These meetings normally involve small numbers of teachers who teach the same students but in different subject areas, meeting once every month or two to share concerns and strategies for improving Māori student achievement. The meeting is led by a facilitator who has observed the teachers in the classroom. The intended focus of co-construction groups is the analysis of a teaching-learning problem shared by the teaching team, using some form of evidence of learning and then developing a group goal. At a subsequent meeting, they analyse what they have been doing to improve their practice relative to that goal. For co-construction meetings to work as intended, teacher participants require relevant evidence from both the observations as well as regarding student achievement and other achievement-related data such as attendance, disciplinary events, and so on.
Sustainability of Te Kotahitanga
Te Kotahitanga as a programme is long-term, and Phase 3-4 schools were assured funding for a six year time period with the expectation that the programme become self-sustaining over time. In theory, sustainability would evolve as a function of the expertise and commitment of school personnel. This would occur directly through the PD initiative providing teachers with the skills and understandings underpinning the Effective Teaching Profile and also indirectly through higher expectations for Māori across the school that would be reflected in school-wide accountability for Māori student achievement and other positive outcomes for Māori students.
Te Kotahitanga "uses GPILSEO as a mnemonic device to aid in referencing" (Bishop, Berryman, Cavanagh, & Teddy, 2007, p. 195) so that the reforms flowing from the project are sustainable. GPILSEO requires a school-wide Goal; new Pedagogy; new Institutions and structures for support: Leadership that is responsive, transformative, proactive and distributed; strategies for Spreading reform; Evaluating progress; and establishing school Ownership of the reform. Key school personnel such as principals, deputy principals, deans, and heads of departments were not themselves the primary focus of professional development, but commitments made at the school level through the principal and the Board of Trustees were expected to have an overall impact on how Te Kotahitanga schools operate and how they respect and respond to Māori students and their communities. And though the Board of Trustees must approve the school's participation in the project and Māori communities and whānau receive information about the project, their involvement and participation are not formal components in the programme.
Thus, it is important to qualify that the focus of Te Kotahitanga during Phases 3-4 has been on a generation of the programme that was primarily teacher professional development, rather than overall school or systemic reform or on the development of other aspects such as enhanced networks between the school and its Māori community. The evaluation results in this report are focused on Phases 3-4 of Te Kotahitanga, utilising data gathered and analyses conducted in 2008-2009. The evaluation does not encompass the next generation of Te Kotahitanga as it later evolved in Phase 5, which will commence in 2010 with the new cohort of schools. Phase 5 is described as having a greater focus on overall school reform and factors such as accountability, systematic use of evidence on student outcomes at the school and teaching team levels, and relationships between school and community; this includes highlighting the GPILSEO framework at school level, discussed in more detail in Chapter 6 (Bishop et al., 2010). Nevertheless, changes made to the programme in the Phase 5 schools have been informed and influenced by the findings emerging from implementation in Phases 3-4, including the findings of this evaluation of Te Kotahitanga as primarily teacher professional development.
In order to address How well and in what ways does Te Kotahitanga work towards the goal of improving Māori student achievement, our evaluation addresses both the strengths as well as the limitations revealed by findings about programme impact and sustainability over time.
Ministry of Education Note:
The evaluations of both Phase 3 and Phase 4 synthesised in this final report were used formatively in the design of the new implementation phase of Te Kotahitanga. Phase 5 is therefore significantly different in design from Phases 3 and 4. Many of the modifications are in response to recommendations made within this report. They have been made through an iterative process of collaboration and critique between the evaluation team, the Ministry of Education and the University of Waikato Te Kotahitanga team.
Te Kotahitanga in Phase 5 has evolved from being a purely professional development programme aimed at improving teacher practice, to a wider programme that sets out to achieve whole-school change in addition to improved teacher practice. A stronger emphasis is now placed on school leadership and evidence to inform teaching practice. These intervention points recognise the importance of influencing both policy and practice to affect sustainable, whole-school change.
In the Phase 5 model, leaders in each of the participating schools are provided with specific support. This additional support is provided to enable leaders to increase their pedagogical leadership and establish whole-school policy that supports Māori student achievement in a culturally responsive environment.
The importance of using evidence to inform practice in Phase 5 of Te Kotahitanga is crucial. It is intended that systematic measures of student outcomes are reported on an annual basis to the Ministry, the school's board of trustees, students and families. While there are some very strong markers of success within this evaluation, it is clear that schools continue to need support to develop greater capability and capacity in this area. Helping schools to use evidence to inform teaching and learning, particularly as it relates to Māori, is a strong focus in the Phase 5 implementation model.
There are also some important changes to the design of the teacher professional development programme itself. The evaluation identifies some of the issues in measuring the success of a year nine and year ten teacher development programme by NCEA results. In Phase 5, Te Kotahitanga now includes all teachers in a three- year cycle of professional development. As years 9-10 remain a priority group, teachers in this area will begin the process first.
The evaluation also identified that the professional development programme did not differentiate between teachers who were working at 'High' 'Moderate' or 'Low' implementation in Phases 3 and 4. Undoubtedly, teachers, like their students, have different learning needs. In response to this recommendation, Phase 5 has more specific, targeted professional development to better cater for teachers' needs. New descriptors within the Effective Teaching Profile provide the means for an in-school team to differentiate the needs of their staff, and to respond in ways that include using high implementers as shadow coaches.
In summary, this evaluation report, has already played an important role in informing the design of Te Kotahitanga. The professional development model of phase five is significantly different in design, in response to recommendations that are made within this report. The University of Waikato Te Kotahitanga team and the Ministry of Education are anticipating that these design changes within Te Kotahitanga will further enable Māori to enjoy education success, as Māori, in English-medium schools. The Ministry of Education continues to support the philosophies that underpin Te Kotahitanga. It has begun to explore ways in which wider gains can be made from the knowledge gained within the programme.