Evaluation of Te Kotahitanga: 2004-2008
In 2007, Victoria University was contracted by the Ministry to produce an external evaluation of the effectiveness of Te Kotahitanga. It is the first external evaluation of Te Kotahitanga.
This is the full technical report of the evaluation of Te Kotahitanga in 22 schools from phase three and four of the programme, from 2004-2008. Substantive findings from the evaluation report concluded that Te Kotahitanga is a sound and effective process for improving classroom teaching and learning for Māori students.
Also available on Education Counts is the Te Kotahitanga Summary Report, which outlines the key findings.
Author(s): Luanna Meyer, Wally Penetito, Anne Hynds, Catherine Savage, Rawiri Hindle, and Christine Sleeter. Report for the Ministry of Education.
Date Published: August 2010
This report is available as a download (please refer to the 'Downloads' inset box). To view the individual chapters please refer to the 'Sections' inset box. For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
Chapter 2: Evaluation Research Method
Our bicultural evaluation research team was contracted by the Ministry of Education to carry out a comprehensive evaluation of the Te Kotahitanga Programme, led by the University of Waikato Te Kotahitanga Research Project and implemented in 33 secondary schools on the North Island (12 Phase 3 schools and 21 Phase 4 schools) at the time of the evaluation. For this evaluation research, we gathered data at all 12 Phase 3 schools and at 10 Phase 4 schools selected as representative of the 21 schools.
The overall aim of Te Kotahitanga has been to investigate effective teacher professional development strategies leading to culturally responsive pedagogies and improvements in the educational achievement of Māori students in mainstream secondary school classrooms in New Zealand. Thus, Te Kotahitanga focuses on professional development support for teachers across the curriculum to build more effective teaching and learning relationships with Year 9-10 Māori students in secondary classrooms and improving Māori student learning outcomes.
The primary research question for this evaluation is: How well and in what ways does Te Kotahitanga work towards the goal of improving Māori student achievement?
To address this overarching evaluation question, we developed the evaluation research design approach; specific data collection procedures including those for documents review, achievement data records, interviews, and classroom observations; and sub-sets of evaluation questions appropriate for each aspect and respondent group. These steps were carried out following the specifications of the original Ministry of Education request for proposals, our proposal, ethics review, and in consultation with members of a national advisory group, key researchers from the Waikato team, and senior Ministry of Education personnel with responsibility for the project.
This final report synthesises the information previously reported in two interim evaluation reports based on findings from evaluation activities at the two sets of schools. The first interim report (March 2009) focused on the 12 Phase 3 schools that began participation in Te Kotahitanga late in 2003 with implementation phased in across 2004-2005, while the second interim report (June 2009) focused on 10 of the 21 Phase 4 schools that began participation in Te Kotahitanga in 2007.
This final synthesis report highlights findings based on evidence gathered at the 22 schools. We also summarise issues and recommendations for consideration in ongoing and future planning that we consider relevant to Te Kotahitanga and related initiatives directed to the development of culturally responsive pedagogies and enhancing Māori student social and achievement outcomes.
Overview of the evaluation methodology
The evaluation project was mixed-methods, involving both quantitative and qualitative methods comprising multiple data sources that informed one another and allowed triangulation of emerging findings (Creswell, 2009). Comprehensive data were gathered from the 22 schools during school visits and from other sources, including participant perspectives, review of individual school reports, and student outcome data including achievement results.
To investigate aspects of the teacher professional development model and its impact on classroom teaching and learning interactions, we conducted in vivo classroom observations following a detailed observation protocol to allow analyses across the data. We also observed teacher professional development sessions and co-construction meetings and interviewed a sample of teachers following these sessions in order to clarify and elaborate agenda and issues. To investigate perspectives of key stakeholders and constituents regarding how well and in what ways Te Kotahitanga works towards the goal of improving student achievement, we conducted individual and small group interviews with teachers, principals, deputy principals, deans, heads of departments, Te Kotahitanga facilitators, Board of Trustees chairpersons, and focus groups of whānau and the students themselves.
To investigate student outcomes associated with Te Kotahitanga implementation, we utilised multiple data sources encompassing three broad categories: student achievement, student behaviour, and student attitudes about their learning. Sources of evidence on achievement and behavior included formal assessment information as well as school reports and interviews with various constituent groups. Formal achievement results sourced by our project included Year 11-13 NCEA achievement data for the original 12 schools from the NZQA and the Ministry of Education Benchmark Indicators databases.
Sources of information on student attitudes about their learning included interviews with school personnel and whānau/family as well as what the students had to say about their learning, about the project, and about being Māori in schools. To investigate Māori student perspectives on Te Kotahitanga and on being Māori in their schools, we interviewed focus groups of Māori students from Years 10 to 13.
We reviewed available data for student social and educational outcomes in Years 9-10 that could show the immediate effects of Te Kotahitanga, including information about student attendance, retention, percentage representation in different ability bands for core subjects, percentage representation in the school's disciplinary statistics, and preliminary achievement assessments such as asTTle. We reviewed a large sample of Te Kotahitanga milestone reports submitted by schools to the Ministry of Education; available school data on student outcomes including Year 9-10 assessments and behavioural data; and interviewed key personnel from the University of Waikato Te Kotahitanga Research Team.
For the Phase 3 schools, we compared longer term Year 11-13 student outcomes in the senior school—where the project is not directly involved—at Te Kotahitanga schools with student achievement at a matched group of comparison schools. These data were obtained from the Ministry of Education annual benchmark reports and from records of the NZQA on Year 11-13 NCEA performance at Levels 1, 2 and 3. The NCEA achievement data analyses include the percentage of Year 9 students who attained NCEA Level 1 two years later when they were in Year 11 from 2004-2008; literacy and numeracy attainment; credits attained in Year 11 in different subject areas; and the percentage of students attaining University Entrance. Where possible, these analyses are disaggregated for Māori learners; in some cases, only the total school population can be reported where the source databases are not disaggregated by ethnicity. We also summarised the percentage of Māori and New Zealand European school leavers who had attained at least a Year 12 qualification for the "baseline" years 2004-2007; these data are not included in the report because 2008 results—the first year in which it would be reasonable to evaluate this outcome for the Phase 3 schools—were not included in the Ministry of Education benchmark report within the timeframe available to us.
Observations, interviews and focus groups were conducted individually or in pairs by members of the research team comprising experienced Māori and non-Māori researchers using specific protocols that had been piloted. More information regarding the observation and interview protocols and questions are provided in the relevant sections of the report, along with information on how data were analysed using either SPSS (for quantitative student achievement data) or NVivo (for qualitative data from the interview transcriptions). Transcriptions of interview recordings and observation notes were checked by the researchers, and coding was carried out by trained and experienced coders familiar with such data sets according to keywords and phrases. These categories were identified based on reviews of selected interview transcripts followed by research team group discussions, then cross-checked as data were coded and analysed throughout coding to allow modifications and additions. Specific analysis procedures are described throughout the report for particular data sources.
Bicultural dimension of the evaluation
Given the focus of this evaluation on the nature of teaching and learning activities for Māori students in mainstream schools, it was crucial that this evaluation research be carried out in adherence with the principles of biculturalism and that our team encompassed cultural expertise as well as other expertise required for evaluation research. To achieve this, seven key points are pertinent:
- The cultural composition of our team includes Māori and non-Māori members both within the VUW research team as well as being represented by additional international experts experienced in cultural pedagogies and independent Māori researchers contracted in the regions of the schools participating in the project;
- Three of the six Māori research team members were involved only in the data collection on site in schools, whereas the three Māori research team members at VUW took part in every aspect of the evaluation;
- The research team affirmed Māori cultural protocols at every opportunity during school visits. These included formal powhiri and less formal elements of mihimihi, hongi me te hariru, waiata, and karakia when appropriate;
- Whānau group meetings were informal but included whakatau, karakia, sharing of kai, and poroporoaki. Each of these meetings was led by one of the Māori researchers with another member of the research team responsible for taking notes. Meetings with students were more formal and constrained by time, but also included Māori cultural elements whenever possible. We checked back with whānau, students and others to confirm the accuracy of our notes with what was said, and more detail on these processes is provided in this chapter;
- There were occasions, though rare, when Māori teachers and whānau members felt more comfortable commenting in te reo Māori. They were delighted when there were researchers who could reciprocate in kind;
- The mixed method approach of quantitative and qualitative research was generally welcomed. Teachers in particular were familiar with the data collection and analyses of formal school assessments but could also appreciate the need to elicit other kinds of data such as that derived from observations and interviews. On every occasion, Māori teachers, students and whānau members were enthusiastic about sharing with us what they knew about Te Kotahitanga, what they knew about constraints in the schools, and what they thought was needed to address issues of Māori schooling achievements.
- Finally, our national advisory group included Māori knowledge, expertise and experience that provided further input and fresh eyes in reviewing key aspects of the evaluation plan and findings.
Ethics review and approval
As this research involves direct contact with and gathering data from and about school personnel, whānau, and school-aged students, appropriate and rigorous procedures for participant consent, data collection, and protection of privacy and confidentiality were followed. These consent protocols, information sheets, and letters distributed to participants to gain their consent are included in Appendix 2. Participation in observations and interviews was voluntary, according to the evaluation research requirements.
The proposed data collection approach, data collection measures and questions, and processes for obtaining consent and protecting the privacy of natural persons (and the identities of the individual schools) were comprehensively reviewed and fully approved by the VUW Human Ethics Committee. Our ethics protocols guaranteed confidentiality to individual participants from the schools, including that their identity would neither be revealed in our reports nor would their schools be able to associate data with particular persons. Even for very small groups where there is a risk that someone's privacy would not be protected (e.g., a BoT chair or the principal), we have attempted throughout the report to disguise those identities so that they cannot be traced to individual schools and thus identified to others.
All data are kept according to strict ethical guidelines in locked and password-protected files at the Jessie Hetherington Centre for Educational Research at VUW. These will be kept for a proscribed period of time as required, and raw data will be destroyed after 5-10 years depending on the nature of the data.
Data collection for the evaluation
Selection of schools
Our sample for gathering data comprised all 12 Phase 3 and 10 of the 21 Phase 4 Te Kotahitanga schools in 2008 to gather representative data on site at each school systematically. We also visited 10 of the newly selected Phase 5 schools in October 2009 to conduct observations in classrooms focused on four compulsory subjects to provide a comparison sample of non-Te Kotahitanga trained teachers. In all, we gathered comprehensive data at 22 project schools (Phases 3-4) and selected comparison data at 10 additional pre-implementation schools (Phase 5).
With regard to the selection of 10 of the Phase 4 schools for our sample, it was not feasible to site visit all 21 Phase 4 schools in order to replicate the methodology used to collect data in the 12 Phase 3 schools. We judged that carrying out parallel-intensity data collection and analyses at a non-biased sample of 10 of the Phase 4 schools—approximately half—would be preferable to a more cursory review across all schools. By replicating Phase 3 procedures, we would also have a comparable data set for comparison across the two phases of Te Kotahitanga to examine for recency and/or sustainability effects.
There are slight regional variations across the Phase 3 and Phase 4 schools. Phase 3 schools are predominantly located in Northland, Auckland, and rural regions of the North Island. Phase 4 schools were from three geographic regions—Auckland, Bay of Plenty and the Waikato. Like the Phase 3 schools, the Phase 4 evaluation sample included both coed (N=7) and single-sex (N=3) schools, and schools were selected to include those with large, medium and small sized student rolls as with the Phase 3 total sample. Given that several schools from Northland had been involved in the Phase 3 evaluation, we did not include additional Northland Phase 4 schools in the evaluation; rural or small towns represented by Phase 4 schools are in the Bay of Plenty. Both Phase 3 and Phase 4 samples include several Auckland schools as a major region with large percentages of Māori students in mainstream schools. Phase 3 did not include schools from the Waikato, and we included Phase 4 Waikato schools given that this area also enrols significant numbers of Māori in mainstream schools. Similar decision processes were used to select the ten Phase 5 schools that provided baseline data.
Thus, school selection was not biased with respect to factors considered likely to influence results. Initially, 11 Phase 4 schools were approached about participation on the assumption that we could do all 11, but that if one declined, we would still have a reasonable sample of 10 schools. Indeed, one school declined participation citing an upcoming ERO report during the same timeframe along with other pressures at the time.2
For the Phase 3 report, all 12 schools were involved hence the identities of the schools is known. While the identities of the 21 Phase 4 Te Kotahitanga schools are also public knowledge, the identities of the 10 schools that were involved in this evaluation will be kept confidential by our project though each school's identity is of course known to participants at the schools. Similarly, while the total sample of Phase 5 schools is public knowledge, the 10 participating in classroom observations is known only to the evaluation project and to the schools.
School site visits
Site visits were conducted at the schools with from 2-5 research team members visiting each school for between 1-3 days depending on school size. More detailed biographies are provided for all members of the evaluation research team in Appendix 8. School site visits were scheduled by geographic regions including schools in Northland, Auckland city, Auckland suburbs, Waikato, and Bay of Plenty; urban, suburban and rural schools are represented in the sample of school visits.
Data were gathered on site at 11 of the 12 Phase 3 schools in March-April 2008; at the twelfth school, the majority of the data collection occurred in October 2008 (during the time period for Phase 4 data collection). This was because that school's lead facilitator position had been vacant at the start of the year and it was felt that the new lead facilitator should be in the role for at least one full term prior to our site visit. We did visit this school for selected interviews (e.g., the principal) for one day during the timeframe for the other school visits, though most interviews were carried out during the October visit.
As is appropriate for evaluation research to investigate the impact of a particular programme, purposive sampling was used to identify samples for data collection (Kline, 2009; McMillan, 1996; Shadish, Cook, & Campbell, 2001). Participation in the school site visits, interviews, and observations carried out by our evaluation team was voluntary to individual participants but not to schools as they were expected to participate in the evaluation as part of their participation in Te Kotahitanga. All Phase 3 schools agreed to participation. One of the Phase 4 schools originally approached by our team declined participation due to other commitments that meant the visit timeframe could not be accommodated without serious compromise to either our visit or that associated with another major activity; a similar school fitting our geographic visit schedule was then approached and agreed. Similarly, individual participation in interviews and/or observations was through informed consent and thus voluntary. The actual participants were recruited by the school's facilitation team working in concert with the principal. While this could have resulted in pressure to participate by school leaders, no person at any of the schools we visited indicated to us that they had been unduly pressured or forced to agree to participation. Indeed, participants seemed overwhelmingly positive about the opportunity to participate.
The large numbers of observations and interviews conducted across the 22 schools also provide multiple replications for data collection, thus enhancing the external validity of findings (Kline, 2009). Finally, the validity of data from naturalistic observations and independent interviews is also supported by low-inference data collection procedures used by the evaluation project and participant familiarity with similar processes such as the facilitator observations conducted each term (Cozby, 2009; Huck & Cormier, 1996; McMillan, 1996). The quasi-experimental research design and sampling procedures used in our evaluation research are characteristic of educational research where random assignment, non-voluntary participation, and deception regarding the purpose of data collection are neither possible nor would they be desirable. The procedures followed here represent best practice in evaluation research of this nature, reducing as much as possible the likelihood that these data are somehow non-representative of the broad range of stakeholder perspectives or with regard to classroom teaching and learning activities.
Once the visit and dates had been agreed with the school principal in consultation with the school's lead facilitator, subsequent details for the school site visit were negotiated by the project director and the lead facilitator. We provided the school with a template for the scheduled dates and that particular school's daily schedule that was filled in by us to reflect a target number of observations in different subject areas and both individual and focus group interviews; the schedule also included after-school session times for meetings with the BoT chair and whānau. The schedule we provided indicated the number of researchers who would be gathering data at the school on the day/s, with pairings included for designated time period whenever two researchers were required for focus group interviews. A sample schedule is provided in Appendix 4. The lead facilitator worked within the school to accommodate our visit by filling in the template with as close to the desired number and range of interviews and observations, and it was agreed that our initial template was intended to be a guide such that the school could make various changes while trying to accommodate the basic framework and numbers of types of sessions. All within-school negotiations to solicit observations and interviews on a voluntary basis were done by the lead facilitator—generally working with the principal or principal designee (one of the deputy principals)—to seek voluntary participation from teachers, students, family members and others, who were under no obligation to agree.
The final schedule was provided to the evaluation project director at least several days prior to the visit to allow checking that required activities were included (e.g., an interview with the principal, observations across a range of subjects) and to finalise the research team assignments for the visit (e.g., who would do particular interviews and observations). For the 3-4 weeks of scheduled site visits, the research team was split into two teams according to the original schedule of activities; at each school, approximately half the team was Māori, fluency in Māori was represented by at least one researcher, and a Māori male included so that we could participate appropriately in cultural protocols. With rare exception, focus group interviews with Māori whānau and student were led by a Māori researcher as facilitator.
Some of the schools we visited welcomed our research team on campus with a pōwhiri, and we were prepared with our response including waiata.3
Māori student experiences in mainstream classrooms and schools are the focus of Te Kotahitanga. The teaching and learning activities in the classroom are the authentic measure of shifts in teachers' view of students away from deficit to strengths-based perspectives and the transformation of classroom instruction from transmission models to more discursive, interactive models building on student culture, experiences and understandings.
The research team observed 336 lessons across the Year 9-10 curriculum at 22 Phase 3-4 schools in 2008. Participation in the observations carried out by our evaluation team was voluntary, and teachers were recruited by the school's lead facilitator to participate in the observations. We requested that the teachers we observe would include both male and female teachers of diverse ethnicities including Māori and non-Māori, a range of subjects across Years 9-10, different levels of teaching experience, and at different ages. With the exception of a very small number of double class periods, virtually all our observations covered a full class period ranging from 45-60 minutes depending on the individual schools' daily schedule. Observations were alternated with interviews as much as was practicable, so that the observer could be in the classroom at the start of the lesson and remain until the students had left (i.e., for the entire class period of that lesson).
Of the 204 observations in March-April 2008 at the 12 Phase 3 schools, we observed from no fewer than 5 at the smallest school to a maximum of 34 at the largest school. The largest number of observations (nearly 100 of the total) occurred in core subjects such as English, mathematics, and science, and the smallest number of observations (no fewer than 4 observations each) were carried out in elective subjects such as Japanese, te reo Māori, dance, and health.
Of 132 classroom observations in October 2008 at 10 Phase 4 schools, we observed from no fewer than 5 at the smallest schools to a maximum of 29 at two of the largest schools. Approximately 60% of these observations occurred in the core subjects of English, mathematics, science, and social studies, and the smallest number of observations occurred in elective subjects such as Japanese, te reo Māori, technology, and drama.
The original research design had posited a comparison of classrooms using Te Kotahitanga versus those not using Te Kotahitanga, thus we requested that some of the teachers we observed be non-participants in the project, invited as experienced (not first year/beginning) teachers who may not be participating because they were new to the school and not yet trained or had chosen for whatever reason not to participate. We emphasised that we did not want the non-participant group to be selected or seen as teachers opposed to the programme or beginning teachers, which would bias our data in unknown ways. The non-participant group observed was disproportionately small at each school and would not have provided a "random" sample, given that teacher consent to be observed would likely bias the data. Also, non-Te Kotahitanga teachers at the Phase 3 schools were primarily new to the school or even new to teaching, hence other factors influencing their practice could affect the validity of any comparison with experienced teachers; we had asked that beginning teachers not be included in the sample for our observations. Any teacher at these schools could likely be influenced and affected in unknown ways by a model operating within their schools for more than four years, whereas a comparison would require confidence that the contrast group is naive with respect to the model and hasn't been influenced accordingly.
At the Phase 4 schools, the overwhelming majority of teachers whom we observed were Te Kotahitanga trained participants at the time of our visits (N=116). These schools were late in the second training year of the project, with a third additional training year to follow. Hence, in theory, these schools could provide a comparison group of non-Te Kotahitanga teachers. The non-participants whom we observed (N=15) fell into one of three groups: (a) not yet trained but scheduled for future training (as transfers from another school, part of a final cohort to be trained, or new hires); (b) not trained because they had chosen not to participate in Te Kotahitanga; or (c) had been trained in Te Kotahitanga previously but had withdrawn from the programme and were no longer participating in the PD activities. While in principle this could allow some comparison across the two groups, there are design issues that make this comparison questionable: (a) the two groups differ greatly in size, and the group of non-participants is so small that it can be questioned whether this is a representative group; (b) the small non-participant group also represents a select group of teachers who volunteered to be observed, who may differ in ways biasing the findings from those who declined being observed; and (c) even though the participation period was shorter than in Phase 3 schools, Te Kotahitanga is nevertheless intended to be a school-wide initiative that could have school-wide impact, including influencing teachers prior to being involved formally in the model.
Thus, we judged that a comparison of Te Kotahitanga with non-Te Kotahitanga teachers would be inappropriate using this small sample of teachers from the Phase 4 schools. With the support of the University of Waikato Research Team, we approached 12 of the newly selected Phase 5 Te Kotahitanga schools in September 2009 to carry out observations in October that could provide us with a comparison sample. Ten of these schools agreed providing an additional comparison sample of 98 observations; two schools declined, indicating non-project related reasons for being unable to accommodate our visit during the required timeframe (e.g., a scheduled ERO visit). The comparison design is quasi-experimental given that schools were not randomly assigned to the three cohort groups (3, 4, or 5). Nevertheless, 10 schools contributing 98 classroom observations provide a sizeable dataset that is valid for comparison purposes. These data also provide Te Kotahitanga with baseline, pre-intervention data for future comparisons to evaluate effectiveness of future implementation of project activities and professional development to support Māori student achievement.
A classroom observation data sheet was developed by core project personnel including the following information:
- Basic demographic information including an observation record number (assigned at the end of observations prior to coding); school; teacher name/ethnicity/Te Kotahitanga participation; class level; number of students including, if available, numbers of Māori and non-Māori; curriculum area; the lesson topic; name of the observer; and date of the observation
- Room environment: the observer drew a diagram of the classroom including furniture, seating, whiteboards, location of materials, teacher positioning, student seating/grouping, visual display on the board of aims for the day's lesson, and where the observer was situated. Teacher movement during the period was recorded (e.g., movement from "front and centre" to groups). Space was also provided to include description of visuals related to Māori culture and/or Māori icons (required if present) plus a record of classroom changes and/or comments regarding teacher position and movement.
- Lesson Narratives: a running record was made of the first and final five minutes of the lesson, including how the teacher greeted students, whether and how expectations were set, references to Māori culture/names etc, and how the lesson was concluded including checks for student understanding of the learning outcomes. Space was also provided to record Māori curriculum content (if evidence) including use of Māori intellectual knowledge in the substance of the curriculum at any time during the lesson.
- Effective Teaching Profile (ETP): Space provided to record evidence and examples of the six major dimensions comprising Manākitanga (caring for students as culturally located individuals); Mana motuhake (high expectations for learning); Whakapiringatanga (managing the classroom for learning); Wānanga (discursive teaching practices and student-student learning interactions); Ako (range of strategies to facilitate learning); and Kotahitanga (promote, monitor and reflect on learning outcomes with students).
- Teaching and Learning Types: After each 10 minute interval, all types observed during that interval were ticked including teacher presentations with different types of questions, group work, individual seatwork, project activities, student-led presentations, and non-academic and transition times.
The observation data sheet is included in Appendix 6 along with instructions for recording exemplars of the Effective Teaching Profile (Appendix 5). The observation form was trialed through practice observations carried out by members of the team independently coding video samples of classroom lessons and through in vivo observation of a social studies lesson in another educational setting (not one of the project schools). Team members compared and discussed their results following observations to reach consensus on procedures for future observations. A formal training session was conducted with the observation team as a whole prior to the first day of observations during school visits, and periodic de-briefs discussions were held away from the schools at the end of the day during the time of school visits.
Individual interviews were conducted in a private space involving only the interviewee and the researcher conducting the interview. All interviews were digitally recorded using high quality, small digital recorders generally positioned on a table or chair close to both persons; note-taking was not done so that the flow of conversation would be personal and uninterrupted. Individual interviewees were shown the full list of questions at the time of the interview, and a set of indicative questions was made available for use by the lead facilitator to share with interviewees in advance of the interviews; most interviewees had not seen this list in advance, but they indicated knowledge of the evaluation focus and they had either previously signed consent or did so at the time of the interview. The choice of interviewer was influenced by scheduling logistics, but whenever possible certain interviews were carried out by particular members of the research team (e.g., most principal interviews were done by one of the two project co-directors, and Māori whānau and student interviews involved at least one Māori interviewer).
Focus group interviews were also held in a location that was separate from other activities; these were conducted by two researchers using note-taking rather than digital recording. Following introductions, one researcher served as facilitator to introduce the questions and would begin by reading out all the 4-5 focus group questions to the group then returning to the questions one at a time to allow for group responses. This group facilitator gave full attention to group responses including making decisions along the way regarding the need for probe questions or examples for clarification. The second researcher assumed a listening role that was focused on taking detailed notes to record virtually all verbal responses in writing. Immediately after the group indicated that they had made all comments considered relevant to each question or issue, the note-taker read out the recorded responses to the group to allow for additions and edits and to check for accuracy. Changes and additions were then made at that time according to input from the group. This process encourages focus groups to take an active role in listening to input from everyone in the group (one at a time rather than speaking all at once) and ensures that all voices are heard rather than allowing domination by a one or a few members. Just as importantly, it has an advantage over digital recording followed by transcription in providing immediate member-checking of the validity and reliability of the information recorded by the researcher/s, and participants commented favourably on the process and the accuracy of our notes. Pairings of researchers conducting the focus groups was influenced by scheduling logistics (e.g., ensuring that one researcher wasn't taking notes—an intensive task—for two back-to-back sessions) as well as other key factors such as ensuring that all Māori student and whānau focus groups were led by a Māori facilitator to use te reo and adhere to Māori cultural protocols.
Teachers are central to the experiences of both Māori and non-Māori students in mainstream schools. The main purpose of the Te Kotahitanga project is to shift teachers' view of students away from a deficit view, and to shift classroom instruction from a transmission model to a more discursive, interactive model. To explore the impact of Te Kotahitanga on teachers and teaching at schools, we conducted individual observations of full-period classroom lessons, teacher professional development sessions, and individual and focus group interviews with teachers. Data collection for each aspect was guided by the overall evaluation research questions with a particular focus on teacher and teaching aspects as well as the dimensions of the Effective Teaching Profile (ETP) expected to be reflected by teachers.
As part of our data collection to investigate how teachers implemented Te Kotahitanga practices in their teaching and student learning across subjects, we interviewed 85 teachers in the 12 Phase 3 schools and 65 teachers in the 10 Phase 4 schools, for a total of 150 teacher interviews.
Teacher interviews at the original Phase 3 schools were guided by the three main evaluation questions (full interview protocols are available in Appendix 3):
- To what extent do teachers believe Te Kotahitanga helps them to develop their vision for education outcomes for Māori students and their sense of agency for improving Māori student achievement?
- To what extent do teachers believe Te Kotahitanga is helping them improve their teaching of Māori students, and of students more generally? What has been its impact?
- How do teachers evaluate the various elements of the Te Kotahitanga professional development process and program?
Each of these explicit questions was followed by probe questions as needed to pursue issues requiring further clarification, explanation and/or examples. Rather than following exactly the same set of questions for our teacher interviews at the Phase 4 schools later in the year, we revised the set of questions based on our analysis of the Phase 3 interview data in order to focus on and elaborate with respect to three key issues emerging from the earlier interviews:
- Teacher expectations for Māori students
- Evidence of enhanced learning outcomes for students
- Teacher knowledge and understandings of culturally responsive practice in teaching and learning.
At all schools, individual interviews were conducted by one of the researchers at a private place in the school setting during the scheduled site visit; however, a small number of teachers (3) asked that they be interviewed off-site and/or at another time rather than at the school during the visits. We granted these requests and did not pursue the reasons for the request.
At the Phase 4 schools, focus group interviews with 2 or more teachers were also conducted in October 2008 in addition to individual interviews. These focus group interviews covered five main (revised set of) questions specifically to ascertain teachers' knowledge and understandings of culturally responsive pedagogies:
- What do you think is most important for teachers to do to promote Māori student achievement? Why?
- How can you tell if what a teacher is doing is working?
- What does it mean to you to enable students to learn 'as Māori' and to be themselves 'as Māori'?
- Can you give an example of "caring for students as culturally located individuals"?
- How is Te Kotahitanga different from good teaching generally?
These questions were shared with the teachers at the start of the group meeting, then returned to one by one for comment from the group. Each question could also be followed by probe questions for elaboration, clarification, explanation or to provide examples. In all interviews, teachers were encouraged to add any comments they wished to make about teaching using Te Kotahitanga approaches, strategies to enhance Māori achievement, caring for Māori students as learners, and other issues.
Principal and school leader interviews
Twenty of the 22 school principals were interviewed individually, generally in their offices, by one of the senior research team members based at VUW. In addition, we sought interviews at each school with one or more Deputy Principals (most schools will have 2 or more DPs with particular duties), one or more Deans, and at least 2-3 Heads of Departments (who are also teachers). In all, 60 of these additional school leaders (19 Heads of Departments, 19 DPs and 22 Deans) were interviewed either individually or in small groups of 2-3 by one of our researchers. All interviews were recorded digitally for later transcription.
One principal was away during the data collection timeframe (we did interview the Deputy Principal who was Acting Principal), and one additional interview of another principal was accidentally deleted from the file. We reviewed our notes from the individual schools' visits by the research team and judged that it was unlikely that information from the remaining 2 principals would have differed significantly from the 10 available interviews, so these were not rescheduled.
The full list of interview questions addressed to principals, deputy principals and other school leaders is included in Appendix 3. In general, questions for the principals and deputy principals focused on issues and considerations arising from the project's main research questions, including specific topics such as:
- How the principles of Te Kotahitanga are reflected in his/her role as principal, how the principal supports Te Kotahitanga in the school, how the school strives to affirm students' identities as Māori and what the challenges are for his/her school in implementation
- Impressions of the quality of the Te Kotahitanga professional development process, content and implementation
- Ways in which the project had helped the school gather and use evidence to enhance Māori student achievement and other outcomes
- Perceptions of the value of project outcomes for Māori students, all students, and school personnel
- The nature of feedback from teachers, parents and others about Te Kotahitanga and its focus on Māori achievement
- Whether and in what ways school policies and procedures changed as a result of participation in Te Kotahitanga.
Principals were also asked about the sustainability of Te Kotahitanga and invited to make specific suggestions to support sustainability as well as offer advice to other schools starting Te Kotahitanga. At the conclusion of the interview, they were also asked if there was any other issue they saw as critical that was not covered in the interview.
Questions for the Heads of Departments and Deans included:
- How well is Te Kotahitanga being implemented in this school? How have you worked to implement the intervention?
- What are the reasons teachers have for deciding whether or not to participate in Te Kotahitanga?
- How valuable are the outcomes of the project for all students, both Māori and non-Māori.
- [For those who have been in the school for several years] What changes have you noticed in the school since Te Kotahitanga has been implemented?
Lead facilitators have a critical role in the implementation of Te Kotahitanga at the school level. While lead facilitators appeared to have substantive relevant experience related to their role as professional development coordinators and regarding teaching Māori students, not all were themselves Māori. Generally, however, the lead facilitators were seen to have credibility in their role amongst teachers with whom they worked on a regular basis. One issue that did arise was the relationship of the lead facilitator with school management, as they did not generally sit on the school's senior management team that met regularly to inform school policy and practices. This meant that their influence was highly dependent upon informal networks with middle management and with the principal's as well as each lead facilitator's communication skills and capacities. Several lead facilitators explained that they were very new in their role. In some schools, there had been two or even more people in the lead facilitator role since joining the project in late 2003. All 22 lead facilitators were interviewed individually. One lead facilitator was not appointed in time for the interviews at the other Phase 3 schools but was instead interviewed later in the year when we gathered data at Phase 4 schools.
Facilitators also play critical roles in the implementation of Te Kotahitanga at schools. In all, we interviewed 32 professionals working as facilitators at 18 of the Phase 3-4 schools; not all facilitators could be interviewed as many were part-time and hence not present in the schools on the day/s scheduled for our visit. In smaller schools, there may be only one person (in addition to the lead facilitator) in this role, and thus these interviews were individual. In larger schools where there were two or more facilitators, the interview may have been carried out individually or with 2-4 persons together.
As was the case with other groups, interviews were conducted in a private space by one member of the research team and recorded digitally for later transcription and coding.
Interviews focused on the facilitators' beliefs about the impact of the Te Kotahitanga programme at their school on teachers and Māori students. The questions derived from the overall evaluation research questions focused on these broad issues and considerations, including specific questions such as:
- How Te Kotahitanga professional development operated within the school
- What they saw as valuable outcomes of Te Kotahitanga for students and teachers
- What were the main enablers for getting Te Kotahitanga to work well
- How does Te Kotahitanga affect professional attitudes, school culture, formal school policy and systems and informal practices, or "the way we do things around here"
- What they regarded as the most significant changes made by teachers to improve Māori student achievement
- What kind of training for Te Kotahitanga they had had, how effective it had been, and whether there were any additional training and supports they felt would be helpful.
Facilitators were also asked to tell us in their own words how they would describe Te Kotahitanga to someone, and, if they were new to the role, what their main challenges had been.
Each of the main questions could be followed by additional probe questions where clarification or explanation was needed, or where it appeared that examples would enrich the data.
In total, we interviewed 214 Māori students who were enrolled in Years 10-13 about their experiences at the Phase 3 and Phase 4 schools. At the 12 Phase 3 schools, 22 focus groups comprising 122 students met with us, and 17 focus groups comprising 92 students at 10 of the Phase 4 schools met with our research team. At each school, we requested that at least one student focus group be scheduled for up to an hour, and at most schools we met with two student focus groups.
The interviews with students were focused on how students felt about school, their learning, learning as Māori and other issues of cultural identity, and aspects of Te Kotahitanga. As in the other focus groups, the full set of questions was shared with students prior to returning to the questions one at a time for responses. The questions guiding these interviews were:
- How does this school support your learning? How do your teachers show they care about your learning, not just generally but each of you personally? What do they do or say to show they care about your learning?
- How do teachers make learning relevant to your own lives? Can you think of some examples where students could bring their ideas, experiences or questions into classroom learning activities, in different subject areas?
- What does being Māori mean for you? Does "being Māori" look different in school compared to how it looks outside school?
- Do you feel that you can "be Māori" in this school? In different classrooms? Can you give some examples of how teachers let you know that they respect and understand "being Māori"? What if anything makes you feel you aren't supposed to "be Māori" in the classroom?
- How do the teachers incorporate Māori culture and understandings into different subjects? Activities that you do in school? Other?
Follow-up questions were pursued whenever issues were raised that required clarification, elaboration or examples. In the interviews, we highlighted more the question regarding whether "being Māori" looked different in school compared to outside of school in order to emphasise our interest in how Māori identity was represented in and out of school as well as whether students felt that this identity was welcomed and celebrated in the classroom.
Each focus group was facilitated by two researchers, at least one of whom was Māori. The group met in a variety of venues chosen by the school including the library, the marae meeting house, in the common area, in Te Kotahitanga project space, and outside. One researcher served as interviewer, reading out the questions one at a time. Students were encouraged to respond while detailed notes were taken by the second researcher to record each response. When students no longer added comments in response to a particular question, the note-taker read back all responses to the group for any edits, additions or deletions.
Whānau and BOT interviews
The perspectives of Māori parents and whānau are crucial to any evaluation of the impact of a programme that has at the heart the goal of meeting the educational aspirations of Māori. Indeed, Te Kotahitanga is grounded in the idea that schooling of Māori students should be accountable to the Māori community and whānau. In principle, family and the local community are meant to play a key role in New Zealand's model of self-managing schools, such that each school is governed by a community Board of Trustees that sets strategic directions for the school within national requirements and that has responsibility for fiscal oversight and employment of the school principal. It would be the Board of Trustees that would have the ultimate authority to decide whether or not the school would participate in an initiative such as Te Kotahitanga, generally with the advice of and information provided by the principal.
We requested an interview with the Board of Trustees (BOT) elected chairperson at each of the project schools, to be organised at a time convenient to the BOT Chair either during the school day or after school during our scheduled evaluation visit if possible. This interview was done individually, in a private room, by a senior member of the evaluation team and recorded digitally for later transcription. Interviews were carried out with 9 BOT chairs at 8 of the 12 Phase 3 schools (2 of these were the newly elected BOT chair and the person who had been chair when the decision was made to participate in Te Kotahitanga) and 6 BOT chairs at 6 of the 10 Phase 4 schools.
It was beyond the scope of this evaluation to gather data from a large sample of parents and whānau. However, we were interested in how this critical group saw Te Kotahitanga and their children's achievement at the project at the schools, so we requested a meeting to interview Māori whānau at each of the schools. As with other interviewees, we relied on the school (generally the lead facilitator working with the school principal) to organise a focus group meeting with Māori whānau after school hours, generally at the marae if there were one connected to and representing the iwi of the school community. In all, we held 19 Māori whānau focus groups at 17 of the 22 schools.
Because of the small sample sizes, the fact that not all schools participated in these interviews, and that those who did were selected by Te Kotahitanga personnel for participation, this sample is not representative of Māori whānau/families whose children attend these schools. We will, however, identify emerging patterns where there were strong themes emerging across the data.
The practice adopted for our interviews with the Boards of Trustees chairpersons and whānau groups requires that as researchers we are cautious about collapsing discourses from interviews with individuals, namely, teachers and students, with those representing the institution of the school and the community. Pedagogy as the activity engaged in by students and teachers is at a different level of discourse from those concerned with schools as institutions. Interviews undertaken to date of both whānau groups and Boards of Trustees members attest to this difference. Both whānau and board members are obviously much less familiar with the project than those operating within the school and especially classrooms. This is only a concern in the sense that participation in schooling and attitudes to formalised learning are critical components of what is known about student performance in the life of schools generally. What the home and the community know about what goes on in classrooms and schools remains a 'hidden garden' unless action is taken to close the information gap.
Procedures for the whānau focus group interviews
The focus group interviews with whānau were conducted using a process involving two interviewers, at least one of whom was Māori. The Māori researcher served as primary interviewer though both interviewers contributed to the discussion (the latter was non-Māori in some cases). The meeting was digitally recorded for later transcription except in a few instances where whānau indicated they were not comfortable with the recorder and for those meetings using the note-taking method already described. In those instances, it was negotiated that notes would be written to read back to whānau to verify the accuracy of what had been recorded for the group. The focus group meetings followed marae informal meeting protocols and generally began with mihi mihi, and the research team provided for a light supper and drinks so a karakia was also said. These meetings ranged from approximately one hour to as much as two hours long, and no attempt was made to impose a rigid ending time though we were conscious of family commitments to return home within a reasonable time.
Whānau were asked the following questions following from the evaluation research questions:
- What does Te Kotahitanga mean to you?
- Have you noticed any change in your children's interest in school that might be related to Te Kotahitanga? Tell us about this?
- How well is school working for your children academically? In general?
- Have you noticed any increase in the presence of Māori culture in the school in the last few years? Can you tell us about this?
- What changes have you noticed in how teachers approach students and why do you think that has occurred? What if any changes seem to have occurred as a result of Te Kotahitanga?
- In what ways have you been involved in the school? Has this changed in the past four years?
- Is there anything else you'd like to say?
The discussion with the parents generally followed these questions, but it was more free-flowing in allowing directions chosen by the parents rather than proceeding through a list as noted above.
Procedures for the BOT Chair interviews
Our questions for the chair of the Board of Trustees were organised around the following issues:
- Why and how the Board supported Te Kotahitanga
- How Te Kotahitanga had an impact on the school, benefits to the school, negative effects for the school, and what changes had been noticed or reported to the Board
- Impressions about the quality of the Te Kotahitanga programme and how these impressions had been informed
- What they considered to be valuable outcomes for Māori students and for all/other students
- What they considered as the main enablers and barriers for Te Kotahitanga and what advice they would give to another school considering adopting the programme
- What they felt was needed to sustain Te Kotahitanga
Finally, each interviewee was asked whether there was anything else that hadn't been discussed in the interview which was important to mention.
Coding and data analyses
The results reported here across schools focus on the extent to which classroom teaching and learning reflected implementation of the ETP. We also provide examples of the range of implementation of the different aspects of the ETP including missed opportunities. Classroom observations were coded as High Implementation, Implementation, or Low Implementation of the ETP. High Implementation and Low Implementation (including missed opportunities) were coded as specified below, and "Implementation" was coded for observations that did not fit within either the High or Low categories as follows:
- Some evidence of at least 5 of the 6 ETP dimensions
- Strong evidence for at least 2 ETP dimensions
- Must include evidence of culturally responsive pedagogy
- Must reference learning outcomes/objectives/aims
- Evidence of positive teacher-student relationships
- Positive classroom management supporting learning
- No evidence of any of the ETP dimensions observed
- Alternatively, weak examples or missed opportunities
- Misconceptions or inaccuracies/wrong message
- Mismanagement of the classroom disrupting learning
These three categories were defined following independent review of a sub-sample of the observations from the Phase 3 data analysis by the authors. Once consensus was reached on the coding criteria, different pairs of researcher coders were assigned to code each observation, making sure that no one was coding an observation that he/she had done personally and ensuring that each researcher coder was paired with one another for at least some of the data. Observations were assigned to one of the three quality categories where the two independent coders agreed on the category. Where there was disagreement, a team of 3-4 researchers discussed the observation and reached consensus regarding how it would be coded. The baseline Phase 5 observations were carried out by four researchers (Hindle, Hynds, Penetito, Savage) and scored independently by two researchers who had not participated in those observations (Meyer, Neal). Again, any disagreement was resolved through scoring the observation by a third independent coder who had not been involved in that particular classroom observation
Results were summarised by subjects across schools and by school across subjects, and exemplars of different levels of implementation were identified.
Interview transcription, coding and analyses
Interviews were recorded on digital recorders as separate files. All interviews were typed into Word document transcriptions by experienced transcribers, with a sample of all interviews checked for accuracy by specific researcher interviewers; in some cases, researcher interviewers were asked to clarify text in the transcription where questions arose. The Word documents were then coded using NVivo by experienced coders with advanced training and experience using NVivo with similar interview and focus group data. Members of the team reviewed printed transcripts and met to review possible codes towards identifying themes in the interview data based on their experiences having carried out the interviews. To identify codes, two core researchers from the VUW based team reviewed a sub-sample of complete interview transcriptions to identify possible nodes and words for coding using NVivo, and the full set of codes was discussed further with the international consultant and an additional researcher team member prior to coding all the data. Once the data were coded, themes were identified from the data by those same team members and salient quotes identified to illustrate the themes.
Teacher interview results in particular were organised according to three key issues: (a) what teachers told us about their own expectations for Māori students and the evidence of outcomes for students as a function of Te Kotahitanga; (b) teachers' beliefs knowledge and understanding of culturally responsive pedagogies; and (c) teachers' experiences with the Te Kotahitanga programme and their perceptions of its impact on their teaching. In order to probe further teacher beliefs about expectations for Māori student achievement, follow-up questions focused on what they regarded as the most important educational outcomes for Māori students as well as for all students and how well they thought their own school was doing to achieve those outcomes. Information was noted regarding how many statements were made within selected themes. However, it is important to emphasise that these data are not intended at this point to be interpreted quantitatively nor subjected to statistical analyses to draw inferences about the representativeness of the issue raised. Rather, the number of comments is included to give an indication that a particular theme was mentioned often or rarely, which is likely to be of interest in inferences about the extent to which a particular interpretation is in the forefront of teacher perspectives. However, the main purpose of the interviews and the analyses of these interviews is to identify patterns in teacher responses to Te Kotahitanga and how they view effective teaching of Māori students towards better meeting the educational aspirations of Māori.
Procedures to evaluate student outcomes
At the original 12 Phase 3 schools, virtually all teachers teaching Year 9-10 students were trained in the model in either 2004 or 2005; only a few had either never been trained, had not yet been trained (e.g., new hires), or had chosen not to participate. As Bishop and his colleagues have emphasised, Te Kotahitanga has not been implemented through random assignment of schools to the project versus comparison conditions, nor have teachers within schools been randomly assigned to the project. Thus, the approach taken to investigate student outcomes associated with Te Kotahitanga was restricted to a quasi-experimental research design and data analyses appropriate for this design.
The evaluation proposal had anticipated comparing: (a) project outcomes pre and post the introduction of Te Kotahitanga; (b) results across the same time period for students in classes taught by Te Kotahitanga teachers compared to students in classes taught by non-Te Kotahitanga teachers; and (c) comparisons across the same time period for Phase 3 schools with a comparable group of schools not participating in Te Kotahitanga.
Of these, comparing results for students in classes taught by participants versus non-participants (b) did not prove feasible nor valid for several reasons. At Phase 3 schools in their fourth year of implementation in 2008, nearly all teachers were directly involved in Te Kotahitanga at the time of data collection. Even teachers not currently involved had been trained and participated at some point in the project. At the Phase 4 schools, only one teacher cohort was nearing the end of the first year of full implementation following the training year, and a second teacher cohort was nearing the end of the training year. Attempting comparisons for individual student results is complicated by the reality of student experiences: as is true for all students, Māori students are enrolled in 5-6 subject periods daily including classes taught by both teachers participating and teachers not participating in Te Kotahitanga. To attribute student results to teacher participation in Te Kotahitanga would require quantification of percentages of classroom participation each term and across years, requiring a level of detail regarding student attendance that was not available to us even if this were possible. Even if it were possible to quantify each student's exposure to Te Kotahitanga teachers across the school day and year, one would also have to qualify this exposure by the implementation quality demonstrated by each teacher. Thus, quantifying the "influence" of portions of the school's day on each student's achievement is a complex matter beyond the scope of this evaluation. In addition to requiring many more hundreds of observations than were funded for this evaluation project, these multiple challenges to the validity of such comparisons would undermine their value.
Hence, our evaluation data focus on (a) and (c). Here too, the evaluation was limited by the extent of participation in Te Kotahitanga. For (a), the project required valid and reliable school level data on individual student outcomes including achievement and factors related to achievement. Year 9-10 student outcome evidence available is overviewed below. Year 9-10 evidence of results collected directly from schools included interview data providing information on outcomes perceived by various stakeholders (students, teachers, whānau, school leaders, and so on) as resulting from Te Kotahitanga.
Evaluation of longer term student achievement outcomes in Years 11-13 when students are completing NCEA credits requires that students were exposed to Te Kotahitanga teaching reflecting the ETP beginning in Year 9. As explained in chapter 1 (see also Table 1), comparisons would not be valid for Phase 4 schools. The timelines for implementation of Te Kotahitanga mean that the only valid comparisons at Phase 3 schools are those for NCEA Level 1 for Year 11 students in 2008 (those results are made available in 2009; 2009 results are not available in the relevant databases until mid-2010). Hence, we were able to report evidence with regard to (c) by comparing NCEA Level 1 student results for the 12 Phase 3 Te Kotahitanga and 12 similar, comparison schools across the years starting prior to implementation of Te Kotahitanga and extending to 2008.
The evaluation investigated the use of multiple data sources covering a range of quantitative and qualitative evidence of outcomes for students. Sources of evidence and project evaluation activities associated with each include:
- Student Interviews: Focus group interviews were carried out with students in Years 9-13 at the Phase 3 schools in April 2008 and at the Phase 4 schools in October 2008, investigating student perspectives about culturally responsive teaching and learning; attitudes about their learning; perceptions of being Māori in school; and other aspects related to Te Kotahitanga.
- Student Achievement in Years 9-10: We report limited assessment data previously administered by the schools for the University of Waikato research team, including measures of numeracy and literacy. However, not all schools were systematically assessing students nor were their data management systems fully operational for reporting these data.
- Student Achievement in Years 11-13: NCEA and other qualifications attainment information was sourced by our evaluation project directly from NZQA and from the statistics in the Ministry of Education's Benchmark Indicators for 2004 to 2008. However, selected achievement outcomes sourced from the Benchmarks database do not break down results by ethnicity, so can be reported only for all students.
- Māori Student Attendance and Retention: These factors are noted explicitly within the Te Kotahitanga contracts as dependent variables expected to be affected by Te Kotahitanga. Direct evidence was not available from the Ministry of Education databases nor from milestone reports to the Ministry of Education. A request to the schools for these data met with variable response, and the schools' annual reports and Te Kotahitanga milestone reports did not include this information. We have been unable to source the data and thus are unable to provide these measures across the Phase 3-4 schools.
- Percentage of Māori students in School Discipline Data: The contracts also note that Māori are currently over-represented in stand-down, suspension and exclusion statistics, so that the percentage of Māori in these statistics would be expected to become more proportionate to their school population percentage as a function of the project. Again, these data were difficult to obtain, and a request to the schools for these data met with variable response.
- Percentage of Māori students in High and Low Ability Bands: For core Year 9-10 subjects, virtually all schools we visited utilised some form of grouping classes of students according to different achievement or ability levels. The proportion of Māori at the different levels would provide an indication of student achievement but also school and teacher expectations. All schools noted at least one accelerant or high ability class and one low achievement group (in addition to the group labeled as special needs who were receiving special services). We attempted to determine the percentage of Māori students in the one highest and one lowest ability/achievement band. These data are not recorded in any formal Ministry of Education-related database, and a request to the schools for these data met with variable response. Schools indicated that they did not record these data in their permanent databases either annually or over time.
Another challenge for the evaluation project was the timeframe. It is important to identify which years are appropriately identified as "baseline" or pre-Te Kotahitanga and which are implementation years during which the effects of Te Kotahitanga might be evidenced. The Phase 3 school implementation schedule means that only some year 9-10 classes were directly involved in the Te Kotahitanga model in 2004, depending on which teachers were in the first cohort. The first hui for teachers had occurred late in 2003, so that 2004 is realistically a "training" year, and this first cohort might not be expected to be well-versed in Te Kotahitanga until the 2005 year. For the second cohort of teachers, 2005 was the "training" year involving additional year 9-10 classes, with full implementation for this cohort in 2006. For the Phase 4 schools for which 2008-2009 were still training years for most teachers, it seems precipitous to expect student achievement results to be evident.
The majority of teachers who teach year 9-10 classes in the junior school are also involved in teaching senior secondary subjects. Thus, one might expect teachers to extend their understandings to how they approached teaching at years 11-13 as well. However, the generalisation of new teaching practices to the senior school is not within the scope of our evaluation nor, to our knowledge, has this issue been investigated by the University of Waikato team. Whether or not teachers do generalise the skills practiced in Te Kotahitanga to their teaching in the senior secondary school is, of course, an empirical question that could and perhaps should be investigated.
Hence, for purposes of analysing student outcomes we generally regarded 2004 as a "baseline" year for student outcome measures and 2006 as the first full year of implementation at the Phase 3 schools and 2008 as the first year in which one might expect NCEA achievement to show any effects. However, the Te Kotahitanga project does not carry out any activities focused on teachers teaching Year 11-13 subjects; if what the students experience in Years 11-13 is unchanged and does not reflect culturally responsive pedagogies, this could be expected to have a (negative) impact on student outcomes once again as Māori students move to the senior school.
Procedures for matching comparison schools
Each of the NCEA data analyses described in Table 2 will be presented across time and for the 12 Te Kotahitanga schools in comparison to a sample of 12 "like" schools matched school-by-school with project schools as closely as possible using to the following criteria:
- North Island state schools: As Te Kotahitanga has been implemented only on the North Island to date and has not included area schools or schools with a special character, the 12 schools selected as comparison schools are also North Island and of a similar type (state schools, either coed or single-sex) and matched within one decile level;
- Percentage of Māori students at the school in 2004: the 12 Te Kotahitanga and 12 comparison schools had an identical mean average of 41.4%, with no match showing more than a 20% difference;
- Percentage of Māori students leaving school with at least a Year 12 qualification in 2004: across schools, the two samples showed a mean average difference of only 1.5%, with no match showing more than an 11% difference;
- Geographic location/region and school size: Following application of the above criteria, comparison matches were then selected based on geographical region (e.g., both rural, small town, urban) and school size.
Table 2 shows these statistics, with school matching data reported across the 12 rows.
Table 3 shows the Māori percentage of the school roll for each year from 2004 to 2008 for the 24 schools. As indicated, the total student roll was larger at the Te Kotahitanga schools (4,628 compared to 3,816). However, these samples are sufficiently large and the standard deviations comparable across the two groups so that this numerical difference would not be expected to affect the validity of further comparisons.28.228.2
|Te Kotahitanga schools||Matching comparison schools|
|School ID||% Māori Leavers*||% Māori Roll||No of Māori Students||School ID||% Māori Leavers*||% Māori Roll||No of Māori Students|
|Year||12 Te Kotahitanga schools||12 comparison schools|
|Mean||Median||SD||SE mean||Mean||Median||SD||SE Mean|
For each of the achievement indicators described in the student outcomes chapter 2 of the report, data that are analysed were obtained from the Ministry of Education Benchmark Indicators based on school annual reports. Individual school results for the 12 Te Kotahitanga and 12 comparison schools were entered into a data set for statistical analysis using SPSS. As 2004 was a training year for the first Te Kotahitanga teacher cohort focusing on Years 9-10, we suggest that this is an appropriate "baseline" year for further analyses of NCEA results for Years 11-13.
Comparison of NCEA achievement across schools for all students
We report comparisons of various NCEA achievement outcomes reported in the Ministry of Education Benchmark indicators across the years 2004 to 2008 for the 12 Phase 3 project and the 12 comparison schools in this chapter rather than in the student outcomes chapter. The Ministry of Education Benchmark indicators do not provide a breakdown by ethnicity for all data, so that it was not always possible for us to utilise this data set in order to report on Māori achievement across this time period for some indicators. Frequency statistics were generated using SPSS and tests of significance used where individual student numbers were sufficient for these procedures.
- Note also that we site-visited and collected data at one of the Phase 3 schools during this same October timeframe. As explained in our Phase 3 report, that school was recruiting a new lead facilitator at the time of the April Phase 3 site visits. We felt it would be more appropriate to return in October to complete data collection at that school once the new lead facilitator had had at least one full term to establish Te Kotahitanga PD routines.
- At one school, however, the kaumātua challenged the principal who mistakenly stood with our research team rather than with whānau welcoming us onto the marae, an incident revealing that not all school leaders are cognisant of appropriate cultural protocols.