Evaluation of Te Kotahitanga: 2004-2008
In 2007, Victoria University was contracted by the Ministry to produce an external evaluation of the effectiveness of Te Kotahitanga. It is the first external evaluation of Te Kotahitanga.
This is the full technical report of the evaluation of Te Kotahitanga in 22 schools from phase three and four of the programme, from 2004-2008. Substantive findings from the evaluation report concluded that Te Kotahitanga is a sound and effective process for improving classroom teaching and learning for Māori students.
Also available on Education Counts is the Te Kotahitanga Summary Report, which outlines the key findings.
Author(s): Luanna Meyer, Wally Penetito, Anne Hynds, Catherine Savage, Rawiri Hindle, and Christine Sleeter. Report for the Ministry of Education.
Date Published: August 2010
This report is available as a download (please refer to the 'Downloads' inset box). To view the individual chapters please refer to the 'Sections' inset box. For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
Chapter 5: Impact on Students
The overarching question for the Te Kotahitanga evaluation research is: How well and in what ways does Te Kotahitanga work toward the goal of improving Māori student achievement? To evaluate the impact of Te Kotahitanga professional development for teachers on outcomes for students, three broad categories of student outcomes for Māori (as well as for all students) are relevant: student achievement, student behavior, and student attitudes about their learning. We investigate the perspectives of teachers, principals and other school leaders, Te Kotahitanga facilitators, family/whānau, BOT chairs, and Māori students regarding outcomes for Māori students including their achievement. We also report available data on achievement and achievement-related behaviour sourced from schools, the Ministry of Education, and the NZQA to investigate for changes over time associated with implementation of Te Kotahitanga.
Te Kotahitanga is focused on teacher professional development for the purpose of enhancing Māori student achievement. Nevertheless, there is a risk associated with any initiative to raise student achievement if deficit theorising is allowed and legitimised as the explanation for lower achievement outcomes. The messages associated with Te Kotahitanga are not deficit oriented: Instead, the emphasis is upon the acquisition of new skills and understandings by school personnel so that teachers and schools are culturally responsive to Māori as the primary approach to enhancing Māori achievement. Whānau put it best:
When Te Kotahitanga was first introduced at this school—I think some of the kids got the wrong message, because it was like Māoris[sic] aren't brainy enough and that is why we need this programme. So that has been an assumption, and unfortunately some of our kids have picked up on that stereotype of Māoris[sic] are underachievers, and they have lost that connection to our tipuna then they don't know about how clever as a people we are, our links back, our tipuna were creators, composers of kapahaka, moe te te (mōteatea), artists, navigators, entrepreneurs, business people, chiefs—so my question is back to the school—How is being Māori valued here at the school? What messages are kids picking up?
The extent to which schools become culturally responsive for Māori is the first step on the road to enhanced educational outcomes. Teachers are challenged to assume agency for Māori student achievement in striving for culturally responsive pedagogies and must confront deficit thinking rather than seeing underachievement as primarily a student or family problem. Hence, we introduce this chapter on student outcomes with perceptions regarding the place of Māori identity in Te Kotahitanga schools.
Learning and belonging as Māori
Educationalists emphasise that a sense of belonging and feeling valued are key to student engagement in school leading to enhanced achievement outcomes. As one teacher put it:
For me it's I like the inclusiveness of it, me being included in their world as much as them in this world. But for me that's the neat thing about Te Kotahitanga is the inclusiveness of them in the classroom too ... the manakitanga concept that we've all wanted to have … in the classroom. Where everyone feels that they are cared for and they are acknowledged and that they look after others.
The cultural features of schools can "match" the cultural identities of some students but represent a mismatch for others. Ironically, New Zealand schools can be a mismatch for Māori who are the tangata whenua of Aotearoa, such that the cultural identity of schools as social systems, the cultural foundations of mainstream curricula, and the majority of school personnel all differ in crucial ways for Māori. School achievement is not simply an academic matter, but is a composite of multiple influences and outcomes for children and their communities.
School personnel on Māori student identity
Facilitators emphasised the importance of creating schools that are safe environments for all students to learn, where all students feel they belong, and where students can bring valued experiences to the learning process. They perceived that cultural safety had not been a common experience for Māori students in the past. As one facilitator put it:
I think the most important thing is for teachers to create an atmosphere in their classroom where the students feel safe and comfortable and is an environment in which they can learn. Because I think in the past there have been a lot of students, Māori students, have not felt comfortable. They have not had good relationships with the teachers and I think this programme is helping the teachers to build good relationships and therefore create that safe environment.
Although many teachers noted the relevance of culture to teaching and learning, few discussed specifically the development of a culturally-grounded identity as an educational outcome for students. Most teachers seemed to view "caring for students as culturally located individuals" as walking a line between acknowledging culture and stereotyping cultural characteristics. Some teachers did give examples, however, of how the programme had made them aware of cultural identities and what it meant to students to be able to learn as Māori:
I think it's important that Māori students feel comfortable coming into my room… that they can be themselves… and that also means that I show respect for their language and their culture ... so little things like using Māori phrases, working hard to get my pronunciation right… but also that I show that respect for all the different ethnicities in my classroom… Classrooms shouldn't be monocultural... that's one thing that I picked up from the TK PD… that you do as a teacher make an effort… I'm a New Zealand European and I think before this programme I never really thought about kids' culture or their background, whereas now I make more of an effort with establishing relationships.
Teachers were also able to express the importance of acknowledging Māori students as Māori as integral to promoting Māori student achievement:
That's … the beginning of promoting Māori achievement. Know them as Māori and respect them for that, and also that it's an honour to be Māori and be proud of your culture.
School leaders such as Deans, Deputy Principals and Heads of Departments at the Phase 3 schools in particular included cultural identity as an important student outcome they hoped would be promoted by teachers as a result of their Te Kotahitanga professional development activities. Among the varied and multiple benefits they saw for students as a result of the programme, these professionals wanted students to perceive that their Māori culture was valued and respected. They also expected there would be specific evidence of these values and respect, which we discuss in more detail in the chapters on the impact of the programme on the classroom and on the school.
Student perspectives on learning as Māori
Students discussed what it meant to be Māori at school:
To me, it means expressing the culture of being a Māori and not being afraid or shy to show it.
I'm tangata whenua, I can carry my Māori culture to the next generation.
Students were also able to discuss ways in which schools and teachers did or did not enable them to be Māori at school:
Nothing makes me personally feel like I shouldn't be Māori, because that's just who I am and they can't change that. Some things like with the teachers it feels like they're targeting us and its like oh, is it because I'm Māori? Or is it because of the people I'm hanging out with that are also Māori?
The Māori teachers they always encourage you to take Māori and they say 'You should be involved for NCEA'. It's good because the Māori teachers they encourage you so you don't lose your culture.
In several student focus groups, students discussed how teachers attempted to speak te reo Māori in the classroom:
Our maths teacher she always tried to speak Māori which is very cool because she like write the date up in Māori for our class. She tried to say some instructions in Māori like what she does, so yes we found it pretty cool how some of our teachers try to [speak Māori].
Yeah he's learning he's on the same road as us. Yes we always learn, we teach him new things and he like tries to talk to us in Māori and he says what's this or what's the Māori name for this and then says it all the time. But he's learnt heaps, like when he first came into our class he didn't know any Māori, didn't know how to say things and we just teach him now.
Students appreciated teachers who tried to understand and acknowledge the experiences of being Māori and use Māori culture and/or language in their classrooms:
Ms C speaks to us in Māori and sometimes she says the numbers in Māori, the countdown—she comes to the Marae and the wananga a lot and when we have visitors she comes and listens to the whaikorero.
In dance and drama, on a wall there are all these Māori words. Every day she picks a different way to say Good Morning in Māori.
Whānau perspectives on Māori identity
As had the students and school personnel, whānau also emphasised that enhanced achievement outcomes required environments where students' socio-emotional and cultural lives within the school were nurtured as a cultural context. One parent explained:
Well what I think it [Te Kotahitanga] is, it's a programme that's aimed at our Māori students, mainly but not specifically. It's also for the rest of the school, but to bring their achievement up for our Māori children, to make them also feel that they're part of the school system, that their language and their culture can be integrated into the classroom. That it's not something to be ashamed of. I think it enhances more of the Māoritanga, Māori tikanga in the classrooms, [where] the teachers are making their lessons with the Māori kind of flavour to it.
Parents commented that New Zealand schools were not really designed for Māori nor did they reflect Māori values and experiences, thus entailing a "mainstream" that is disempowering for Māori students:
I can see from the teachers there is rejection. It's a new programme, but the problem is that we're still in the colonial system, it's not just the teachers, the system needs to change, there is work that needs to be done, because we (the whānau) can see problems.
I agree the principal is supportive but in the mainstream curriculum how much is directly about Māori history, entrepreneurs… Māori role models.. that to me is directly linked to the value of Māori in the school. It is a sad thing to hear my children say "Why do they say Māori don't attend school?" Because my kids always attend, they go to school. So it's about the messages the kids get about being Māori in the mainstream. That's why you have to look at the system; otherwise Te Kotahitanga is a Māori concept trying to work in a non-Māori environment.
For whānau, achievement goes beyond the formal curriculum and includes cultural knowledge and experiences for their children. They feel strongly that these should also be engendered as part of their children's education:
My dream for my child, my children, is something that I've not been able to give them because I didn't learn te reo Māori, tikanga Māori all those things. I'd be so proud to see my son standing and doing the paepae, doing the whaikorero, that sort of thing. Our Māori culture is not something that is easy to learn somewhere else, and if they can do it now, it's going to go with them for the rest of his life. That's my dream for him.
For me achievement is my son having more sort of involvement and wanting to learn more Māori more tikanga, more te reo, just being involved because there's not a lot out there, and this is one of the only places where they're able to get it while they're growing up if they're not living back on the marae.
Whānau were pleased that Te Kotahitanga, in their view, supported students culturally and instilled in them increased pride in being Māori along with increased interest in being part of their school. They perceived that their children were more willing to participate in the marae at their school as part of this change:
Our tamariki thought the only time you could go to the marae was if you were in the kapahaka. Now more Māori children are going to the marae.
High expectations for achievement
Having high expectations for learning and behaviour is essential to enhancing outcomes for Māori students. Teachers generally expressed that expectations should be high for all students and that this was important for their own classrooms, though there was a lack of clarity regarding what high expectations means for Māori students. Teachers discussed their expectations for students mainly in relationship to academic attainment; several teachers linked academics with broader life options, life-long learning, or development of academic self-confidence. Most teachers discussed academic achievement in terms of student attainment of NCEA qualifications, sometimes tied to university preparation.
Teachers expressed high expectations for Māori, regardless of their evaluation of "ability", and were able to explain why it was important to strive for higher levels:
For me it's about choices … I agree so it's about giving them the knowledge, tools, skills to be the very best to go on and achieve in anything they want to do ... but also that they know they have a choice to go to university, polytech or go out and get a job ... whatever they want to do… I think the school tries really hard to promote that and communicate that to the students.
For the average Māori girl coming in here, she should be capable of leaving with her level three. And I'd hope that she'd at least have the choice to go to university, if university wasn't for her, that at least she'd have that choice.
There were non-specific discussions of achievement or outcomes such as improved attendance and classroom behaviour which could reflect ambivalence when students arrive in their classes with low achievement levels:
I think the most important outcome is that they feel like they are a success and they feel like they've done their best. Irregardless of where they end up, whether they end up at university, or become mothers or whatever they end up doing, to feel confident and successful and that they have achieved something and they've done their best. It's different for everyone. Māori /non-Māori, in this class that I'm struggling with, it's a mixed ability class and there are some kids in that class that are really, really struggling, and so my expectations for them are obviously to do their best. Well their best is lower than my expectations for someone who is able to achieve more.
Some teachers may also have given up on some students, and streaming was related to negative attitudes that may affect both teachers and students, despite original intentions that streamed groups provide acceptable structures for teaching and learning:
I think all students should at least get level one NCEA if they can. The reality is that, I guess at the age of 16 if they want to come back, then they will probably be able to do it. If they want to do it. But as a teacher, I'm a teacher of a year 12 lower ability class, they all would be over 16 and have come back. They don't really want to, they aren't trying, they don't want to do it.
Teachers also talked about setting an appropriate level of challenge for students:
You need to, as the teacher, make that effort… You also need to have clear and high expectations for the kids … don't threaten things that you're not going to follow up on. You need to have clear consequences for things, and that's not just for Māori kids, that's for all kids. You need to recognise and celebrate their achievements … and you need to challenge them to 'step up' and if you set that challenge for them and help them they will do it. They can achieve. You have to know your Māori kids in your classes, but you still need to expect them to learn, let them know that they can do it. They can do well in class academically.
Finally, teachers were able to articulate lifelong learning goals for their students, including the need to instill self-confidence that students could reach their goals:
The thing that I find with the Māori students in particular, but across the board also, is the confidence. If I can get my students to leave here with the confidence to move forward and to have some strategies to deal with things. And a lot of them too, to have that self-esteem lifted, which will give them that confidence to say "Yeah, I'm going to have a go at that. Yes, I am good at that." It is a lot across the board too, but I do find that there are a lot of Māori students who are reluctant to acknowledge their talents. And I'd like to see that brought out, and I actually feel that we are doing alright with that.
Confidence in their own ability, opening their eyes to what they can actually achieve. They are not closed off in a little box and they are not coming to school because they've got to jump through hoops and then when the students are 15 or whatever age they can go. It's kind of 'right you didn't know this and now you do know this and what you can do with it.
Principals described their conviction that discussions among teachers had changed regarding expectations for Māori students, and teachers were seen as being more focused on their agency for promoting learning rather than seeing students as unable to learn. They emphasised how teacher conversations had shifted from deficit theorising to taking responsibility for supporting higher achievement:
Yeah, "let's talk about achievement. Let's talk about learning. And let's talk about needs of students." As opposed to "They can't, they shouldn't, they won't be able to" type of deficit stuff... You move away from that and start talking about learning, and it's encouraging.
Well the one thing, again it's changed [is] this whole deficit theorising. You know teachers have always been very good about "Well if Wally would only put his shoes on to come to school" or whatever it is that is going to make him learn better. What can you expect when he comes from that family?" and so on. A lot of that talk has gone away, and I suppose that has been the training, making them aware that they had been deficit theorising in the past.
Principals appreciated that the programme required the school and its teachers to become more aware of Māori students and how they were doing in school:
Valuable outcomes for students? I think that there is more attention, and teachers are more aware of their Māori students in their class, and that the trying or implementation of new strategies or ideas is [happening], and that Māori students are feeling more valued in particularly, I can only talk about my classes, but [there is] just more inclusiveness and more activities aimed for their achievement. (Head of Department)
Whānau too noticed how expectations changed students' attitudes towards school. One parent felt that the fact that a science teacher was himself Māori had a great deal to do with student engagement:
For the first time in science, they have had a Māori teacher and it is amazing the impact of that. They are encouraged and pushed by their science teacher, and my kids are so excited about science. That has never happened before, at home they come together and they help each other, that is amazing… and I think it's the comparison of having a Māori teacher compared with a non-Māori teacher. It's natural, he identifies with them, and for me as a parent he is very considerate. When I had my interview with him and he talked to me about my kids' work, he didn't talk down to me. I notice with my kids in science that he pushed them to do their best, he pushes that responsibility onto them for their learning, but he supports them too.
Impact on attendance, engagement, and retention
Ultimately, Te Kotahitanga seeks to enhance Māori student achievement through teacher professional development towards culturally responsive schools and classrooms. Enhanced achievement requires, of course, that students attend school, participate in classroom and assessment activities, and engage as active learners. Interviewees most often discussed the impact of Te Kotahitanga on these achievement-related behaviours more so than the specifics of achievement test scores or NCEA results.
The many Māori students with whom we met affirmed that those teachers who cared about them as learners had an impact on their performance in school and in their classes. Students identified teachers who demonstrated caring, listened to them and took the time to help them at school. These relational aspects were seen as important parts of respecting and understanding students as Māori. Students stated they were more likely to work harder if they had a positive relationship with their teacher:
He teaches heaps, shows us what to do, explains this. Makes it interesting by letting us find out how to do it, by making us use our brain.
I think it varies on like the class you're in and the relationship you have with your teacher. If you get along with the teacher, then they're more likely to help you and push you. Like they will come to you individually, if they're trying to help you, they'll come to you individually and make sure you understand the work and understand what you're doing. But if they're not, if they don't really pay attention to you, they'll just ignore [you]. They'll just leave you where you're sitting.
School personnel were emphatic about the importance of engaging students in the teaching and learning process as fundamental to any improvements in achievement. One teacher elaborated on improved attendance. She perceived that Māori students were now coming to her classes as a result of the change in relationships and the additional efforts she made to find out about her students, demonstrating caring and a real interest in their achievement:
At the beginning of the year there were quite a few that were truant, and not bringing their gear, but now they all participate [because] I got to know them better. I gave them some questionnaires to ask them about themselves and told them a bit about me, and we just got a relationship where they know I care that they are not participating or not in my class and they know I'll find them around the school or when I see them I'll ask them where they were… So they know they just thought it was easier to come to class, or they started enjoying our lessons. Because a lot of them didn't know each other at the start, and a lot of the activities that we did got them to know each other a bit better. So I think they were happier to work together and to be in the class... And a couple of them I put a lot of time in outside of class. I've fought for them to not be into special measures because they weren't coming to class. Because they were coming to my class and succeeding in my class it showed they were okay, they could be good in their other classes.
Many other teachers commented about the direct relationship between attendance and achievement:
The valuable outcome for students would be higher achievement. Less truancy, [so] that those students who avoid school will attend regularly. That as teachers we don't focus on things that we can't control, like what's going on at home, or whatever, that we just focus on what's going on in our classroom. So at the end of the day, better results and higher retention rates.
I think the most important thing is attendance at school, and for them to attend school regularly, and attend all their classes [and] would be feeling comfortable in the classroom. And the key thing for the teacher to do is to make those students feel comfortable in the classroom.
Most teacher comments were about socio-emotional outcomes and the extent to which students were made to feel comfortable in school so that they could learn:
Valuable outcome for students is they must be happy, and if they are happy they will be in class, and if they are learning they will be in class, they don't see that as a waste of time. For teachers, student achievement, when your student achieves, you know you are doing a good job, right, academically.
We're after achievement which can be interpreted in many ways but step one is them wanting to be here, and being in a classroom environment where they feel comfortable and they will come more days than they would otherwise.
Te Kotahitanga professional development is really linked into increasing student achievement and increasing student attendance by creating an environment that they are actually going to be more comfortable within that they will actually feel that it is more a school that they want to be a part of and attend and participate in and thereby raise their educational standards because attendance is a huge part of achieving that goal.
Teachers discussed how Māori students would achieve if classroom instruction is engaging. They attributed a growing awareness that teachers have the power to make instruction more engaging to Te Kotahitanga and took responsibility for making this happen:
I would say the engagement levels and the participation levels are much higher. For myself, within the project, my expectations now are of a hundred percent participation and a hundred percent engagement.
Principals had a great deal to say about achievement-related factors. Across the 22 Phase 3 and Phase 4 schools site-visited by the evaluation team, principals affirmed overwhelmingly that the project had made a difference to Māori student achievement through changes in the factors that led to enhanced outcomes. They saw Te Kotahitanga as an unqualified success in changing the climate of their schools for Māori students and teacher-student relationships in particular. They felt strongly that the project had resulted in changes in teachers as well as improvements in a range of socio-emotional, behavioural and achievement outcomes. Thus, principals attributed the improvements in achievement to changes in teacher behaviour that, in turn, led to increased student engagement in learning activities which then resulted in higher academic performance.
Principals discussed how achievement began with enhanced attendance and engagement, and they overwhelmingly perceived that Te Kotahitanga had led to changes in Māori student engagement, attendance and discipline:
[Te Kotahitanga also] looks at actual engagement within the classroom—the type of interaction with the teacher, the attendance, those sorts of things, the ability to compare attendance in different classes. One of the biggest issues identified is the selection of students to attend classes. They'll go to this teacher, this teacher's stuffed, that one, this one, and this one… really brought that into sharp focus. And that evidence when it's been shared in the co-construction [meeting], it has been "Oh!"
Look at this group of boys here—they were engaged for three-fourths of the lesson.
Principals recognised that engagement and attendance are key factors for achievement and maintained that Te Kotahitanga had had a major impact on whether students came to school and participated in learning:
What I'd like to hope will happen is that the Māori kids in particular—I was just about to say engage a bit—but instead that they are comfortable in the learning environment that we have, and it's not something that's foreign to them. Our classrooms have to be diverse by nature of our school population. We are very diverse. I'd like to feel that within each classroom at some stage, within each period where there are Māori kids, that there is something there that gives them a bit of flavour that "Hey we belong here" and that there is something for them, PI kids and Asian kids and so on.
I think that to be comfortable in school, and to want to come to school, to enjoy what is happening at school, to know that the teacher respects me, he wants me to achieve, he's going to go crook at me if I do something wrong but it will be at my behaviour and not at me that he's critical of. That's the vision I see. And whilst that's for Māori kids, I want it to be the same for all of them as well. Every kid that comes here I want them to feel comfortable in the place.
They concurred with the project focus on achievement of Māori students while emphasising their conviction that educational improvements for Māori led to and was associated with improvements for all students and for the entire school. They pointed out that Te Kotahitanga was actually a tool for having a positive impact on all students:
I don't hear any conversations of "why should we do this for Māori? I'm teaching a whole class." That kind of discussion I don't hear anymore, so I think that people have understood through TK what we are trying to achieve here. And feel comfortable that what they are doing is going to benefit other students in the class anyways.
School leaders talked about changes in retention that they attributed to Te Kotahitanga:
One of the things we've seen over the last few years is a change in the retention of Māori kids in school, and coming back for Year 12 and Year 13. We used to have a lot of those that would come back because they had nothing, and then they'd only need to survive a short period of time in Year 12 and then disappear. These students [now] tend to be staying right through, and when you talk to them now…they're staying…. For a lot of kids, [school] is probably the most stable environment that they're in. (Principal)
Particularly notable was the strong support for Te Kotahitanga from the Māori families at both Phase 3 and Phase 4 project schools who shared their views with us. This strong support was evident despite what they described as limited engagement with the school around Te Kotahitanga and varied levels of knowledge about what the programme is. Whānau contrasted their children's love for school and attending school against their own memories of schools as places where they had not been made to feel welcome:
Some of our whānau …they are whakamā about coming into school. School wasn't great for them.
I can see teachers who want to do well but are struggling in the classroom, but I see parents too, families who are struggling in other ways, low income, solo parents, pressure just over the fence… so there are other influences that we need to consider when we think about parents and whānau being involved. Some families don't feel comfortable coming into school, if they didn't do well. So how do we pull them in, that is where whānau can help and having more Māori faces in the school?
One mother described her personal struggles at school and how this affected her dealings with the school regarding her own tamariki:
Cause of the angers that I have from when I was going to school come back at you. That same time where I think nothing has changed; teachers are still the same, the principals, I mean that's the stuff that I was aware of. But when I come into a school I'm terrified, I mean it's really stressful for me. It's a lot different now, because I've got a different relationship with the teacher here, but [it's been] hugely stressful, and that's because when I come into school its because I've got to deal with this disciplinary problem, I've got to deal with bad behaviour, with possible suspension, expulsion, it's hugely stressful, and the impact of that reality is that parents don't know how to support their tamariki. You know, I'm talking about Māori parents [who] don't know how to support their children.
Generally, parents talked about the school and the teachers in extremely positive ways, and they were convinced that Te Kotahitanga was having a positive impact on their children's social and educational outcomes in comparison to previous practices:
I can see, you know, him moving on to stay to year 13, just purely because of that support and that work that's here. And the role models you know the people that they put in front of them, this is what you can achieve.
They have got a really strong relationship and the teachers are encouraging them. I don't think it's like in our days when you were just given a worksheet when you walked in a class; there you go sit down do your worksheet, hand it in at the end of the lesson, and no communication happens. And now the kids are coming home and saying they've done this and they've done that and they're allowed to speak their mind to a certain extent. When I first heard a couple of things I thought well you know that's giving a bit too much leeway to the kids but you see it's developed in them. They've come a long way.
One mother said that although she did not know a great deal about Te Kotahitanga, she had seen a big change in her daughter's attitude towards school in comparison with previous years:
[My] daughter didn't used to like coming to school, not interesting subjects. She used to say "Why can't teachers teach us something really outside in the world, not just in here, not teach the same old thing". She found it much more interesting this year, her grades have shot up. She used not to be interested, now she's one of the top students in her class. She can see results—certificates in year 11.
Finally, all the Chairs of the Boards of Trustees who were interviewed were enthusiastic about and committed to Te Kotahitanga as an approach to enhance Māori social and educational outcomes that was at the same time beneficial for all students—Māori and non- Māori. Two BOT Chairs commented on the extent to which there was systematic sharing of evidence about student achievement with the school community:
It would be good if we had feedback perhaps from this project or from Te Kotahitanga progress that was aimed at lay people, Board lay people, to reassure ourselves and reformed advocates about 'Yes it is making a difference, these are the moves bit by bit.
[Evidence] might be something that the lead facilitator has, and perhaps we would not get on a regular basis, but I have seen there has been data since the programme has been in place. It would be interesting to see, though, perhaps comparisons, just very rough comparisons to the other schools that have been involved. But I do have an idea that there is some of that information available already.
Data on achievement-related behavioural factors
Various achievement-related behaviours that could be influenced by Te Kotahitanga are explicitly referenced as part of Goal Setting in the Te Kotahitanga GEPRISP model. Anecdotally, school personnel, students, and whānau alike provided numerous examples of how these kinds of factors had been influenced for the benefit of Māori students. In particular, the following factors might be recorded systematically in future for Māori and other students in order to assess levels of student engagement and/or teacher expectations objectively:
(a) Attendance, showing decreases over time in unexcused absences from school;
(b) Retention, showing increased percentage of students returning to school the year following attainment of their 16th birthday, the time when they may legally leave school, as well as the numbers and proportion of Māori students at Year 13 compared to Year 11;
(c) Streaming placement, showing increased percentage representation in high and decreased percentage representation in low ability/achievement bands over time as such placements are affected not only by academic ability but also by teacher/school expectations and judgements about behaviour;
(d) Disciplinary statistics, showing decreases in number and percentages for stand-downs, suspensions, and exclusions.
We determined that these data were not available from either the Ministry of Education or the project so requested these data directly from the 12 Phase 3 schools for the 2004 and 2007 school years. With some exceptions, the schools were unable to provide us with these comparative statistics. One difficulty was that some data required particular capacity in the school's Student Management System (SMS): KAMAR can accommodate these data but was not implemented until fairly recently into these schools, replacing previous SMS systems that were apparently unable to accommodate the necessary data. According to milestone reports submitted to the Ministry by the schools, at least some of the Phase 3 schools still had not succeeded in establishing the necessary databases by 2008.
Similarly, attendance data were provided by only a handful of the schools, and most indicated that their previous SMS did not have the capacity for such data whereas KAMAR would now make it feasible to keep these data for evaluation across years. One school which has done this comparison reported that Māori attendance was at 82% for 2007; however, 2004 data could not be reported. The data reported for another school for both 2004 and 2007 were virtually identical for Year 10 students and showed a slight decrease in 2007 for Year 9 (with Māori student absences increasing from slightly fewer than 10% in 2004 to just over 13% in 2007).
Providing data on the percentage of Māori students in the highest and lowest ability and achievement bands proved equally problematic. One school indicated verbally that this comparison was available, but the information was not received despite our requests. The few schools that did respond were able to provide current data only. In one school, however, Māori students comprised 23% of the top stream in Year 9 and 18% of the top stream in Year 10 in 2004, whereas in 2007 Māori comprised 33% of the top stream in Year 9 and 21% of the top stream in Year 10.
Most schools were not able to provide us with a compilation of the percentage of Māori students who returned to school in the year following reaching their 16th birthday for 2004 and/or 2007.
Interventions other than Te Kotahitanga may also have had an influence on some of these data. For example, several schools initiated restorative justice programmes and other strategies to reduce the number of stand-downs, suspensions and exclusions from 2004 levels to 2007. At one school, a total of 97 stand-downs, suspensions and exclusions in 2004 was reduced to 11 across the three types by 2007; Māori students accounted for 100% of the disciplinary actions in 2007 in contrast to nearly 90% of the actions in 2004, but the reductions in numbers of students affected in each category was dramatic. Rather than 66 stand-down incidents in 2004, there were only 2 in all of 2007, for example. In two other schools, one school decreased these disciplinary actions from 43 in 2004 (of which 86% were Māori) to 30 in 2007 (of which 57% were Māori), while another school decreased disciplinary actions from 90 in 2004 (of which 47% were Māori) to 57 in 2007 (of which 43% were Māori). A fourth school was able to provide the comparison for stand-downs only from 2004 to 2007, not for suspensions and exclusions. The other Phase 3 schools did not provide these data within the evaluation timeframe.
These are the kinds of behavioural data that might be influenced by Te Kotahitanga. Although we were unable to assess these factors, they could be evaluated in future through systematic record-keeping in a consistent format across schools.
School reports on achievement outcomes
School personnel, family/whānau, and the students themselves discussed extensively their perceptions of the major impact that Te Kotahitanga activities had and continued to have on student achievement outcomes. They often referred to data shared with them at school level and also noted presentations by the Waikato research team that included information on changes in student achievement. Family members and BOT chairs had generally heard about achievement outcomes through a project newsletter distributed by the school to homes or at a meeting where the principal, lead facilitator and/or other school personnel shared success stories as well as specific achievement information on assessments and/or the NCEA.
School leader reports on achievement
All principals affirmed the importance of improving Māori student outcomes. They stressed that they had been motivated to participate in Te Kotahitanga from the beginning because of their personal concerns that Māori student aspirations were not being met in comparison to outcomes for other, non-Māori students at the school and nationally. Secondly, while they recognised and valued the focus of Te Kotahitanga on improving outcomes for Māori students and determined to keep that focus, they acknowledged that the programme also had a positive impact on outcomes for all students. Principals gave examples of changes they attributed to Te Kotahitanga:
We are seeing an improvement in that over the last three years, the percentage of Māori students achieving NCEA level 1 is now equivalent to the percentage of Māori students in that year group. (Principal)
In 2002, with our leaver data, 1 in 3 of our Māori students left with no formal qualification. In the 2006 data, we had not one Māori student who left in that category. Not one. And we're very, very proud of that. (Principal)
Since we've had Te Kotahitanga. over the last three years there's been less than five stand-downs or suspensions as a result of in-class behaviour. (Deputy Principal)
Similarly, other school leaders talked about changes to Māori student achievement:
If you have a look at 2004 I mean this is level one [of NCEA], this is the percentage, there's a big gain there. (Deputy Principal)
Principals varied in the extent to which they were able to provide student outcome data about the benefits of Te Kotahitanga for Māori students and for all students. Some emphasised that it was really 'early days' for student achievement changes to be evident, given that full implementation was still a work in progress at many schools and Māori students were being taught by both Te Kotahitanga and non-Te Kotahitanga teachers:
I believe it will all happen, but it is going to be—in talking with the Phase 3 schools—it has been not until the third year that they are getting any real significant change.
The limited achievement outcome data available may have been partly influenced by their reliance on a designated Deputy Principal whose position profile generally included responsibility for data collection, reporting and summaries provided to the Ministry of Education and for other purposes. Some principals also expressed frustration with data management system challenges that complicated their efforts to summarise data they did have so that it could be returned to the facilitation team and teachers in a timely way. These are clearly issues that would need to be addressed not only for this project, but for any initiative designed to enhance achievement and relying on assessment results to document effects.
At the Phase 4 schools as many as one-half to one-third of the teachers were still awaiting Te Kotahitanga training, thus it is perhaps not surprising that most principals focused their comments about student achievement outcomes for Māori on attitude shifts in the school rather than assessment results or other data on achievement. They mentioned assessments but in the context of changing expectations by school staff—especially teachers—rather than actual changes in student scores on particular assessments:
I've only been here, just coming up on three years, and we've been using asTTle data to share with staff. But it wasn't until the end of last year that I would say that most staff took cognisance of it. They had to use it at the beginning of last year, but they've done nothing with it. They just as much ranked or rated or verified their own stereotypes. [Teachers said things like] "Can't do anything with these kids" "Look at them! They're dumb!" "It's a waste of time, it's all their fault". While we were starting from of position of, here's the data and it's empowering them and all that, they still haven't changed their mind frames.
It wasn't until the end of last year, where the continual use of Te Kotahitanga that sometimes these pennies dropped. And I would say now most of the staff value that data as a part of what student's don't know and therefore what they need to know—as opposed to a justification for their achievement.
Principals saw major changes in how many teachers viewed Māori students and the expectations they had for them:
Yeah, "let's talk about achievement. Let's talk about learning. And let's talk about needs of students." As opposed to "They can't, they shouldn't, they won't be able to" type of deficit stuff. And all credit [is due] to Russell there, because he initiated that idea of the removal of the deficit theorising. It's very easy to fall into that, and just say "Oh they can't." You move away from that and start talking about learning, and it's encouraging.
And so I suppose in terms of our thinking- over this year for me, personally it may have started off as a project, but I gradually came to realise how very important it is. It's the best tool or best set of approaches for raising Māori student achievement that I've seen.
The importance of gathering data systematically on student achievement outcomes in a manner that enables teachers to use those data to enhance teaching and learning was high on the agenda of principals even where this was still a work in progress:
Sometimes, especially around the data, I think [teachers] were left to their own devices a bit too much. There was these expectations around about data that needed to be provided and I feel as though not enough support was given to them to lead that process.
It's no use introducing it here unless you think it's going to have some way of meeting the needs of Māori or being appropriate or whatever. So we've always looked at that data. But as a result of being on Te Kotahitanga, the data we are going to collect and analyse now for our junior kids is going to be better data, rather than broad brush stuff.
These principals were clear about the kinds of achievement and other outcomes they expected to attain for Māori students in the future:
I'd have to say student [outcomes] is the most important. That's what drives me, I want to see every student leave the school with a qualification, feeling confident in who she is and knowing what her next step is going to be out in the big world.
Well I'd like to see a lot more Māori students staying on, so retention is an important thing. We've got a pattern of Māori students—no matter what level they are at when they come in at year nine, whether they are in our top- high motivation class, or they have special learning needs—it seems that very few remain until [NCEA] level 3, to the end.
At the Phase 3 schools, where the project had a longer history and the vast majority of teachers had become full participants in Te Kotahitanga by 2006, principals and deputy principals also varied in the extent to which they could provide student outcome data about the benefits of the project for Māori and for all students. Some of the changes discussed by principals included behaviours associated with achievement:
One of the things we've seen over the last few years is a change in the retention of Māori kids in school, and coming back for Year 12 and Year 13. We used to have a lot of those that would come back because they had nothing, and then they'd only need to survive a short period of time in Year 12 and then disappear. These students [now] tend to be staying right through, and when you talk to them now…they're staying. For a lot of kids, [school] is probably the most stable environment that they're in. (Principal)
Since we've had Te Kotahitanga, over the last three years there's been less than five stand-downs or suspensions as a result of in-class behaviour. (Deputy Principal)
The Phase 3 school leaders were also clear, however, that their schools had been and were involved in various initiatives to enhance achievement, in addition to Te Kotahitanga, thus it was not really possible to attribute changes in student outcomes to any one particular programme:
The school was on a journey anyway, and Te Kotahitanga was just one tool of perhaps a number of different other projects that were also running, so it's not like the outcomes were a result of Te Kotahitanga—Te Kotahitanga has been a tool, but you've been on that journey… Te Kotahitanga was just one of those [initiatives] hanging off the umbrella. But without it, it certainly would have been longer. (Principal)
The Ministry is so disorganized in terms of what they're thrusting at the school like a lolly parade that you're almost in a protection position as a principal. You know it's really good that the numeracy and Te Kotahitanga, there's really good synergy between them, because it's cooperative learning strategies, it's data driven. (Principal)
There was discussion about how various initiatives were contributing to better outcomes:
We can't say that we're attributing this purely to Te Kotahitanga, because as a researcher you know that, but we've got academic counseling in this school in a big way, we've got what we call 'Restorative Thinking' for students whose behaviour has sort of, takes a bit of a dive occasionally--a whole programme around that--and we've got Te Kotahitanga. So we feel, or I feel, that those three programmes impact. Also we've got the bilingual unit, so that has to play a part in it. (Principal)
Another criticism was the fact that the initial successes of Te Kotahitanga in terms of literacy, for example, were linked specifically to Te Kotahitanga, but in fact during that time we had several strategies, several initiatives running in the school, and that was a combination of things. One of things [teachers] objected to is that it was tied specifically to Te Kotahitanga. (Deputy Principal)
Nevertheless, principals saw changes overall in aspirations, expectations and achievement outcomes for Māori students resulting either directly or indirectly from their schools' participation in Te Kotahitanga:
We are seeing an improvement in that over the last three years, the percentage of Māori students achieving NCEA level 1 is now equivalent to the percentage of Māori students in that year group.
In 2002, with our leaver data, 1 in 3 of our Māori students left with no formal qualification. In the 2006 data, we had not one Māori student who left in that category. Not one. And we're very, very proud of that.
Teacher and facilitator reports on achievement
Few teachers at either Phase 3 or Phase 4 schools referred to school-wide data as evidence on increased student achievement, and results of the teacher interviews suggest that schools were only just beginning to engage in data-based school-wide discussions of student outcomes. At Phase 4 schools, there was general awareness of NCEA achievement patterns at their schools and goals for improvement:
We have to get to an acceptable level of Level 1—figures should be 70% for the whole school. For Māori, I think it's really low—the Level 1 for whole the school is 51 % and 52%—when compared to others of the same decile its good, but overall not good—it's lower again for Māori students. I think its about mid 20%, very hard to accept as a teacher.
There were, however, more comments from teachers about student achievement outcomes and using data to inform teaching and learning at the Phase 3 schools:
Te Kotahitanga means to me, firstly, as individual teachers and as a school we have a collective vision to improve the academic results and emphasis, I think, on the results on paper of our Māori students. So, improve their learning experience, yes, sure, but ultimately at the end of the day the results on paper—that hard and fast statistics is, I guess ultimately, what it is about.
Facilitators and lead facilitators generally struggled with how to make the teacher professional development activities and feedback more data-based in providing teachers with timely data on student achievement that could be used to inform future teaching and learning activities. They considered that most data were primarily anecdotal, rather than based on evidence of impact on Māori student achievement; partly, this was the result of having only limited data on achievement available as part of the process:
Anecdotally we've got a lot of really good evidence here. The data is coming through. It's just slowly coming through. But I don't actually think we'll see an effective change through the data, we've got some time.
Evidence [was a challenge]. So what was the evidence that our teachers were bringing to co-construction, because we thought that that was pretty useless, some of the evidence, and we wanted more, meatier evidence to help our kids.
Some facilitators reported that some teachers were reluctant to share their own classroom assessment data:
My own experiences as a facilitator were that we had a real reluctance for teachers to share data. In that they tended to share impressions initially. They tended to share anecdotal, off the cuff; I'll talk while I'm at the table kind of conversations. Which were still useful in the initial stages, in that we were talking about people who basically had nobody asking any questions about what was happening in their classroom.
Missing across all schools was a regular and systematic achievement data feedback loop between the deputy principal responsible for these data and the facilitation team who might then be able to use the data in their work with teachers:
Data [are a challenge and we need] some system, perhaps from Waikato, on what we can do, how we can work with the data, collect it all. I'm not sure. I know we have to do that ourselves, but some direction would be helpful.
Facilitators explained that teachers did bring different types of assessment results and/or evidence of Māori student achievement to co-construction meetings:
At a co-construction meeting, the teachers are bringing their assessment results, what level the students are at. They're comparing; they might even look at students' books or projects or whatever, and so one teacher might bring the books and they might look perfect and another one is sitting across the table going "Oh my goodness … how do you get them…? You know, what strategies do you use to get them to, to do that work?"
We started to talk about, 'OK so how do we know that we are adding value? How do we know that where those students start and where the students end up in terms of the teaching and learning that's gone on?' [A science teacher] made the decision through the feedback meeting process that she wanted to implement pre-testing get a sense of where students were at. She started with the pre-test and the post-test and then she started to desegregate the scores so she was looking at, in terms of the teaching and learning that goes on in the classroom, how effective am I being with the Māori students? So what're the gain scores for Māori students and for non-Māori students? And does that tell me anything different? She then also looked at individual students and where she might have to provide additional support. And the other thing that she was doing was that at the end, after each pre test those results were shared with students, so the pre test results were shared with students and she would then identify the core pieces of work that needed to be done in terms of the learning outcomes from that unit. So it was very much that the pre test data was used to inform both teachers and students. And then the post test data, so students knew what their gain scores were because prior to that the culture in the classroom had been, the best marks was the best achievement. And she was shifting it from the gain from where you were to where you are now is what matters.
Assessment data on student achievement
Table 9 summarises available information [Phase 3] encompassing each of two major student groups—the Year 9-10 students currently involved in Te Kotahitanga and the Year 11-13 students who complete NCEA following instruction in both Years 9-10 by teachers participating in the Te Kotahitanga model. Our evaluation project was not contracted to conduct assessments of Year 9-10 student achievement. However, the results of two student achievement measures have been analysed for Year 9-10 students by the University of Waikato Te Kotahitanga research team, and preliminary results have been reported by the project for the Essential Skills Achievement (ESA) measure in 2004 and 2005 and for the asTTle measure for 2005 (Bishop, Berryman, Cavanagh & Teddy, 2007; Bishop et al., 2009). AsTTle data have also been reported in Waikato milestone reports up to 2008 at 8 of the original 12 Phase 3 schools; difficulties with their student management systems have interfered with the ability of 4 of these schools to record and submit asTTle results across this timeframe.
The ultimate goal of work carried out through Te Kotahitanga is to enhance Māori student achievement. Hence, it is relevant to investigate Year 11-13 NCEA achievement by Māori students as a function of project involvement during Years 9-10. Enhanced NCEA achievement would be expected if: (a) Māori students' achievement were positively affected by their learning experiences in the earlier Years 9-10; and (b) teachers transferred discursive and culturally responsive practices into their teaching in the senior school. There is no evidence regarding (b), but (a) can be evaluated at Phase 3 schools only. Specifically, analyses of longer term outcomes on the NCEA for students in Years 11-13 (levels 1-3 of the NCEA) require data from 2009 NCEA achievement as the first cohort of students experiencing full implementation of Te Kotahitanga starting in Year 9 could be completing Year 13.
|Measurement Type||Description||Year Groups|
|Year 9-10 Achievement (Te Kotahitanga measure)a|
|Essential Skills Assessment (ESA)||Information Skills, Finding Information in Prose Text (Secondary)|
Pretest and post-test reported:
|Assessment Tool for Teaching and Learning (asTTle)||Numeracy||
Pretest and posttest reported:
2005: 6 schools
|Year 11-13 Achievement (NCEA data from NZQA & MoE Benchmarks)b|
|NCEA attainment of 40+ credits in Year 11||Percentage of all Year 11 students who attained at least 40 credits at NCEA Level 1 or higher||Results can be reported across 2005-2008, recognising that 2008 is the first year in which Year 9 students exposed to full implementation in 2006 could have reached Year 11|
|NCEA Level 1 in Year 11||Percentage of Year 9 students who attain NCEA Level 1 in Year 11||Results can be reported across 2005-2008, recognising that 2008 is the first year in which Year 9 students exposed to full implementation in 2006 could have reached Year 11|
NCEA Level 1 Literacy & Numeracy requirements in Year 11
Percentage of Year 11 students who met both requirements and number/ percentage of Māori & non-Māori students who met Literacy & Numeracy requirements
Results can be reported across 2005-2008, recognising that 2008 is the first year in which Year 9 students exposed to full implementation in 2006 could have reached Year 11
|Mean # NCEA Lvl 1 credits in selected subjects in Year 11||English, Mathematics, Chemistry, Science, Geography, History, Economics, Physics||Results can be reported across 2005-2008, recognising that 2008 is the first year in which Year 9 students exposed to full implementation in 2006 would have reached Year 11|
|NCEA attainment of 30+ credits in Year 12||Percentage of all Year 12 students who attained at least 30 credits at NCEA Level 2 or higher||Results can be reported across 2005-2008 prior to full implementation, but 2009 data needed as date by which Year 9 students exposed to full implementation in 2006 could reach Year 12|
|NCEA attainment of 30+ credits in Year 13||Percentage of all Year 13 students who attained at least 30 credits at NCEA Level 3 or higher on the framework||Results can be reported across 2005-2008 prior to full implementation, but 2010 data needed as date by which Year 9 students exposed to full implementation in 2006 could reach Year 13|
|University Entrance (UE)||Percentage of Year 13 students who gained UE with required 42 NCEA credits comprised of at least 14 in each of three subject/s.||Results can be reported across 2005-2008 prior to full implementation, but 2010 data needed as date by which Year 9 students exposed to full implementation in 2006 could reach Year 13|
|Year of Attainment of NCEA Level Qualification||NCEA Levels 1, 2 and 3 Certificate attainment in follow up years as a percentage of school roll in Year 9 when each year group enters school—2-4 years later for Level 1, 3-4 years later for Level 2, and 4 years later for Level 3.||Results can be reported for Year 9 entrants in 2002-2006 as of 2008 benchmarks data, providing complete data only for 2002 and 2003 "baseline" year 9 entrant groups; 2004 and 2005 cycles not yet complete, and it's too early for the 2006 Year 9 entrants to be followed, which requires data from 2008 to 2010.|
|Highest Qualification Attained by School Leavers||Percentage of students leaving school who have attained a Year 12 qualification or higher, including NCEA Levels 2-3, CIE, IB, Accelerated Christian Exam, or other overseas awards||Results can be reported across 2005-2008 prior to full implementation, but 2009 data needed as date by which Year 9 students exposed to full implementation in 2006 could have reached age 16+ when they may legally leave school.|
Year 9-10 student achievement outcomes
Limited data are available regarding current achievement outcomes for Year 9-10 students whose teachers at the 12 Phase 3 schools have been involved in Te Kotahitanga since the training years 2004-2005 and what we would regard as full implementation years 2005-2006. The first teacher cohort received their first full year of training in 2004, so can be considered to be engaged in full implementation of the model in 2005. The second teacher cohort received their first full year of training in 2005, so can be considered to be engaged in full implementation of the model in 2006. By 2006, then, one might expect evidence of enhanced student achievement for Year 9 and Year 10 students. To measure Year 9 and Year 10 student achievement, Te Kotahitanga adopted the Assessment Tool for Teaching and Learning (asTTle) for use from 2005 onwards, and the agreement between the project and each school specifies use of this measure accordingly.
Achievement data using an agreed assessment should be reported by all project schools, but there have been difficulties in the adoption of asTTle as a standard measure across schools. Bishop and his colleagues note that there were "teething problems" with the introduction of asTTle testing (2007, p 173). Our review of a large sample of school milestone reports submitted to the Ministry supports this: Some Phase 3 project schools still had not implemented asTTle numeracy assessments in 2008. There has been criticism that the data reported to date may not be representative as they were from some of the Phase 3 schools only. Whether or not measures were implemented, assessments carried out, and data reported do not appear to have been influenced by the project or the Waikato research team, but instead appears to have varied as a function of school policies and administrative issues that are not project-related. According to milestone reports to the Ministry, some Phase 3 and Phase 4 schools were still struggling with their Student Management Systems (SMS) in 2008, including having made recent major shifts from one system to another (e.g., from MUSAC to KAMAR).
This is an issue that goes beyond Te Kotahitanga specifically, and the challenge is not unique to New Zealand secondary schools. Schools could benefit from systematic strategies and mechanisms at school level for feeding information on student outcomes into PD processes at the school; at present, most data systems are structured instead around Ministry of Education reporting purposes and needs.
Longer-term student achievement outcomes: Years 11-13
Te Kotahitanga is focused on professional development for teachers who are teaching Years 9 and 10. However, the majority of teachers who teach year 9-10 classes are also involved in teaching senior secondary subjects across Years 11-13. Thus, one might expect teachers to extend their understandings of culturally responsive pedagogies and effective teaching dimensions to how they approach teaching older students as well. However, whether or not teachers generalise new learnings to teaching practice in the senior secondary school is not within the scope of our evaluation project. Nevertheless, the students who attend Te Kotahitanga project schools for Years 9-10 could be expected to show the impact of Te Kotahitanga on their achievement later, when they move on to Years 11-13 and begin participation in the NCEA. To what extent is this kind of transfer realistic, and when might we expect it this to become evident? For the first Te Kotahitanga cohort—the Phase 3 schools—2004 and 2005 were the first two training years for teachers involved in the model.
Even for the first wave of Phase 3 schools, it was not until 2006 that all teachers interested in doing had been trained and were participating fully in Te Kotahitanga. This would mean that students starting Year 9 in 2006 would be the first student cohort experiencing full implementation of Te Kotahitanga at their schools that year and in the following year as Year 10 students in 2007. These students would commence NCEA Level 1 in Year 11 (2008) continuing onto NCEA Level 2 as early as Year 12 (2009) followed by NCEA Level 3 in Year 13 (2010) (see also Table 9).
Any investigation of the impact of Te Kotahitanga on long-term achievement outcomes must take these implementation timelines into consideration (see also Table 1 in chapter 1). The data reported here are preliminary, and we consider that further NCEA achievement analyses should await results from the 2009 and 2010 NCEA achievement data; this is particularly important as many students across New Zealand are currently completing NCEA Level 1 across both Years 11-12 rather than only in Year 11.
Furthermore, any results based on outcomes in years 11-13 must be qualified given the absence of direct evidence that teaching and learning in the senior subjects and years 11-13 reflect Te Kotahitanga practices: instead, teaching activities in these senior subject classes could reflect traditional practices and be driven by perceptions of the demands of the NCEA rather than modifications occurring as a result of direct intervention from Te Kotahitanga facilitation and co-construction teams.
Total credits attained by Year 11 students
The Benchmark Indicators provide an individual school report of the percentage of Year 11 students enrolled at that school who attained at least 40 NCEA credits at Level 1 or higher on the framework for each of the relevant years 2004-2007. Table 10 and Figure 1 show the results of a comparison between the 12 Te Kotahitanga and 12 comparison schools in each of these four years.
|12 Te Kotahitanga Schools||12 Comparison Schools|
|Year 11 in 2004||76.1||76.2||9.2||2.7||77.7||79.6||11.5||3.3|
|Year 11 in 2005||77.6||78.6||7.5||2.2||78.4||78.7||7.5||2.2|
|Year 11 in 2006||84.4||85.4||6.2||1.8||82.1||83.0||9.4||2.7|
|Year 11 in 2007||83.6||84.1||6.5||1.9||84.2||83.2||6.8||1.7|
|Year 11 in 2008||85.3||86.4||5.2||1.5||84.5||85.6||7.8||2.3|
Figure 1: Percentage of all year 11 students attaining 40 or more credits at Level 1 or higher, 2004–2008
Overall, Year 11 students at the Te Kotahitanga schools were performing slightly but not significantly below those at the comparison schools in 2004, with the percentage of students attaining 40 or more credits rising gradually for all students; in 2008, this percentage is slightly higher at Te Kotahitanga schools. None of these differences are statistically significant.5 Again, these analyses represent all students, Māori and non-Māori, at the schools, as data in the benchmarks indicators do not include a breakdown by ethnicity.
Table 11 and Figure 2 illustrate the percentage of all Year 9 entrants who attained NCEA Level 1 in Year 11 in each year from 2004 – 2008. These data show that the percentage of Year 9 entrants from 2002 who attained NCEA Level 1 in 2004 at 38.1% at the Te Kotahitanga schools was lower than the percentage in the comparison school sample at 43%. By 2008, however, when one would expect to see the impact of Te Kotahitanga, NCEA Level 1 attainment at the Te Kotahitanga schools showed a greater gain in the percentage increase across these years, so that the two sets of schools are nearly identical in Year 11 student performance with respect to gaining NCEA Level 1 (51.6% versus 52.6%). These data are not disaggregated by ethnicity in the Ministry of Education Benchmarks database, so we are unable to analyse this outcome for Māori students only to compare the two school cohorts.
Note also that during this same timeframe, the national average of Year 9 entrants attaining NCEA Level 1 in Year 11 shows an gain of 6.7%, at 48.9% in 2004 to 55.6% in 2008. The 2008 overall percentage is slightly higher nationally than for the 24 Te Kotahitanga and comparison schools, but the national percentage gain across these years (6.7%) is lower than the percentage gain shown at the 12 comparison schools (9.6%) and only half the percentage gain across these years at the 12 Te Kotahitanga schools (13.5%). Thus, these data indicate that overall student achievement is being affected at the Te Kotahitanga schools beyond what is otherwise happening in schools nationally and provide support for the claim that what is good for Māori students is good for all students.
|Year 11 Students||12 Te Kotahitanga schools||12 Comparison Schools|
|Year 11 in 2004 as % of Year 9 entrants in 2002||38.1||35.2||15.4||4.5||43.0||42.5||15.2||4.6|
Year 11 in 2005|
as % of Year 9 entrants in 2003
Year 11 in 2006|
as % of Year 9 entrants in 2004
Year 11 in 200|
as % of Year 9 entrants in 2005
Year 11 in 2008|
as % of Year 9 entrants in 2006
Figure 2: Percentage of Year 9 entrants achieving NCEA Level 1 in Year 11 at 12 Phase 3 Te Kotahitanga and 12 comparison schools
It is relevant to note that 2005-2006 was a major professional development (PD) year for the NCEA in New Zealand secondary schools, and it is likely that this initiative with additional funding provided for school-specific activities would also have an impact on teachers and, in turn, student NCEA performance (Starkey, Yates, Meyer, Hall, Taylor, Stevens, & Toia, 2009).
Level 1 NCEA Literacy and Numeracy attainment in Year 11
Another source of information on student achievement in Year 11 at Level 1 of the NCEA provided by the benchmark indicators is the percentage of all Year 11 students who met both the NCEA Level 1 Literacy and Numeracy requirements in that year. Table 12 and Figure 3 show the results of these analyses across the same 2004-2008 years. As with the previous data, these analyses represent all students, Māori and non-Māori, at the schools, as no breakdown by ethnicity for these statistics is available in the benchmark reports.
Year 11 |
|12 Te Kotahitanga Schools||12 Comparison Schools|
Interestingly, two of the Phase 3 schools showed significant decline on this variable from 2007-2008; a small school declined from over 80% to just under 60%, and a large school declined from nearly 85% to under 60% as well. Clearly results at these two schools are affecting the 2008 average Literacy and Numeracy achievement in Year 11. For project purposes, it would be important to identify factors associated with drops such as these. While the overall percentage also declined in 2008 at comparison schools, this change was more consistent across schools rather than reflecting extreme drops at only some schools.
Figure 3: Percentage of all Year 11 students who met Year 11 Literacy and Numeracy requirements in 2004, 2005, 2006, 2007 and 2008
In addition to these aggregated data summaries available from the Ministry of Education benchmark indicators, we obtained directly from NZQA the data on attainment of both Level 1 Literacy and Level 1 Numeracy separately for Māori and New Zealand European students, for the years 2005-2008 (see Tables 13-14).
|Comparison||NZ European Students||Māori Students|
|Te Kotahitanga Comparison||590||558||94.6%||336||268||79.8%|
|Te Kotahitanga Comparison||646||625||96.7%||332||283||85.2%|
|Te Kotahitanga Comparison||863||821||95.1%||577||478||82.8%|
|Te Kotahitanga Comparison||1,023||899||87.9%||797||577||72.4%|
In 2005, a slightly lower percentage of Māori students were meeting the literacy requirement in Year 11 compared to those attending the comparison schools. All schools show a slight increase in NCEA Level 1 performance for both Māori and New Zealand European students in 2006, then a decline in 2007 and again in 2008.
Interestingly, these data also reveal dramatic increases in the numbers of students enrolled in Year 11 and those attaining NCEA Level 1 in Year 11 from 2006-2008 at these schools—reflecting retention at secondary level. For European students, the Year 11 numbers increased from 590 in 2005 to over 1,000 in 2008. Both the Te Kotahitanga and comparison schools show a numerical increase, but the increased retention of students in year 11 is proportionately highest across both student groups at the Te Kotahitanga schools.
The contrast is particularly evident for Māori students at the Te Kotahitanga schools. In terms of real numbers, there is an average increase in the Year 11 Māori student enrolment of approximately 250% across this 4 year period, with the number of Māori students enrolled (or retained) in Year 11 increasing from 2005 numbers of 376 at the Te Kotahitanga schools and 336 at the comparison schools to 996 and 797, respectively, at these schools in 2008.
These data suggest that a broader range of students were being retained at all schools rather than revealing a pattern whereby lower achieving students left school without attaining this benchmark.
As for literacy, the increase in numbers of students in both groups attaining numeracy L1 is very large, so these results overall are extremely positive, showing considerable improvement in student achievement across 2005-2008. There is a decrease in the percentage of Māori students meeting the numeracy L1 requirements at both the Te Kotahitanga and non-Kotahitanga comparison schools. For NZ European students, there is virtually no change across this time period for the percentage attaining numeracy L1 at the Te Kotahitanga schools compared to a slight decrease for the non-Te Kotahitanga schools.
|Comparison||European Students||Māori Students Only|
|Te Kotahitanga Comparison||590||565||95.8%||336||297||88.4%|
|Te Kotahitanga Comparison||646||627||97.1%||332||304||91.6%|
|Te Kotahitanga Comparison||863||830||96.2%||577||527||91.3%|
|Te Kotahitanga Comparison||1,023||913||89.2%||797||643||80.7%|
NCEA level 1 credits attained in selected subjects by year 11 students
We also examined the mean number of NCEA credits attained by Māori and non-Māori students in selected Year 11 subjects, including three core subjects—English, Mathematics, and Science—taken by all students. Outcomes for two elective (non-required) subjects—History and Physics—are also reported. Tables 15 and 16 report the results of these analyses, including statistical tests for significance of differences possible as these data from NZQA are based on individual student numbers.
These results reveal considerable variation across subjects. Both Māori and European students showed a small decline across the 2005-2008 years in attainment of credits in English, with no significant differences between Te Kotahitanga and comparison schools in 2008. There were differences in Mathematics in 2008 favouring the Te Kotahitanga Phase 3 project schools; there were also significant differences in the first year of implementation 2005 though the significance level favouring the Te Kotahitanga schools increases over time. Achievement in Science was significantly lower for Māori students in the Te Kotahitanga schools than in the comparison schools in 2005. But by 2008 this pattern had reversed, with Māori students achieving significantly more credits in Science in the Te Kotahitanga schools than in the comparison schools.
In the elective subject of History, European students at Te Kotahitanga schools achieved more credits than their counterparts at non-Te Kotahitanga schools, but there were no significant differences for Māori students in this subject at follow-up. In the elective subject of Physics, both Māori and European students are achieving significantly more credits at the Te Kotahitanga schools. These findings could reflect rising expectations whereby students are being encouraged to sample widely across the curriculum when they reach NCEA subjects in Year 11.
Subject with |
Year / Te
|European Students||Māori Students Only|
|European students||Māori students only|
All Year 11 achievement analyses assume that teachers and students have ongoing impact from what they experienced in Years 9-10. There is of course no evidence that this is so, which would require gathering data in the senior school which is outside the scope of our contract.
Attainment of University Entrance in Year 13
As noted throughout the report, it is too early for the direct impact of Te Kotahitanga on Māori students in Year 13 to be evaluated; this could only be done once the initial student cohorts have completed their secondary education. Nevertheless, we include in our report information on the attainment of University Entrance (UE) at Te Kotahitanga schools in comparison to the comparison schools in order to provide evidence regarding whether or not it could be claimed that the presence of this project in schools was associated with either enhanced or depressed performance at graduation, across all students. Furthermore, achieving UE is widely regarded as a sign of enhanced student outcomes. As one principal put it:
The thing is, Māori students are just like any other group of students, they are perfectly capable of getting Level 3 and going to university. So I would like to see a lot more of University Entrance happening.
Under NCEA, the granting of UE is automatic based on student results in Year 13, rather than being dependent on individual intentions to enter university as was previously the case under New Zealand Bursary. Thus, the proportion of students attaining UE over time can provide a measure of progress on overall student achievement after the final year of schooling.
In order to gain UE, a student must pass a minimum of 42 credits overall at Level 3 or above comprising at least 14 credits each in two subjects approved for university entrance and 14 credits in a third approved subject or interdisciplinary cluster of subjects; students must also attain the requisite number of literacy and numeracy credits. Because of different patterns of study, a candidate can gain NCEA Level 3 without gaining University Entrance and vice versa by meeting alternative qualification requirements. Table 17 and Figure 4 present the mean percentage of Year 13 students who gained University Entrance (UE) from the total Year 13 roll in Te Kotahitanga versus the comparison schools from 2004 to 2008.
|12 Te Kotahitanga Schools||12 Comparison Schools|
Figure 4: Percentage of all Year 13 candidates to gain University Entrance by 2004, 2005, 2006, 2007 or 2008
As with other Year 13 achievement results, these results are affected by the overall Year 13 student roll and students who have left school prior to Year 13. In the case of University Entrance, students less likely to be completing requirements towards gaining UE are those mostly likely to have left school by the final year—hence resulting in an increase in the proportion of Year 13 students gaining university entrance. These data reveal that the percentage of Year 13 students gaining UE increased from 2004-2008 in all 24 schools, following a slight decline from 2004 to 2005. Z-tests using differences of proportions revealed statistically significant differences, with a significantly higher percentage of students attaining UE each year from 2004-2008 at the Te Kotahitanga schools. Results for the Te Kotahitanga student cohorts in 2009-2010 are needed to examine specific effects associated with the programme. Interestingly, the national percentages of Year 13 students gaining UE across all secondary schools was 36.9% in 2004 and 43.6% in 2008, a gain of 6.7%, only slightly higher than that achieved by the Te Kotahitanga schools across this same four year time period (6%).
In summary, our evaluation has been able to examine Year 11 student outcomes associated with full implementation of the model at Phase 3 schools. While important as a first step, improvement in Year 11 outcomes is not sufficient. Future achievement goals for all students, including Māori, include at least the attainment of a Year 12 qualification (Ministry of Education, 2009). Further, the ultimate goal reflecting high expectations must encompass NCEA level 3 attainment or its equivalent.
Key findings: Impact on students
- Students reported enhanced valuing of their identity as Māori learners and increases in culturally responsive practices at most schools, and the perceptions of school personnel and whānau provide additional support for growing appreciation for Māori cultural identity in schools. Māori students were proud of Māori culture and identity and felt that, on the whole and in most schools, they were able to "be Māori" in school rather than having to leave that identity outside the school entrance in order to succeed academically.
- Students were able to articulate how teachers showed they valued them as learners and as Māori, and they were able to discuss how teachers had changed in establishing positive relationships with them as learners. They emphasised the importance of teachers' caring about them as persons to support their learning. They commented on how difficult it was for them to care about how well they did and do the work in classes where teachers made it clear they did not. However, there were still perceptions among Māori students in a few schools that a 'double standard' continued to exist whereby Māori students were singled out and disciplined for behavior ignored for students from other cultural groups.
- Students discussed ways in which the school as a whole either did or did not demonstrate valuing of Māori culture and language. They gave examples such as use of powhiri, kapahaka and waiata, and they were able to define places and people—such as the Te Kotahitanga room, the marae, and Māori teachers—who helped them to 'feel Māori' at school in a positive way.
- Interviews with whānau affirmed that their children view school positively, loved coming to school, and had greatly improved their attendance and participation, which many contrasted with their own more negative memories of schools and schooling.
- Teachers, principals and other school leaders affirmed that their own expectations had been raised for and relationships improved with Māori students, and they attributed this shift to Te Kotahitanga rather than to the ETP specifically. Teachers generally did not use the language of the ETP when they discussed Māori learners, expectations, relationships and pedagogy in the classroom. While they noted the relevance of culture to teaching and learning, few discussed specifically that the development of a culturally-grounded identity was an educational outcome for students. The work of facilitators with teachers may require more explicit focus on culturally responsive pedagogies and how to support learning grounded in students' Māori identity towards adding to and enhancing existing conceptions of good teaching. This will also require more dialogue with the Māori community.
- Teachers, facilitators, principals and other school leaders reported improvements in student attendance, participation, motivation, and engagement in school and classroom learning activities which they attributed to Te Kotahitanga.
- There is numerical evidence of enhanced student retention and increases in Māori student enrolment in the senior school and NCEA credit attainment at Year 11 for Phase 3 schools in comparison to 12 matched comparison schools.
- Systematic comparisons of Year 11 student performance between the 12 Te Kotahitanga Phase 3 and 12 matched schools reveal higher increases (gain) across 2004-2008 in the percentage of Year 9 entrants attaining NCEA Level 1 in Year 11; Te Kotahitanga schools also evidenced twice the increase in this percentage gain than the average gain nationally. Comparisons also reveal lower achievement outcomes for literacy and similar achievement outcomes for numeracy at NCEA level 1 in 2008.
- Systematic comparisons of Māori student NCEA achievement at Te Kotahitanga Phase 3 schools compared to matched schools reveal statistically significant differences favouring the Te Kotahitanga schools in mathematics, physics and science, and no differences across the two groups of schools in English and history.
- Te Kotahitanga schools were associated with a statistically significant higher mean percentage of the total school population at Year 13 who gained University Entrance in comparison with the 12 matched comparison schools.
- Note, however, that these analyses are for two sample sizes of 12 that are quite small, hence achieving statistical significance would require greater numerical mean differences than would be the case with a larger sample.