Evaluation of He Kākano: Professional development for leaders in secondary schools 2011-2012

Publication Details

This report details the bicultural approach of the He Kākano evaluation, methodology, data analysis, key findings, strengths and weaknesses and recommendations for future school leadership professional learning programmes intending to promote educational outcomes for Māori student success as Māori.

Author(s): Anne Hynds, Luanna Meyer, Wally Penetito, Robin Averill, Rawiri Hindle, Marama Taiwhati and Flaviu Hodis with Susan Faircloth, Jessie Hetherington Centre for Educational Research, Victoria University of Wellington.

Date Published: January 2014

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Executive Summary

This evaluation investigated (a) the effectiveness of He Kākano in meeting programme goals and evidence of shifts against baseline data, (b) the effectiveness of the delivery and implementation of He Kākano, and (c) ways to strengthen the design and implementation of He Kākano. Other key objectives for this evaluation were to identify examples of effective school-based leadership practices and provide new learning about effective leadership and professional learning in secondary schools.

The evaluation developed agreed indicators in collaboration with the Ministry of Education, the professional learning providers, and with input from our National Evaluation Advisory Group. These included key indicators along with a schema for types of evidence including experiential knowledge (variable), statistics (verifiable data returns), and research and evaluation (quality assured).

This final report details the bicultural approach of the evaluation, methodology, data analysis, key findings, strengths and weaknesses, and recommendations for future school leadership professional learning programmes intending to promote educational outcomes for Māori student success as Māori.

Methods Overview

The independent evaluation utilised bicultural perspectives—Māori and non-Māori—and mixed methods with both quantitative and qualitative data and analyses. The project evaluation team included an indigenous international scholar with recognised expertise in culturally responsive school leadership who participated in selected evaluation activities including visits to schools and conducted an independent review of the draft Final Report prior to final revisions and submission to the Ministry of Education (see Appendix 1). The evaluation procedures and measures, including all protocols for data collection, were reviewed and approved formally by the Victoria University Human Ethics Committee for Education. Multiple sources enabled triangulation of data towards valid interpretation of findings.

Data sources across all project schools included: (a) annual school and school leader surveys to analyse change over time in school leadership and wider school capability to engage in evidence-based inquiry to develop culturally responsive leadership practices, school processes (including governance) and other practices towards building success for Māori students; (b) document analyses including successive school action plans with a focus on goal setting related to Māori student educational and achievement-related outcomes; and (c) statistical analyses of NCEA achievement data across several school years to the latest MOE benchmark data available (2011) comparing selected outcomes of He Kākano schools, Te Kotahitanga schools, and schools nationally.

A nationally representative, purposive sample of nine case study schools was identified, and these schools were visited by the project evaluation team in each of the two project years for more intensive, on-site data collection evaluation activities. These included: (a) observations of culturally responsive pedagogies in classrooms (year 1); (b) observations of co-construction towards culturally responsive leadership in schools (year 2); (c) annual surveys of student attitudes towards their schools and their learning, analysed for Māori and for all students on factors including achievement motivation, teacher affiliation, cultural responsiveness, school safety, and discrimination; and (d) individual and focus group interviews with school leaders (principals, DPs/APs), Manutaki (Regional Coordinators), middle level school leaders (Heads of Departments, Deans), teachers (including a range of curricular areas and the Head of Māori), and Māori students and whānau. Two to three members of the evaluation team carried out each of the school site visits; at least one was Māori and fluent in te reo for adherence to appropriate cultural and language protocols.
To ensure that evaluation personnel had knowledge and understanding of the project approach, each member of the evaluation team attended at least one regional wānanga. Evaluation team members also met with key project personnel on several occasions at the project’s Tauranga office, at the Ministry of Education, and at Victoria University in Wellington in conjunction with several meetings of the evaluation’s National Evaluation Advisory Group.


Constraints

This evaluation comprises a snapshot of ongoing project professional learning activities that occurred nationally and were conducted in schools across the two years of project activities. Data fatigue prevented comparison of some data sets. In the absence of baseline data and systematic comparison across similar schools, these evaluation results cannot be regarded as experimentally validated with the exception of the longitudinal NCEA achievement data where a quasi-experimental design was employed to analyse early trends in these data. Further, findings reported here cannot be attributed solely to the effects of He Kākano—whether positive or negative—as schools reported that multiple programmes were ongoing which could affect student educational outcomes. Finally, issues raised here are not necessarily specific to He Kākano schools or a function of He Kākano leadership activities but rather reflect the general context of school system factors affecting learning and teaching within secondary schools in Aotearoa New Zealand.

Key Findings

Positive results indicated evidence of impact through increased participant perceptions of effectiveness and culturally responsive leadership for school leaders as a function of their engagement in the He Kākano professional learning programme. A comparison of the two School Leaders surveys (2011 and 2012) indicated that principals, deputy principals, assistant principals, heads of departments and deans reported increased awareness and enhanced understandings of culturally responsive schooling and leadership, and systems to support Māori students. These analysed survey results indicated improved perceptions of “effectiveness” across all groups’ improved optimism in terms of students’ achievement/success, and increased perceptions of school leaders’ use of evidence. Participants who responded in the two School Leaders surveys reported an increased use of statistical research and evaluation and experiential knowledge. Participants were still most likely to use experiential knowledge; however, other forms of data use have increased overall.

These survey results were supported by key themes that emerged from analysis of case study school interviews. School leaders across case study schools reported enhanced understandings of their own relational positions, values and beliefs towards Māori students and their communities; increased awareness of Māori students and their current achievement levels; shared responsibility for Māori students and their achievement; increased understandings of the need for goal setting, planning for improvement, collection and analysis of evidence/data related to Māori students’ achievement; and an increased awareness and focus on classrooms/goal setting through in-class observations, professional learning, and appraisal. These shifts in school leaders’ awareness are widely regarded as the foundation for effective school-based leadership practices.

Interview results also indicated that key processes aided a change in school leaders’ attitudes and perceptions. These included the impact of wānanga, and being together on the marae, and the relationships between school leaders and key project personnel.

There were, however, mixed results and contradictions across analysed data sets. Particular challenges included the variability of professional learning implementation and participant engagement as reported by school leader participants across case study schools. Related to this theme of variability was “picking, choosing and gate-keeping”. There was a sense that senior school leaders could pick or choose aspects of the model to implement which they believed most suited their school’s context. Associated with this was a variability of engagement that also indicated that not all senior and middle school leaders had participated in the He Kākano professional learning programme. A lack of engagement could prompt lack of ownership for change, and even resentment. There is a tension between the “one size fits one” model of programme design that focuses on context and flexibility and school leaders making decisions that limit the effectiveness of the He Kākano professional learning model and inhibit collective learning and agency. Evidence from Manutaki interviews indicated that these key project personnel viewed the project model as developing over time, partly due to its exploratory nature. Key reported challenges around their role had been the development of consistent processes and implementation requirements across schools. An operational challenge for evaluation of effectiveness was a change in key project personnel and change in emphasis for the professional learning and development (PLD) model for use in school action planning from the GPILSEO model in 2010 to the Culturally Responsive Leadership model in 2012.


New learning about effective leadership and professional learning in secondary schools

A key objective for this evaluation was to provide new learning about effective leadership and professional learning in secondary schools. There were key challenges and tensions that emerged from data analysis that have implications for effective leadership, professional learning, and change across secondary schools. Activities by well-intentioned school leaders and personnel focused on improving Māori student achievement may have highlighted challenges for students, rather than emphasising the challenges needed for school leadership and school change. Although there are constraints related to the overall representation of Māori students and whānau community members, interview evidence highlighted the presence of emphasised deficit messages about Māori students and their achievement levels. Analysed interviews from whānau members and Māori students also indicated perceptions of a lack of partnership and constraints on the development of relational trust between these Māori community members and school leaders. Key messages are needed that focus on changes to the school policies and practices that are currently responsible for under-serving Māori students. Otherwise, there is serious risk that schools are communicating primarily deficit messages that Māori students alone are responsible for continuing disparities and inequities in achievement outcomes.

Other key challenges emerged from data analyses that have implications for the development of effective leadership and professional learning in secondary schools. An analysis of observations of both in-class teaching (2011) and co-construction hui (2012) provided a “snap-shot” of dialogic and discursive teaching and leadership approaches used in classrooms and teacher meetings within case study schools. Overall, observation results indicated limited use of dialogic teaching and leadership practices across classrooms and teacher meetings. Interview evidence also highlighted variability of senior school leaders’ knowledge of culturally responsive pedagogies that have an evidence base and are known to be highly effective for Māori learners, as well as limited knowledge of effective appraisal and professional learning approaches and systems to support change in classrooms. This was particularly important for senior leaders as they are charged with conducting teacher appraisal and influencing the quality of professional learning systems for teachers across schools. Qualitatively analysed comments across the Principal, Deputy Principal and Assistant Principal School Leaders 2012 surveys indicated that these participants perceived staff as being a major barrier to the development of culturally responsive leadership. It is not clear that these leaders had interrogated the influences of staff resistance, questioned the effectiveness of existing professional learning systems within their schools for teachers or saw that they could influence the learning culture of their organisations through more effective pedagogical and relational leadership practices. Such evidence indicates that more needs to be done to create co-constructed, challenging and effective learning environments for students, their teachers, and school leaders.

Finally, a key challenge for effective school leadership is the ability to build relational trust and cultural competences across diverse student and teacher groups whilst ensuring safe and inclusive learning environments for all. This requires a shared vision of success as well as a valuing of cultural competency and leadership that is prompted within schools and across our society. Although the comparison of 2011 and 2012 student survey data was largely unremarkable, showing virtually no change on key survey factors across the two years for Māori students, there was one statistically significant shift (more negative in 2012 than in 2011) for NZ European students on the Mainstream Safety or the extent to which European students reported that they felt safe in their schools. It may be that these students were responding to changes that they perceived across case study schools, including an increased use of te reo me ona tikanga and culturally responsive practices. Qualitative comments made in surveys by some NZ European students expressed negative views in the use of te reo me ona tikanga. Negative stereotypes, deficit views and prejudice towards Māori students were also reported in interviews with Māori students in four of the nine case study schools. Further research and professional learning is needed to ensure the development of relational trust between Māori and non-Māori student groups related to cultural competency and citizenship in Aotearoa New Zealand. This is needed for the provision of safe learning environments for all student and teacher groups across secondary schools.

Summary of Key Strengths and Challenges Associated with He Kākano

Strengths

  • Principles of culturally responsive school leadership were generally clearly articulated and communicated to school leadership
  • Project wānanga and school-based activities were well received by school leader participants, demonstrated also by high retention of schools participating in the project over time
  • There was evidence of increased and enhanced awareness of school leaders’ own relational positions, values, and beliefs towards Māori students and their communities
  • There were improvements to school leadership monitoring of key educational outcomes for Māori students accompanied by implementation of more effective data systems which, over time, should enable schools to share data with school personnel, students, the wider school community, iwi, and whānau.


Challenges

  • An operational challenge for evaluation of effectiveness was a change in project leadership personnel along with a change in emphasis for the recommended model for use by schools in their action planning, shifting from the GPILSEO model in 2010 to the Culturally Responsive Leadership model in 2012.
  • The tension between the project’s promotion of “one-size-fits-one” and school leaders’ ability to pick and choose aspects of the model for implementation, which can limit engagement of school staff and create resentment. The PLD focus on school leaders’ attendance at the series of regional wānanga and input from the national project team for professional leadership development can be a risk to sustainability if individual schools become overly dependent on outside guidance rather than developing internal capacity and sharing of expertise across schools.
  • The tendency for educational stakeholders at all levels to interpret school efforts to enhance educational outcomes for Māori through deficit perspectives about Māori student underachievement rather than the intended message that the project was about enhancing the effectiveness of under-serving schools to enable Māori students to achieve educational success as Māori.
  • The variability of relationship between regional Manutaki and school leaders which could impact on the implementation of the model within each school. Manutaki needed advanced skills, knowledge and dispositions encompassing Māori cultural knowledge relevant to schools and their communities as well as secondary education expertise necessary for practical input for school implementation of principles of culturally responsive leadership beyond simply awareness of actions and school change.

Key Recommendations from the Evaluation

For future development towards enhancing this priority goal nationally, our recommendations for delivery and implementation to schools are:  
  1. Model Design:  An agreed model for Culturally Responsive Leadership is needed that is theoretically sound, supported by evidence (e.g., BES), is user-friendly for necessary understandings across key constituents, and that has direct links to policy and practices flowing from the model (i.e., not a stand-alone graphic but linked directly to school leadership policy, school systems change, and practice).
  2. Partnership with Māori:  The overarching aim of this professional learning programme for culturally responsive leadership is to enhance support for Māori students achieving educational success as Māori. This aim requires partnership with Māori throughout, including with students, whānau, hapū and iwi. Schools and school leaders will require additional support and advice to develop a meaningful partnership model that goes beyond the present strategies that are primarily information sharing.
  3. Aspirational not Deficit Messages:  Because of the tendency to interpret school efforts to enhance educational outcomes for Māori as being driven by deficit perspectives (as interpreted by students, whānau, and school personnel), clear (perhaps even scripted) messages are needed to reframe this discussion as aspirational and accompanied by high expectations.
  4. Professional Learning Components:  Components of the professional learning  programme must be clearly identified and implemented reliably. For example, there should be clear expectations regarding which school leaders participate in various components and how their participation should result in key actions that are individually accountable.
  5. School Action Planning:  Schools should be provided with a template for reporting that is user-friendly and can drive aspirational goals for students alongside accountable, action-oriented aims for key personnel.
  6. SMART Tools:  Minimal data collection and reporting requirements should be enforced at the individual school level, within the regions, and nationally. Duplicative data collection should be avoided (data fatigue), and measures used must be psychometrically valid. This would not prevent a small set of appropriate measures being available to allow some school choice (e.g., different assessments of literacy), but the choices should be limited, not open-ended, so that national comparisons can be undertaken and longitudinal analyses can be carried out to examine impact.
  7. Student Outcomes:  Initiatives that have as their ultimate goal the enhancement of student outcomes should be required to report attainment of enhanced student outcomes (social and academic). A defined set of possible outcomes to be measured and monitored could be provided from which schools may choose, but there must be more rigorous implementation of monitoring student progress rather than continuing to deliver multiple initiatives while monitoring only process factors (e.g., participation by school personnel).
  8. Project Personnel Roles and Responsibilities:  Clear role descriptions should be in place for different project roles at both the national and school levels. There should be explicit expertise requirements for key roles, including the levels of knowledge of secondary education, cultural expertise, and professional learning approaches needed for a project such as this one. Where staff lack a particular aspect of expertise needed to do the job well, there should be additional training provided and required. Finally, individual key project staff must ensure that their PL activities are consistent with the agreed model and approach. Where supplementary activities are offered to schools, these must be of such a nature that they can be offered and delivered to all participants. If there is slippage regarding which are in fact the critical components of PL provided to participants because of additional personal provisions, the initiative cannot be costed and evaluated with confidence.

Contact Us

Research
For more information about this publication please email the:  Research Mailbox