Summary

Introduction

This report is drawn from an evaluation undertaken for the Ministry of Education (Ministry) to determine the extent to which a school leader's participation in an NLC (Network Learning Community) has impacted on the curriculum design and review processes in their schools. It provides a brief background to both the NLC initiative and the evaluation before summarising findings related to the four main evaluation questions and the implications of these for the initiative moving forward.

More detail is presented in the companion reports. As well as the main report for this evaluation a detailed technical report is available. This technical report contains analyses of two online surveys and ten case studies. In addition, abbreviated forms of each of the case studies are available online.

The NLC initiative

The NLC initiative was first introduced in 2008 "to assist schools in the process of engaging with and implementing The New Zealand Curriculum"¹. Under this initiative, additional funding was provided to the regional School Support Service providers to establish and maintain professional groups, or clusters, of schools. The intention of this funding was to "recognise and grow leadership capability in each of the regions."²

While the composition of the participants and the nature of the support provided has changed since its introduction, the model remains essentially the same. Groups of school leaders are brought together, in professional learning groups, to develop their understanding of The New Zealand Curriculum and to support its implementation in their schools. These groups are led and facilitated by leaders from within the group, known as sector leaders. Advisors from School Support Services provide guidance and resources to the sector leaders. In 2010 there were 195 such groups operating, ranging from small, principal-led groups to larger, secondary school curriculum area groups. In addition there were a number of cross-sector groups.

The evaluation

The evaluation was commissioned by the Ministry to inform future developments of the NLC initiative. The purpose of the evaluation was to determine the extent to which a school leader's participation in an NLC impacted on the curriculum design and review process in his/her school, both during 2009 and into 2010.

There were four key evaluation questions to be considered:

1. To what extent has participation in an NLC increased a school leader's understanding of The New Zealand Curriculum?
2. How, and in what ways, has the NLC supported the school in designing their local curriculum?
3. How has the involvement of the leader in an NLC influenced the way teaching and learning has changed in the school?
4. How, and in what ways, has the NLC professional development model contributed to cluster success, both in terms of the curriculum work and more broadly?

In answering these questions it was necessary to consider the processes through which NLCs have supported the participant schools, the barriers and enablers to their success and how the learnings of the NLC were translated into practice in schools. A range of data collection methods were utilised including a facilitated workshop, interviews, online surveys and document analyses. The workshop involved six advisors from five of the six regions and Ministry personnel. In addition, telephone interviews were undertaken with 11 advisors from across the six regions. All those who attended the workshop were interviewed.

In total, 79 sector leaders and 144 school leaders completed online surveys. These are return rates of approximately 41% and 15% respectively. Ten case studies were also undertaken from around New Zealand, focussed on individual NLCs and the journey they have taken. During these case studies data were gathered from 10 sector leaders and 26 school leaders. Most were interviewed either face-to-face or by telephone, although some completed written questionnaires due to availability difficulties. A companion technical report provides detailed data analyses from the survey and includes the ten case studies in full. Abbreviated case studies are available online.

There are a number of limitations to the findings of this evaluation related to the representativeness of the samples for each of the main data collection methods (surveys and case studies). The very low return rate for the school leader survey is of particular concern. This is likely to be due to difficulties in accessing school based participants due to the autonomous nature of the NLCs and the focus of the advisors on providing support and professional development to the sector leaders, rather than working directly with the NLC as a whole. This sample is comprised largely of principals and representatives from small schools. There is also a large number of respondents from NLCs initiated in 2010. Concerted efforts were made to ensure a broadly representative group of NLCs for the case studies in terms of the composition and location of the NLC. Efforts were also made to ensure a range of foci. However, there is no easy way of determining the extent to which this is the case.

Findings

In this section the key findings related to each of the four evaluation questions above are provided.

School leaders' understanding of The New Zealand Curriculum

Respondents to the school leader survey were asked about their knowledge and/or expertise with regard to The New Zealand Curriculum at two different points in time. Prior to joining the NLC, the reported overall mean level of knowledge and/or expertise across a range of areas related to The New Zealand Curriculum suggests that respondents to the school leader survey were either beginning to work through and think about the implications of The New Zealand Curriculum for their practice including some initial planning, or were including concepts and processes related to The New Zealand Curriculum into their practices. At the time of the survey their level of knowledge and/or expertise had increased across all areas indicated in the survey. Their responses suggest that they were regularly including the concepts or processes into their practices at this time.

On average, the school leader survey respondents attributed this increase in knowledge and/or expertise to their participation in an NLC to between 'some extent' and 'a large extent'. The extent of attribution varied by area of knowledge and by respondent with wide standard deviations for all areas. These findings suggest that the NLC has primarily increased the understanding of respondents in conjunction with other activities, rather than as a stand-alone initiative.

Further, in the case studies, among the commonly identified benefits for individuals were the professional development they received, access to external experts and a wide range of resources, and being able to share ideas and practices. These benefits suggest that the NLCs have provided opportunities for increased understanding of The New Zealand Curriculum, where that is the focus of the NLC.

From these data it would seem that belonging to an NLC has had a positive influence overall on the understanding of aspects of The New Zealand Curriculum of many of the participants in this evaluation. However, the wide standard deviations reported indicate a wide variance in the extent to which this has occurred. Comparative analysis of responses across different sub-groups of school leaders suggest that this is due to the diversity of the NLCs themselves rather than specific demographics. However, those comprised of smaller schools or low decile schools appear to have been more efficacious in this area.

Another key factor is the extent to which understanding The New Zealand Curriculum is the desired goal of the NLCs and what they focus on. In the sector leader survey respondents were asked to identify the goals for their NLCs at the time of the survey. Half the respondents reported that the goal of their NLC was directly related to developing a better understanding of The New Zealand Curriculum and/or implementing it in their schools. This response reflects the change in focus for many of the NLCs by 2010.

Supporting schools in designing their local curriculum

The design of local curricula was a key focus of the 2008, principal-led, professional communities. By 2010 the data from this evaluation suggest that the focus had broadened and it was not clear how many were still focused on this particular goal. The development of a local curriculum was the focus for only three of the ten case studies.

The data collected suggest that where local curriculum was a focus of the NLC there has been benefit for schools in designing their local curricula. Evidence for this can be found in those case studies, Family Ties, Country Calendar and The Sky is the Limi³, where this was an explicit focus. In all three instances participants provided concrete evidence of the extent to which the design of their local curriculum had been influenced and supported by the NLC.

Further, respondents to the school leader survey reported that, overall, they were able to influence the implementation of The New Zealand Curriculum in their schools to a 'large extent', as a result of their involvement in the NLC. This seems to have been primarily due to the large number of respondents who were school principals. Where respondents reported being classroom teachers the mean reported level of influence was only 'very slight'.

Another question in the school leader survey asked respondents to indicate the extent to which they believed the NLC had helped meet their school's needs in implementing The New Zealand Curriculum. The reported overall mean level of help provided was rated as to 'some extent.' Primary school respondents reported more help than secondary, which may be a product of the different foci in the secondary clusters. Many of the primary respondents are likely to have been part of original 2008 clusters.

Influencing the way teaching and learning has changed in the school

The advisors generally felt that there had been insufficient time for the NLC model to influence teaching and learning, believing it requires three to five years for changes to occur. However, the majority of respondents reported that school wide changes had occurred to teaching and learning in their schools with smaller numbers reporting changes at some year levels or within specific subject areas. Nearly half of the respondents reported that these changes could be attributed to the NLC to at least 'some extent', a rating of 3 out of a possible 6.

When asked whether they would choose the NLC model as an effective professional development tool to influence changes to teaching and learning both the sector leaders and school leaders reported, on average, that they would 'probably' do so. Amongst the school leaders approximately half said they would either 'very probably' or 'definitely' do so.

While no one attributed changes to teaching and learning entirely to the NLC, it does appear to have played a part in any reported changes, reflecting the extent to which this initiative is likely to be one part of a wider change process in schools. Again, the extent to which teaching and learning are influenced is likely to be dependent on the efficacy of the individual NLC and the extent to which those participating perceive such influence to be an important outcome.

Contributing to cluster success, both in terms of the curriculum work and more broadly

It is difficult to define one criterion for success across all NLCs. The most common criteria appear to be related to the operation and culture of the NLCs rather than school-based outcomes, although these were also frequently mentioned.

The original policy intent was that the NLCs would support schools in implementing The New Zealand Curriculum. Broadly defined success criteria at a policy and regional level include the desire to raise leadership capacity within the education sector and to develop self-managing and autonomous clusters of schools able to support their own professional learning.

For the advisors a key feature of success was that the NLCs operate effectively as learning communities and that they meet their individual goals. To achieve this they support the development of the sector leaders. For the sector leaders the most commonly reported success criteria were related to the operation of the NLC. The next most commonly reported category included criteria with a focus on teaching and learning, in particular changes to practice. There were also comments related to the provision of professional development.

When asked to rate the success of their NLC, based on their own criteria, the overall mean level reported by the sector leaders was 'very good' (a rating of 4 out of a possible 6) with 10.1% reporting that it was 'excellent'. Throughout the survey results similar means were found for a number of questions suggesting an above average response for the initiative but not an 'excellent' one. Again the large standard deviations for this question, as with others, needs to be noted reflecting the different experiences of the participants.

For the sector leaders the professional development provided and the opportunity to engage with new readings and resources also appears to have been viewed as a success. Most reported they had grown in their own leadership and in their knowledge of teaching and learning. For others the NLC had provided the impetus to achieve things that would otherwise have taken longer. In some instances the pressure of reporting to their colleagues, of having deadlines to meet, seems to have moved curriculum processes along further than if the school leaders involved had been working in isolation.

Arguably networking has been seen as the biggest success of the initiative. Those involved have reportedly enjoyed the opportunity to meet and work with colleagues; sharing practice and supporting each other. In the case studies the most commonly reported benefits for participants related to networking, to sharing ideas and resources in a collegial and safe environment. This is a very broad, largely social definition of networking. Learning what others are doing and measuring one's own achievements against that seem to have been a valuable outcome of being in an NLC for many.

More specifically, the NLCs appear to have been highly successful in providing collegial support for the leaders of smaller schools who feel marginalised in other clusters. The ability to choose who is in their cluster and to engage with similar schools has enabled some NLCs to become strong support networks. Similarly, where secondary subject areas have felt marginalised the opportunity to network with colleagues, to share practice, appears to have been a powerful motivator.

Discussion and implications

When considering the findings from this evaluation, and the implications for the initiative moving forward, it must be remembered that The New Zealand Curriculum is an expansive document; reflecting the multi-faceted and complex nature of teaching and learning. There is any number of potential foci in it for a group of schools to work on. There is also a wide range of professional development initiatives and opportunities available to them to support their professional learning. One of the key issues schools face is determining what to prioritise and how to most effectively meet the needs of their students.

Further, the New Zealand education system is one of the most devolved in the world with schools having a large degree of autonomy with regard to their local curriculum and how it is delivered to their students. This is based on the belief that schools are best suited to determine the needs of the students they teach⁴. However, this has the potential to leave many schools feeling isolated and unsure of what to do next or what to prioritise. There is also the danger that many are simply reinventing the wheel.

The NLC initiative, as it has been implemented, appears to support this view of New Zealand schools as autonomous and largely self-determining. As with much of the policy implemented⁵ in the New Zealand education system there is a heavy reliance on the professional and moral accountability of those within the schools and a leap of faith that the intent of the policy will be implemented . In this initiative the tension appears to be always present between autonomy and accountability; between allowing the NLCs to choose their own path and work independently and expecting them to provide detailed action plans and report against them in a way consistent with a wider policy imperative.

What the data presented in this evaluation show is that the NLC initiative has been successful, if one considers the meeting of individual NLC success criteria as sufficient. The majority of the responses to the questions have been positive. The opportunity to network and to share practices and ideas is welcomed by everyone. There are concerns with the operation of the NLCs and awareness of the challenges they face, yet few would seem to want the NLC initiative to end. In fact, across all the case studies the clear message was that they would continue the NLC in some format regardless of funding⁶.

The key question moving forward is whether the outcomes achieved are sufficient to warrant the initiative continuing in its current format? It was beyond the scope of this evaluation to look at questions of value for money. To answer this question may require more of a focus on what is happening at an individual school level than has been possible for this evaluation.

The idea of Networked Learning Communities, as described in the British literature, is an attractive one with the potential to support sustained growth and learning across and within schools. One consideration for the future is the extent to which the current NLCs should be purposively implemented in a way that is more consistent with Networked Learning Communities⁷. To do so would require greater emphasis on networking between learning communities and on ensuring that the NLCs, and the schools within them, are learning communities; communities that critique and challenge practice, that are firmly focused on improving student outcomes. Discussions with the advisors suggest this would require significantly more external input and resource. Again it is beyond the scope of this evaluation to determine the validity of this perception. It may be that what is required is a shift in the focus and nature of delivery, rather than an increase in resource per se.

Currently, the focus of the NLCs appears to be on sharing practices, ideas and resources. This was consistently reported as a key benefit of belonging to an NLC. The extent to which this sharing influences teaching and learning in schools is less evident. Merely sharing resources does not guarantee change and it is possible that collegial support can affirm the status quo. The culture of the NLCs appears to operate on what is essentially a volunteerism model. That is to say people share what they are willing to share. What and how others choose to use this material is up to them. There appears to be little accountability beyond attending the meetings. There also appears to be little evidence of the critique and challenge, of the professional learning conversations that epitomize learning communities.

It could be argued that the time spent setting up the NLCs and ensuring buy in has been too long and that there is a need for more external input in the initial stages of the NLC along with more clarity of expectation and accountability at a practice level. One of the advisors did suggest that it is possible to force the development of a learning community with a robust programme; while the Dream Sleepers NLC suggests that it can be done. Perhaps, in 2011 it is time to raise the bar in terms of expectations and to implement actively the Networked Learning Community model focused on classroom practice and on enhancing student outcomes.

It is often stated that it takes time to develop sufficient trust for a learning community to develop and that there are no short cuts. Whether this is true is beyond the scope of this evaluation to determine. However, consideration should perhaps be given to how the learning from the NLCs and other cluster work can be utilised to provide sector leaders and the school-based participants with a model for fostering and instilling the necessary professional trust and willingness to critique and challenge practice. In schools children are often told they will be trusted to behave in certain ways; if they break that trust there are consequences. It may be that a similar model with clear expectations and protocols would work for professional groups. A further concern is that an emphasis on collegiality and positive relationships can hinder the development of a professional relationship, where challenge is the norm rather than soft collegiality.

Further, given that clusters have been part of the educational landscape in New Zealand for some time one could question why it still takes so long to develop a culture of open and honest sharing, review and critique of practice. Is it that it has never really developed in any of the cluster work without significant external facilitation, something lacking in the current NLC model, or is it simply that it has not been made clear that this is what was expected.

Another point to consider moving forward is the extent to which the NLCs are viewed as a stand-alone initiative rather than as a learning hub through which information, knowledge and practice are channelled, challenged and critiqued in order to develop new knowledge and new ideas. In this way they would better serve to both broaden and deepen the impact of other initiatives and professional development opportunities. They could provide a forum for the transference of learning between communities and between initiatives much as an integrated curriculum does in schools. There is no doubt that the policy intent of this initiative was to enable the pooling of resources, learning and expertise and on some level this has occurred. Whether the full potential for this to occur has been realised should now be considered.

Beyond all else what it would require to move the NLCs forward into a more robust learning culture is for the NLC participants to be willing to engage at a level beyond collegiality and beyond sharing practice. The data in this evaluation suggest many of those who participated in the NLCs see value in a cluster that enables these activities. They are arguably reaching for 'low hanging fruit', but getting them to reach higher up the tree could be the biggest challenge of all.

Footnotes

1. Ministry of Education. (2010). Request for Proposals with Respect to an Evaluation of the Effectiveness of Network Learning Communities to Schools, p.3
2. Unpublished Ministry of Education working document provided to the evaluation team.
3. Here and elsewhere reference to specific case studies is by an abbreviated version of their titles. The full case study reports can be found in the technical report.
4. The New Zealand governance system features in a large body of literature related to developed systems of governance. For example, see Robinson, V.M.J. & Ward, L. (2005). Lay governance of New Zealand schools: An educational, democratic or managerialist activity? Journal of Educational Administration, 43 (2).
5. For an example of the discussion of the implementation of policy see Ward, L. & Parr, J.M. (Under Review). Digitising our schools: The need for coherence and clarity in policy. Australian Journal of Educational Technology.
6. During the finalisation of this report one of the sector leaders communicated to the evaluation team that their NLC would not be continuing. This was because of a belief that the work of the NLC had been completed.
7. For a discussion of networked learning communities see Katz, Steven & Earl, Lorna. (2010). Learning about networked learning communities. School Effectiveness and School Improvement (21) 1, 27-51