Executive Summary

Introduction

MartinJenkins was contracted in 2006 by the Ministry of Education to undertake the evaluation of the provision of School Support Services (SSS). The SSS contracts are the Ministry's largest investment in centrally-funded professional development. In addition to the SSS agreements, there are a number of other national professional learning and development (PLD) initiatives (a result of Government policy) that are also funded by government. The focus and priorities of the SSS contracts are revised or reshaped annually in response to Government policy and emerging issues and challenges facing the schooling sector.

Research evidence shows that what school leaders and teachers know and do is one of the most important influences on what students learn. In this context, professional learning and development is critical as it enables the Ministry to strengthen the understanding and skills of school leaders and teachers so that they can best meet the learning needs of all students. A multi-year evaluation of the provision of SSS was initiated to better understand aspects of the current system that work or do not work with a view to informing Ministry decisions and considerations about future  professional development provision.

Evaluation approach

Discussions with internal and external stakeholders in early 2007 established that the evaluation of the provision of SSS was expected to serve an improvement and learning purpose in keeping with the long-term, established and non-contestable nature of the school support service agreements. Consequently, the evaluation was designed to provide ongoing feedback to the Ministry so as to strengthen the contracting environment and enable providers to deliver more effectively. An improvement focus would also yield rich insights about the way in which providers and the Ministry operated which in turn would contribute to lifting the outcomes of in-service professional learning and development.

To respond to these requirements and needs from the evaluation, evaluators began by developing a conceptual framework to anchor and guide the data-collection activities over the three-year period. The framework and the rationale underpinning the framework is outlined in detail in Chapter 3.

The following diagram illustrates our approach to the evaluation. On completion of each evaluation activity within each phase, an analytical workshop was held with SSS Directors (to test and validate findings from each activity to ensure robustness of the analysis and interpretation) and with a Ministry Reference Group to test and disseminate findings for wider use within the Ministry. A final internal workshop was also held to explore broad conclusions and to confirm the direction of this final report.

There were three broad phases and key activities in each phase were as outlined below.

Figure 1: Description of the phased approach to the evaluation

Main findings

The Ministry is charged with meeting Government's goals and aspirations for education. Achieving these goals requires the Ministry to equip schools and teachers with what they need to support all students to learn. The Ministry invests in a range of professional learning and development initiatives to support teachers in New Zealand and the SSS represent a significant proportion of this investment. Key messages that emerged from this evaluation can be summarised as follows:

• Data on access, to determine which schools were accessing or not accessing professional development services offered by SSS, was patchy and inadequate to provide an accurate and nationwide picture of schools that participate in professional development. As a result, it was not possible to undertake any trends analysis or identify patterns in the profile and mix of schools accessing professional development from SSS. The evaluation highlighted the importance of this data to build a comprehensive picture of the use of professional development services offered by SSS. In response to this, the Ministry revised the data-collection templates from 2008 onwards, requiring providers to provide more detailed information about delivery of PLD to schools in their region. This allowed the Ministry to build a nationwide picture of access more easily.
• There was variability in the processes and criteria used by providers with respect to school selection. The evaluation found, for instance, there was no systematic or explicit approach to school selection, and schools felt processes around selection lacked transparency. The selection of school tended to be demand-driven (schools that asked for PD received it) and/or network-driven (schools that had a prior relationship with advisors tended to have easier access to the service) and/or through referrals (where schools were referred to the service by the regional Ministry office). More importantly, the findings indicated that there were no explicit criteria for school selection, creating a perception among some school leaders that the resources were not being used efficiently and effectively.
• The delivery of PLD is strongly driven by expectations and aspirations set out in the SSS contracts. The school support services are a collection of professional development services that the Ministry purchases for schools that allows the Ministry to meet government's commitment to education. The services purchased are varied annually to respond to government priorities. Consequently, the contracts offer the Ministry a tremendous opportunity and leverage to shape and influence PLD delivery by ensuring that the contracts are well written and provide a clear direction about Ministry expectations for a given year. Changes introduced to the 09/10 contracts requiring providers to focus on a smaller number of high-priority areas and prioritising support for schools and students that need it most sent a clear signal about current government's focus on lifting student achievement.
• There are a number of Ministry–funded professional development providers and there were significant differences in how these PD contracts were designed, managed and implemented by the Ministry. This created confusion as schools received support from more than one Ministry-funded professional development provider at any given point in time. These variations meant that schools experienced these contracts differently even though they were run by the Ministry, and led to mismatch in expectations as the resources across these contracts were not spread evenly.
• There was an expectation from the National office (responsible for overseeing the contract) that SSS providers would consult with regional Ministry office to identify regional needs and priorities. The evaluation found that these links and relationships were not being exploited sufficiently and regional Ministry office felt excluded from these contracts, resulting in missed opportunities for using regional intelligence when prioritising or selecting schools. These issues were addressed in the 09/10 contracts and there is a now a clear expectation that regional Ministry office would have an active role in the prioritisation and planning processes. The formative evaluation to examine how these plans are playing out revealed that there are significant issues that could impact on the success of this approach/model. These relate to perceived capacity and capability constraints within the regional Ministry offices; lack of clarity around roles and responsibilities; and lack of involvement in the setting of the SSS contract outputs and focus areas.
• The provision of leadership and management support through SSS was part of a wider Leadership Strategy aimed at providing professional learning and development support to principals. In addition to the SSS contracts (focused on first-time principals, aspiring principals, principals in at-risk schools and middle leaders) there were other initiatives such as the Principals Development Planning Centre, focused on providing professional development to experienced principals. The evaluation found that provision of services with respect to leadership and management across the four priority groups identified in the SSS contracts was uneven. The commonly-held view was that first-time principals and principals in schools at-risk absorbed the bulk of the resources. This limited access to the service by school leaders in general.
• The way in which in-depth PD work was delivered to schools was viewed positively by participating schools. As noted earlier, the selection of schools tended to be demand-driven rather than driven by need, which meant that schools that were most in need did not always receive or have access to the support. The evaluation also showed that monitoring success or impact of the in-depth PD tended to be limited to content and pedagogical shifts while other areas of school development, critical to sustainability, were often ignored. Maximising value of the in-depth PD offered to schools requires consideration of impact at all levels: at an individual teacher level; at a department or syndicate level; across all teachers in the school; and at a school/institutional level. The evaluation findings highlight an opportunity for the Ministry to redirect providers' focus and efforts to ensure the support provided is cognizant of the wider system in which the teacher operates. That is, as well as effecting change at an individual teacher and/or syndicate and/or department level, support provided through SSS also responds to issues or constraints in the wider school environment. This can be achieved through specific support provided to a range of groups including school leaders, middle leaders, teachers, provisionally-registered teachers and their mentors, and overseas-trained teachers. Regardless of the group being worked with, advisors can and must be encouraged to work towards leading change within the school.
• Notions of sustainability are central to SSS and the Ministry had an expectation that sustainability goals and principles were built into the design and delivery of professional learning and development support. The evaluation highlighted a high level of variability across providers in this regard and, while in some instances there was intentionality around sustainability, there were other instances when sustainability conversations occurred when the PLD came to an end. This left schools with an acute sense of loss and a feeling of being 'left in the lurch'. To ensure goals of sustainability are achieved, there is a need for a deliberate and planned approach to sustainability.
• Monitor progress in ways similar to how effectiveness of PLD is tracked. This needs to begin with a shared definition of what exit means and a common understanding across providers as to why exit matters.

Lessons learnt

The evaluation findings emphasised the continued importance of PLD because of the role it plays in strengthening/‌enhancing the understanding and skills of school leaders and teachers to best meet the learning needs of all students. The Ministry facilitated access to quality PLD by investing in a range of professional development in the school sector, including SSS. Historically the design and implementation of the SSS contracts were underpinned by principles of equity (ensuring access to quality PLD for all teachers), a focus on strengthening teaching practice and promoting synergy between pre-service and in-service to ensure consistency in curriculum implementation. More recently there has been a shift in policy direction towards a more targeted approach to delivery (focusing on schools with need) and a focus on school development (due to the growing notion of the need to contribute to continuous improvement of schools). In light of these changes, it may be necessary to reflect on the extent to which current design of the SSS meets emerging needs and shifts in policy and how the Ministry can support and contract PLD services from the sector in general.

The findings from the evaluation show that there is value in maintaining a centrally-funded PLD infrastructure as it ensures that the design and delivery of all support are underpinned by cultural responsiveness, teaching as inquiry and pedagogical content knowledge. The challenge the Ministry faces is to better understand which type of PLD investment is most appropriate given current government's priorities, how best to deploy it and the conditions that achieve success. Further the current government has fewer, clearer priorities for education and it may be appropriate to reflect on whether the current approach to PD provision and PD contracts is still relevant.

A way forward

The evaluation activities undertaken have collectively identified the growing importance of good planning, targeting, effective service delivery and ongoing monitoring and evaluations to inform future decisions. Applying these findings suggests the need to structure the design of any future PLD infrastructure around effective processes for policy development, beginning with an understanding of the opportunities including clarifying the policy intent; developing and designing the response; implementing the response and reflecting on and learning from the intervention (monitoring and evaluation). In light of this, it is useful to reflect on the evaluation findings presented in this report with a view to raising questions for consideration by policymakers in the future. These relate to four broad areas:

• Understanding the opportunity
  
  Underpinning the Ministry's strategies and interventions in PLD, there needs to be a clear understanding of the opportunity and how this can be leveraged by the Ministry to better achieve government's goals and aspirations for education. This requires the Ministry to describe a relationship between the theory and the evidence to support effective interventions as this will enable resources to be used more effectively and establish stronger alignment between the different initiatives that have similar objectives. There is also need for promoting consistent messages about expectations, and desired behaviours so as to build a shared understanding of the problem across the different stakeholders.
• Developing a strategic response
  
  Central to any strategy or response is having a clear sense and articulation of the goal or outcome that the organisation and/or the contracts for services are working towards. In the context of SSS contracts, the evaluation indicates that providers have focused on outputs (namely, outputs/schedules in the contract e.g. numeracy, literacy etc) and plan their resourcing and delivery of PLD around these outputs resulting in a siloed and fragmented approach. Developing a strategic response requires a shift towards an outcomes focus as it will enable providers to take all aspects of the wider system into consideration, in order to support cultural change at the school. Adopting this approach requires the Ministry to manage for outcomes and promote a stronger connection between outputs and outcomes across all PLD providers. This shift will need to be supported by a clearer mandate and accountabilities.
• Implementing the response
  
  Historically, the SSS agreements were negotiated and funded centrally by the Ministry of Education and delivered regionally by six universities located across New  Zealand. The Ministry managed and structured the contract and how providers delivered PLD, with the expectation that schools received PLD in a flexible, coordinated way. It was important to allow regions to innovate and not try to force them into centrally-determined frameworks and processes. The trade-off for giving more degrees of freedom to providers is the need to strengthen governance and management arrangements at the centre to ensure that effective accountability is maintained, particularly relating to identifying priority groups; establishing clear criteria and focus for the PLD and providing training, guidelines and templates, and other forms of administrative support to ensure consistency in delivery.
• Monitoring and evaluation
  
  Tracking progress towards outcomes is essential to inform the ongoing development, refinement and improvement of any intervention. Given the complex nature of the PLD and school environment, the ability to respond to issues and opportunities as they arose was essential to ensure that risks were minimised and opportunities leveraged. Given the complex nature of PLD and the myriad of factors that contribute to achieving desired change, it is by implication often difficult to 'unpack' the impact of particular PLD strategies and interventions. However, evaluation of a small number of initiatives and ongoing monitoring can contribute to the growing knowledge base about what works to improve quality of teaching; the process also lifts student engagement and achievement.