Executive Summary

Introduction

MartinJenkins was contracted by the Ministry of Education to undertake the evaluation of Inservice Teacher Education Practice (INSTEP) project. INSTEP was designed to promote a strategic and coherent focus across the inservice teacher education system and build the capability of inservice teacher educators. Essentially the project was expected to develop and establish effective evidence-based approaches to the learning and practice of inservice teacher educators.

Recent publications such as the Teacher Professional learning and Development: Best Evidence Synthesis Iteration (2007) highlighted the critical role played by inservice teacher educators in promoting effective teacher professional development and promoting teacher learning. For a while now there has been some concern within the Ministry that the delivery of inservice teacher education services is variable and anecdotal evidence suggests that the delivery of professional development does not often meet the needs of teachers. Through this project the Ministry aimed to fill gaps in current knowledge and understandings about inservice teacher education practice and trial approaches that would inform future practice. This would contribute to the ongoing development of professional practice of inservice and learning of teacher educators. Meeting the project objectives involved:

• establishing a national group of practising inservice teacher educators to coordinate a coherent approach to developing inservice teacher education knowledge and expertise
• developing and publishing evidence-based professional learning materials for inservice teacher educators
• drawing on the expertise and experience of representatives from the inservice teacher education sector
• piloting and refining approaches to professional learning through inquiry-based research projects
• evaluating and refining the project on an ongoing basis applying the research and development principles underpinning INSTEP.

Evaluation approach

The evaluation (refer evaluation framework presented on pages 15 and 16 of this report) was carried out in two phases:

• Phase one was undertaken between April 2007 and July 2007. This phase involved an examination of the role and contribution of project structures and processes in helping and/or hindering INSTEP from achieving the project goals. Data was gathered from the 12 National Facilitators (NFs) involved nationally in INSTEP as well the INSTEP project team located within the Ministry. The analysis was tested and sharpened in discussions with the wider group of national facilitators and the Evaluation Advisory Group set up within the Ministry in June 2007.
• Phase two was initiated in July 2007 and concluded in December 2008. This phase involved two sets of activities: a longitudinal case study research (with two data collection points) and a sector survey. Four case study NF pods were selected in consultation with the INSTEP project team and each case study included in-depth interviews with the National Facilitator, two Regional Facilitators, 2-4 inservice teacher educators within the provider organisation involved in INSTEP, some teachers and school leaders. The sector survey was undertaken with all participants in INSTEP over the three-year period including Sector Reference Group (SRG) members. In total 361 participants were sent the questionnaire and we achieved a response rate of 55%. The aim of phase two was to identify early indicators of change and assess the extent to which there was evidence to suggest that shifts in inservice teacher education practice had occurred and the extent to which these were sustained after INSTEP.

This report is a synthesis of data gathered from evaluative activities carried out in phase two and the main findings have been presented in three parts:

• Understanding pre INSTEP context for ISTE practice
• Demonstrating the value of INSTEP
• Implications for the future.

Part 1: Understanding pre-INSTE context for ISTE practice

INSTEP was designed promote a strategic and coherent focus across the system in response to the variability in the quality and consistency in inservice teacher education practice in New Zealand. Therefore building an understanding of the pre-INSTEP context was a critical step as it set the backdrop against which the value and merit of INSTEP could be meaningfully understood. The evaluation findings highlighted a number of challenges for ISTEs due to the fact that the teachers or school principals who took on the role of an inservice teacher educator had not been adequately supported to make this transition by their employers. Consequently ISTEs found certain aspects of their role particularly challenging in the initial stages. Notably, their lack of experience in working with adults, output driven nature of advisory work within School Support Services, perceived pressure to act as the 'expert', and their lack of facilitation experience were identified as factors that impacted on their ability to be effective in delivering professional development to teachers.

These challenges combined with the mixed skill sets of the current pool of inservice teacher educators had contributed to the variability in practice observed by the Ministry. ISTEs said that the following areas of practice were most significantly impacted by these challenges:

• Quality of data analysis as ISTEs own professional development and learning had not extended their skills in this area resulting in pockets of good practice in relation to data analysis
• Insufficient time spent on planning and diagnostic processes when designing professional development and learning interventions with and for teachers
• Coaching and mentoring newer staff within the organisations as there were no institutional systems and processes for induction and ongoing training and support for new staff
• Developing and growing as reflective practitioners, as pressure to deliver on outputs meant that there was insufficient time spent critically reflecting on practice.

The issues and challenges faced by inservice teacher educators pre-INSTEP clearly suggest that INSTEP was a timely and an important intervention in developing a national approach to improving the quality of inservice teacher education practice.

Part 2: Demonstrating the value of INSTEP

A key project outcome for INSTEP was the development of a model for the ongoing professional development of professional development facilitators owned and operated in a self-sustaining way of teacher education professionals across the country. Therefore determining the extent to which this outcome was achieved was an important step for this evaluation. In this section we discuss the value and benefits of INSTEP for participants in bringing about the shifts desired in knowledge, skills and attributes regarding ISTE practice. The discussion is presented around the theory of change diagram developed to represent and convey the essence of INSTEP and its contribution to participants. The diagram illustrates how INSTEP has contributed to bringing about the desired shifts in ISTE knowledge and practice and the nature of this contribution to improved teaching quality which in turn contributes to improved student outcomes.

This diagram has been generated from the evidence presented by participants and illustrates how INSTEP has contributed to the outcomes in the real world. The analysis of the findings from INSTEP presented in this section is wrapped around this analytical model.

Figure 1: How INSTEP works to bring about desired shifts - an analytical model

Implementation of INSTEP

Participants described the INSTEP project as an "invaluable investment in building capability of ISTEs". Despite the challenges posed by the research and development approach, the project's goals of bringing together practitioners from across the sector to work collaboratively to examine, inquire and build knowledge about the practice and learning of ISTEs was highly commended by all participants. The investment in INSTEP was seen as an acknowledgement by the Ministry of the importance and value of ISTE as a lever for affecting change in the teaching and learning area and served to bring about greater consistency and coherence in practice. While participants recognised that there were inbuilt tensions across providers given the current contestable environment (which runs counter to the collaborative inquiry promoted within INSTEP), there was some degree of openness to engage and share different interpretations and approaches to inservice teacher education.

The key elements in the design of the INSTEP 'intervention' that enabled the project to achieve positive outcomes were:

• The research and development approach
• Leadership by a core group of national facilitators
• Inquiry/action research within communities of practice
• Management by a project team located within the Ministry
• Concurrent development of learning materials
• Additional support through research mentors, online communities, international speakers etc.

Participants undertake inquiry into their practice and trial new approaches

In keeping with the R&D nature of the project, participants were given fair amount of flexibility to frame their inquiry into their own practice around what mattered most to them as ISTEs. The INSTEP project structures provided a broad framework/ infrastructure within which this inquiry into practice took place (eg national facilitators supporting regional facilitators who in turn supported inquiry into practice by ISTEs; national facilitators inquiring into ISTE practice and identifying principles underpinning their work) and offered ISTEs the opportunity to identify problems of practice or practice puzzles and pursue this inquiry to achieve better outcomes for teachers and students.

Over the course of the INSTEP project ISTEs developed a range of different approaches to the inquiry, each of which was designed to facilitate critical reflection on a problem of practice identified by them. The research reports provided to the Ministry by the NFs at the conclusion of INSTEP document the nature of inquiry undertaken by each of the 12 pods within the project. Our analysis of these reports showed that there was significant variability across the pods and that across the 12 pods, changes were occurring at two broad levels: at an individual ISTE level and at the provider organisational/institutional level. The longitudinal case studies were selected to offer insights about how these changes played out over time and the factors that helped or hindered the sustainability of these shifts. In this section we discuss examples of the broad approaches used within INSTEP to inquire into different dimensions of ISTE practice. This is not an exhaustive list of the inquiry undertaken by all participants and is intended to only illustrate the scope and breadth of the inquiry.

• Professional learning progressions. The progressions were developed around the dimensions of practice identified in the early stages of INSTEP. Within each of these dimensions a series of steps were described which then formed a learning progression for use by ISTEs involved in INSTEP in this case. Each ISTE situated them on a learning continuum and steps on the progressions provided a direction for the professional learning of the ISTE.
• Professional learning groups. Participants trialled various pod structures for supporting and fostering the professional learning experiences of ISTEs within their contexts. Regardless of the approach, ISTEs found the opportunity to engage in conversations about practice very useful and most institutions were looking to ways in which they could sustain the gains made through INSTEP in this regard.
• Applying theories of Model II learning in action. Getting buy-in from teachers and school leaders is a critical step to achieving positive outcomes from a professional development intervention. While most ISTEs understood the value of getting buy-in, they often found it difficult to engage with teachers who were rigid or put up barriers to learning. By examining their own practice vis-à-vis theories of learning, participants acquired new skills and tools for engaging teachers and school leaders.
• De-privatising practice. Many ISTEs pointed out that their practice had evolved over the years on the basis of their experience or additional study that they may have undertaken. Within INSTEP, they trialled alternative approaches to examining their practice more explicitly within professional learning communities and developed a deeper understanding of what works or doesn't work in relation to how they work with teachers.
• Taking a school-based approach to inservice. In some cases, lead teachers in schools took on the role of inservice teacher educators and worked with other teachers to examine and inquire into teaching practices in relation to literacy.

Impact on Inservice Teacher Educators

Through the trialling of the various approaches to inquire into ISTE practice facilitated through INSTEP, ISTEs gained significant insights about what constituted effective ISTE practice and the knowledge and theoretical base that influence, support and shape ISTE practice. Through the longitudinal case study research the evaluation explored the extent to which the early effects and perceived value of INSTEP had been sustained and embedded into the everyday practice of ISTEs.

Participants unanimously stated that they found the INSTEP project to be invaluable. Adopting a R&D approach over three years and investing in understanding ISTE practice in great depth had contributed significantly to the knowledge base around this area. This investment was seen as an acknowledgement by the Ministry of the importance of inservice teacher education as a lever for change and enhanced connections and collaboration between ISTE organisations within New Zealand. Prior to INSTEP, the sector was seen as a collection of individuals and providers with different views and concepts about what works in inservice teacher education resulting in variable and inconsistent practice. The investment in developing the Teacher Professional Learning and Development: Best Evidence Synthesis Iteration (BES) was seen as the first step towards building a knowledge-base around what works in promoting learning for teachers. The INSTEP project was felt to have extended these understandings and heightened awareness and understanding of what constituted effective practice and also extended the knowledge and skills of ISTEs more generally. The benefits and value of INSTEP on ISTEs and their organisations can be summarised as:

Shared understanding of role and purpose of ISTEs across the sector

Quality teaching and learning in any community requires a shared vision and understanding of what is to be achieved in practice. An underpinning rationale for INSTEP was to address a gap in educational research and literature about the practice and learning of inservice teacher educators. By adopting an inquiry approach for the development of practice, it was felt that ISTEs across the sector would build a shared understanding of what constituted effective ISTE practice. Therefore ascertaining the extent to which INSTEP had been successful in building this shared understanding was critical to the evaluation. The evaluation findings suggest that there is a more sophisticated understanding of the role of ISTEs. Where as in the past ISTEs were seen as purely facilitators of professional development, now they were increasingly seeing themselves as agents of change and as facilitators of learning. Through the project ISTEs have developed a shared understanding of three key aspects to their role as teacher educators: ISTEs as pedagogical leaders; ISTEs as change agents; and ISTEs as inquirers. These roles combined with their pedagogical content knowledge were felt to constitute effective ISTE practice and critical to affecting change in teaching practices leading to improved student outcomes. ISTEs also believed that these roles worked collectively to enhance their impact on teachers.

Impact on teachers

ISTEs commented that engagement from teachers was a vital clue that their efforts and focus on critically examining their practice was working. Through videoing their practice, peer observation to get an external perspective on their professional development approaches and strategies, and modelling appropriate behaviours, ISTEs ensured that their own practice was based on sound theories of teaching and learning. Teachers involved in INSTEP noticed this and aimed to apply these practices in their own work with other teachers and students. Our conversations with teachers supported these ideas as teachers involved in INSTEP commented on the open and engaging approach taken by the ISTE and their desire to listen to the teacher and frame the conversation appropriately.

While teachers acknowledged that they acquired new knowledge and skills through their involvement in INSTEP, the longitudinal case study research findings indicated that sustaining these shifts posed some challenges to teachers. Lack of support from school leadership and management and a culture of resistance in the school were identified as two critical impediments to sustainability.

Impact on students

Unlike other PD interventions where impact on students is paramount, in the case of INSTEP it was apparent early on that impact on students was likely to be achieved indirectly through impact on teachers. Essentially the project was aimed at developing and establishing effective evidence-base focused on the learning and practice of inservice teacher educators and was seen as a capability building project. Therefore, ISTEs viewed and tracked impact on students through their ability to ensure that teaching practice was informed by student achievement and outcomes. The inquiry cycle in the INSTEP materials clearly illustrates the linkages between ISTE inquiry and its impact on student outcomes and the evaluation suggests that ISTEs are consciously anchoring their inquiry in teacher needs which in turn is anchored in student needs.

The Controller and Auditor -General's report on Ministry of Education's suite of professional development support for teachers notes that 'although an analysis of student achievement information can identify areas in need of improvement, relationships between the professional development received by teachers and student achievement are complex. The performance of students can be influenced by a range of factors and circumstances". This was even more challenging in INSTEP which was essentially a capability building project and aimed at strengthening inservice teacher education practice through inquiry approaches. By building the capability of ISTEs INSTEP enhanced the relevance and appropriateness of the professional development and leaning opportunities for teachers which in turn created a positive learning environment for students as reported by teachers in the cases selected for this evaluation.

Impact on sector

A key objective for INSTEP was to support professional leadership and ongoing improvement within the inservice teacher education sector. Consequently evidence of sector taking ownership and leading the discourse on quality of inservice teacher education can be seen as an important indicator of success in achieving this objective. The survey data gathered from the Sector Reference Group members indicated that INSTEP has had a reasonable impact on building a sense of ownership or community across the ISTE sector given the timeframe for the project. For instance when asked about the extent to which INSTEP had built a sense of community across the ISTE sector, 61% felt that it was 'significant' or 'growing while another 39% felt that it was 'minimal' or had 'no impact'. However, Ministry expectations of getting the sector to take the leadership in this area has not been met as the sector still sees a strong role for the Ministry in continuing to lead this discourse.

Part 3: Implications for the future

The findings from the evaluation indicates that INSTEP has been an invaluable and timely intervention in bringing about an awareness and understanding about what constituted effective ISTE practice across the sector. The project reinforced the basic principles articulated in the Best Evidence Synthesis on Teacher professional development and learning and demonstrated to ISTEs that when they examined their practice collaboratively, challenged each other's ways of working and shared and discussed ways in which they determined effectiveness of their work, they were able to achieve far greater engagement from teachers in the professional development and learning. The value of adopting an inquiry-approach in developing practice is well documented in Ki te Aotūroa – Improving Inservice Teacher Educator Learning and Practice. This evaluation report offers additional insights about the ways in which participating in INSTEP has contributed to bringing about shifts in knowledge, skills and expertise of ISTEs and identifies early indicators of change for the project.

Our analysis indicates that INSTEP has had an impact at a number of levels:

• at an individual ISTE level
• at a group level
• at an organisational level
• at the sector level.

We see these levels as embedded within each other suggesting that there may be a time dimension to these impacts. For example, for changes in individual ISTE level to generate impact at the wider sector level requires time as it involves bringing about shifts in the world view of different sector groups. Further there are a number of other contextual factors that can impede these shifts from occurring easily such as the contestable nature of the environment and this need to be recognised. The following diagram illustrates the particular focus at each level:

Figure 2: INSTEP levels of impact

At an individual ISTE level, INSTEP can be deemed to be a success and all ISTEs involved in INSTEP were unanimous in their view that their views about their practice had been transformed. Focusing on the "I" and engaging in problems of personal professional practice, gathering and examining evidence of this practice and trialling approaches that challenge or push this practice, ISTEs involved in INSTEP had developed deeper understandings of his/her role as pedagogical leaders. Further tools like the video, audio transcripts or peer observations have led to de-privatising practice which in turn has created significant learning opportunities for ISTEs. As one ISTE put it, "there is no going back" and this illustrates the significance of the shifts that have been made and the value of applying these new understandings in their work.

At a group level, there is evidence to show that ISTEs are engaging in collaborative inquiry into their practice within their professional learning groups established during INSTEP. In some instances, these groups are formed around output groups within the School Support services contract such as literacy and numeracy, to discuss and debate issues relating to their practice. Through the consolidation initiatives, other groups have formed such as the Assess to Learn (ATOL) or Literacy Professional Development Project (LPDP) which aim to work across geographical boundaries or institutional boundaries.

The focus on the 'WE', as a community of inservice teacher educators, is clearly growing and taking shape and needs to be supported to investigate cross-cutting issues for the wider community. A key success factor in achieving change at a group level appears to be commonality of interest and purpose.

At an organisational level, the focus has been on "OUR" institution and INSTEP has made significant strides in getting provider organisations involved in the case study research to think differently about how they structure, support and monitor effectiveness of their advisory work. As a result, case study organisations have significantly reshaped their structures and systems particularly in relation to their induction programmes, professional development days, how they support ongoing professional development of their staff, performance appraisal systems and creating professional learning groups to facilitate ongoing inquiry into practice. This is a critical first step towards sustaining the benefits and lessons from INSTEP.

A closer examination of these structures and systems reveals that within these broader institution-wide changes, in most instances the inquiry on practice related issues still tend to be individually, "I" focussed. This is an emerging issue that needs to be addressed by the management teams, particularly in the larger provider organisations. Just as responsibility for improving student outcomes is a collective responsibility, so too is the responsibility of improving quality and effectiveness of ISTE practice. This means that over time the focus needs to extend beyond individual improvement to explore how ISTEs can contribute to lifting the quality of the services provided by their institution as a whole. This requires ISTEs to escalate the inquiry to include practice issues that face the entire organisation. It also allows the organisation to tap into the tacit knowledge of advisors to collectively reflect on aspects of their service including issues such as prioritisation and decision- making processes regarding selection of schools; aligning professional delivery to regional needs; gathering evidence of success. Focusing on these issues will help transition INSTEP from an individually focused intervention to bringing about shifts in the professional development provision system.

At the sector level, the focus is on "US" which assumes a level of ownership across the sector for the quality and coherence in approaches to inservice teacher education. However, as noted earlier, impact of INSTEP on the sector appears to be the weakest suggesting that more needs to be done in this regard. While sector reference group members acknowledged that there was need for a coherent sector leadership in the future, the contestable environment that ISTE providers operate in does not create the incentives to bring about this level of cohesion. This requires the Ministry to re-consider and review the structures, systems and processes that help/hinder the development of sector leadership for inservice teacher education and how it can overcome these.