Evaluation methodology

Evaluation of INSTEP

Sankar and Bennie (2006) in a paper on the Evaluability of INSTEP proposed a framework for evaluation of INSTEP which was subsequently agreed to by the Ministry. The framework was structured around a theory-driven approach to evaluation that placed current understanding about ISTE learning and practice at the centre of an evaluative inquiry. The theory of change model developed as an anchor for the evaluation reflected the unique design of the INSTEP project and was flexible enough to adapt to the evolving nature of the design. After its implementation there was a shift in emphasis towards establishing an evidence base about the processes used to achieve these outcomes and objectives. The following diagram reflects the understandings of how INSTEP was expected to contribute to bringing about desired shifts in ISTE practice leading to teaching practice. This diagram (illustrating the linkages between the different components of the project) was used to frame the project and to surface critical questions and relationships to be explored in the evaluation.

Figure 3: Understanding INSTEP

Applying this framework highlighted the different levels of change in ISTE practice that INSTEP aimed to bring about changes to INSTEP participants’ own level of understanding and knowledge about their practice; changes to ISTE practice more generally; increased consistency and professionalism within and across the wider ISTE sector; and contributing to a body of knowledge about ISTE pedagogy. All these changes were in order to achieve the goal of effective, quality teaching which in turn will achieve improved student outcomes. The framework identified seven lines of inquiry that would collectively determine whether INSTEP had been successful in achieving its intended objectives. The lines of inquiry as identified in the INSTEP evaluation framework were as follows:

Figure 4: An evaluation framework for INSTEP

Of these seven areas of inquiry, the Ministry identified two lines of inquiry as being the most critical for the Ministry, for the future, and for contributing to the knowledge base around impact of INSTEP. These were: understanding the role and contribution of project structures and processes in facilitating implementation of INSTEP (for the Ministry) and ascertaining the knowledge, skills and attitudinal shifts made by INSTEP participants and the impacts of these shifts in their everyday context (impact of INSTEP). The time and budget constraints surrounding INSTEP meant that only a limited number of evaluative activities could be undertaken within this evaluation. Consequently, MartinJenkins was contracted to undertake two phases of work:

• Phase one – Early exploratory phase focusing on the role of project structures and processes in helping and/or hindering INSTEP project from achieving its goals (report submitted in 2007)
• Phase two – Identifying early indicators of change in ISTE practice as evidenced in the practice of the national facilitators in their real world context (the current report).
  

Phase one – Early exploratory phase (April 2007)

Phase one examined the role and contribution of the project structures and processes in helping/hindering the INSTEP project from achieving its goals. As indicated in the above framework, INSTEP aimed to lift the capability of the sector and key actors in the sector including the wider community of inservice teacher educators, teachers and school leaders. Plus, it would contribute to developing a knowledge base around what constitutes effective inservice teacher education practice in New Zealand. By bringing together a group of national facilitators working as a community of learners and supporting the learning of ISTEs more generally through inquiry-based approaches, the project aimed to bring about shifts in current knowledge and understandings about ISTE practice. Therefore, the extent to which the project structures supported the NFs to achieve the broader goals for the INSTEP project was a critical first step. Phase one drew on data gathered through in-depth conversations with the NFs and INSTEP project team members as well as through an in-depth review of relevant documents and reports. In examining the influence and impact of the project design, the evaluation in this phase focused on three key questions:

• To what extent have project structures and processes helped or hindered the INSTEP project from achieving its goals?
• Which features of the structures and processes have worked and why?
• What features did not work and why?
  

Some of the key findings were:

• Contribution of the R&D approach to the project: the relative fluidity and emergent nature of R&D projects vis-à-vis conventional research projects posed some challenges for participants.
• Leadership by a group of national facilitators: the national team was the main conduit through which the Ministry and the INSTEP project team aimed to effect change in the ISTE sector and to build a sense of coherence around ISTE practice. The findings indicated that while a national team of facilitators led by a Ministry project team has been an efficient way of implementing the R&D model at a national level, it raised some challenges for the Ministry. The evaluation highlighted that one of the primary purposes of INSTEP – to build a sense of coherence and consistency in ISTE provision through a shared understanding of what constitutes effective ISTE practice – may not have been fully achieved. While attempts by the project team to synthesise current understanding and learnings from the research projects have gone some way towards meeting this core purpose, the lack of sharing across the individual projects has meant that collective understanding amongst participants as to what constitutes effective ISTE practice has not been built.
• Inquiry/action research within communities of practice: inquiry approaches provided the opportunity to critically reflect on what was happening in a given situation, with the ultimate goal of achieving improved learning outcomes for teachers and students. In the context of INSTEP, the inquiry approach was seen as a learning process that helped practitioners to create new knowledge which they then used to inform their planning and actions. In INSTEP this was facilitated in a number of ways, including monthly meetings of national facilitators. In these, participants’ collectively inquired into their own beliefs and assumptions, and asked hard questions about the lens through which they selected and interpreted evidence or observations. They also considered the impact that their decisions and actions had on the people they worked with – teachers and school leaders.
• Management by a project team located within the Ministry: in INSTEP the project structures operated at a number of levels – at a sector level through the Sector Reference Group; at a Ministry level through the Project Advisory Group; at a regional level through the twelve research clusters; and at a national level through the national team which includes the national facilitators and the Ministry project team. This made the project management aspect of INSTEP complex and onerous.
• Concurrent development of materials: a key objective of INSTEP was to “develop and publish evidence-based professional learning materials for inservice teacher educators”. In order to meet this objective the project team consciously included a member with strong communication and publishing experience. While inclusion of this objective within INSTEP was contentious in the early stages of this project, on reflection, this objective has been hugely beneficial as it helped anchor the project and to keep it focused on outcomes; that is, capture learnings and insights about what constitutes effective ISTE practice.
  

Phase two – Early indicators of change (July 2007 – December 2008)

This report responds to questions posed in Phase two of the evaluation and draws together the analysis from two sets of activities undertaken in this phase:

• Longitudinal case study-based research;
• Sector survey.
  

Longitudinal case study research

The longitudinal case study research was undertaken to map the practice changes evident in inservice teacher education among INSTEP participants and to provide a description of how these changes were contributing to strengthening or building effective teaching practice more generally in schools. Using the national facilitator as the lens into the project, the ‘case’, this phase of the evaluation aimed to generate insights about the question ‘How and in what ways and under what circumstances has participating in INSTEP transformed NFs and RFs beliefs, attitudes and knowledge about ISTE practice and theory?’

The cases of interest were the NFs, as they were the brokers who helped create connections between different players in the ISTE system and introduced new knowledge and ways of working into their practice. Therefore understanding them, hearing their stories and learning about how they applied the new insights and knowledge in every aspect of their work was an important part of demonstrating impact of INSTEP.

In each case, we interviewed individuals who fall within the NF’s sphere of influence and gathered data from all the players in this sphere to triangulate and validate the emerging picture. This included interviews with the NF and representatives from the RF network, management team in the NF’s own organisation, other ISTEs, teachers and school leaders. The following diagram shows the different voices that were captured within each case:

Figure 5: Defining the case

In total, four case studies were undertaken. There were two data collection points over a 12-month cycle. The cases were selected in discussions with the Evaluation Advisory Group and the INSTEP project team. At the time of selection, there was evidence to show that these cases were trialling and implementing approaches to bring about system - level shifts. By delving deeper into these cases, the evaluation aimed to provide a deeper understanding of the challenges surrounding sustainability of these shifts. Within this area of change, the sample covered private and publicly funded providers to study the nature of influence across the contrasting environments. The four cases were:

• Otago University, Dunedin College of Education
• Auckland University, Team Solutions
• Evaluation Associates, private provider, Auckland
• Education Associates, private provider, Dunedin.
  

Sector survey

A key objective of INSTEP was to transfer learning and understanding gained through INSTEP about inservice teacher education practice to the sector more widely so as to embed the system-level shifts in practice. Indicators identified in consultation with the INSTEP project team to demonstrate these shifts were:

• evidence of deliberate use of inquiry cycle (or similar) in planning, designing and implementing professional development initiatives and their own practice
• evidence of deliberate use of evidence (linking their practice to teacher practice to student outcomes) to inform decision-making at each stage of their work
• evidence of professional learning within the institutions operating in this sector
• evidence of sector taking leadership for continuous improvement
• evidence of conversations across professional development providers.
  

Survey design and administration

Key features of the survey design were:

• use of an online web-based survey instrument, designed to take 10-15 minutes to complete
• INSTEP participants contacted by email by the INSTEP project coordinator to participate
• list of participants included three broad groups – inservice teacher educators including the national facilitators and regional facilitators involved in INSTEP, School Support Service Directors and members of the Sector Reference Group.
  

The survey design and administration process was:

• sample frame – list of all INSTEP participants and contact details were obtained from the Ministry of Education
• sample frame was cleaned – preliminary email contact made to confirm appropriate contact email address and to give advance notice of survey
• questionnaire design was tested with the INSTEP project team
• survey sent on 12 September 2008.
  

Response rate

We achieved a final response rate of 55% (161 participants).

Figure 6: Distribution of respondents by their role in INSTEP

Analysis and reporting

This report is a synthesis of the data gathered from both research activities and comments on the nature of shifts that is occurring as a result of the INSTEP intervention. The process for developing this report was as follows:

• The evaluation team undertook the four case studies over a 12-month period
• The understanding generated from the case study led to design of the survey questionnaire which was administered in October 2008
• Findings from both research activities were ‘tested’ at a number of different forums including the consolidation workshops, the internal Evaluation Advisory Group and the INSTEP project team
• Revisions were made as a result of the feedback from key stakeholders and finalised.
  

The structure of the final report:

• Part 1: Understanding pre INSTEP context for ISTE practice
• Part 2: Demonstrating value of INSTEP
• Part 3: Implications for the future.