Evaluation of the Teacher-Led Innovation Fund: Final Report

Publication Details

The purpose of this evaluation was to provide timely and relevant information to:

  • Inform continuous improvement of TLIF’s design, implementation and monitoring;
  • Complement monitoring and review information sources to provide information about how the fund is operating;
  • Expand the knowledge base about the nature of innovative teaching practice, collaborative inquiry, and mobilisation of new knowledge.

Author(s): Uni Services, University of Auckland. Report prepared for the Ministry of Education.

Date Published: April 2018

Executive Summary

The Teacher-led Innovation Fund

The Teacher-led Innovation Fund (TLIF) is a Ministry of Education initiative designed to support quality practice that improves student achievement and that can be shared and adapted for use across schools and kāhui ako. It is open to all primary and secondary kaiako/teachers in state and state-integrated kura/schools that meet application criteria. TLIF is one of three initiatives that are part of the broader IES Government initiative aimed at raising student achievement and strengthening teaching and education leadership. TLIF provides funding for groups of teachers to develop innovative teaching practice in order to improve learning outcomes, particularly for Māori students and Pasifika students, students with special education needs and students from low socio-economic backgrounds.

TLIF provides teachers with time and expert support to inquire into their practice to find ways to help students succeed; and to share what works with other schools and educators. TLIF aims to have long-term impacts on students, the workforce, early learning centres/schools/kura and on teaching practice.

The underpinning principles of the TLIF fund are:

  • Collaborative Inquiry: groups of teachers working together to understand their practice and the impact on students
  • Expertise: leveraging expertise within schools and kāhui ako, complemented by support from external experts
  • Flexibility and adaptability: project teams and fund administration need to respond to change
  • Knowledge mobilisation: knowledge gained from projects needs to be shared.

Purpose of the Evaluation

The purpose of this evaluation was to provide timely and relevant information to:

  • Inform continuous improvement of TLIF's design, implementation and monitoring;
  • Complement monitoring and review information sources to provide information about how the fund is operating;
  • Expand the knowledge base about the nature of innovative teaching practice, collaborative inquiry, and mobilisation of new knowledge.

Key Evaluation Questions

The evaluation was designed to provide insights into the following key evaluation questions:

  1. What does an effective TLIF project look like? How can success be described?
  2. To what extent are TLIF projects achieving early outcomes, that is, developing sector capability and capacity for collaborative inquiry?
  3. To what extent is new knowledge about effective teaching, particularly for target groups, being mobilised from TLIF projects?
  4. What factors (barriers, enablers, others) contribute to successful and less successful outcomes for TLIF projects?
  5. To what extent is TLIF making a difference for learners?

Summary of Methodology

Conducted in 2017, this evaluation took a mixed-methods approach, drawing on both qualitative and quantitative data from several sources and involving TLIF project participants in both completed and in-progress projects (teachers, project leads and external experts) as outlined below.

 

Sample and data sources

Questionnaires:
A total of 149 respondents completed the online teacher questionnaire. Of these, 108 were TLIF project team members (72.5%) and 41 were project leads (27.5%). The 41 project lead responses indicated a project-level response rate of 47%. Of the 41 projects respondents were involved in, 16 (39%) were nearing completion or completed. Twenty two were well underway (53.7%) and three were recently started (7.3%). A total of 23 external experts also responded to the online questionnaire.

Documents:
Final reports from Round 1 were analysed and key themes extracted relating to:

  • Project Constraints: Explanations about the reasons project participants were doing and not doing particular things;
  • Actions: Descriptions of what project participants were doing and not doing;
  • Project Consequences: Indications of what happened as a result of TLIF project participants' actions.

In addition, for case study projects, all available documentation including concept forms, proposals and reports, was reviewed and summarised prior to case study data collection.

Case studies:
Case studies were based on data from interviews, focus groups, observations and documents

  • Case study A: Primary - eight interviews (with five teachers), four observations;
  • Case study B: Secondary - 10 interviews and two observations;
  • Case study External Experts: 23 questionnaire responses two focus groups and five interviews;
  • Case study Proposals: five interviews with those responsible for both successful and unsuccessful proposals.

Summary of Key Findings

Findings from across the data sources indicate notable successes in terms of the initiative overall, and some problematic patterns that could be addressed in ways that continue to strengthen the TLIF initiative. The findings are summarised below.

Problematic Patterns Successes and Promise
  • Data confidence and capability issues.
  • Lower impact on teachers' skills (in particular data use skills) or on students than on dispositions, practice shifts or knowledge and understandings.
  • Discussions tended to be more supportive/collegial than challenging/rigorous.
  • Often almost exclusive emphasis on successes and little attention to failures.
  • Attention to priority learners or target groups in proposals not always carried through to the data collection, analysis and reports.
  • Teacher confidence lower in relation to collaborative items than items relating to own inquiry and practice.
  • Uncertainty about the robustness of claims about the impact of innovations.
  • Overall very high regard for the opportunities presented through involvement in a TLIF project.
  • Big pockets of profound positivity about the TLIF experience.
  • Almost unanimous endorsement of TLIF.
  • Impact on dispositions, practice shifts and knowledge and understandings.
  • Establishment of conditions for collaborative inquiry.
  • Opportunities to make deprivatising practice and engaging in professional dialogue about it a norm.
  • Innovative (for the teacher/s involved) practices being tried and explored.
  • Attention to student voice for insights into both teaching and learning.
  • Some exceptional projects.


These points are elaborated below with reference to data from the evaluation.

Problematic Patterns
  • Data confidence and capability stands out as the biggest issue facing teachers, teams and those supporting them. Teachers themselves reported much lower confidence in their own ability to analyse data in ways that give important insights (6.18), than other aspects of their inquiry work. Those issues were also evident project reports where the quality of the data collected was often questionable, with typically generic descriptions of analyses and generality in claims about impact and causality. For example, claims like this one-"Overall the inquiry…has resulted in the uplifting of student achievement, particularly for our target students, and the teaching and learning of our teachers" were often left without evidence to support them, and no indication that data had been collected or analysed in a way that supported the claim of success.
  • As a consequence of the issues regarding data capability, there was often a lack of robustness in the claims able to be made about the impact of innovations on learners and uncertainty about what knowledge could be mobilised to others.
  • Overall, teachers reported lower impact on their skills (in particular data use skills) (6.22) or on students (6.30) than on dispositions (6.91), practice shifts (6.79) or knowledge and understandings (6.75).
  • Discussions tended to be more supportive (7.55) or collegial (7.50) than challenging (7.17) or rigorous (7.10). That finding is important since we found a robust positive relationship between the quality of project discussions (including rigour and challenge) and impact of TLIF. That is, the better teachers rated their project discussions, the stronger they perceived the impact of the project on their knowledge (r = 0.75**), dispositions (r = 0.64**), practice shifts (r = 0.67**), skills (r = 0.55**) and on students (r = 0.59**).
  • The finding that there was often an almost exclusive emphasis on success and very much less attention to any failures is problematic given the likelihood of failures being associated with innovation, the importance of attention to failure and its causes to puzzles of practice, and the role of fallibility and open mindedness as dispositions conducive to high quality inquiry
  • Attention to priority learners or target groups evident in proposals was not always carried through to the data, data analysis and reports.
  • Teacher confidence lower in relation to items about ability to collaborate with (and influence) other than items relating to own inquiry and practice.
  • Uncertainty about the robustness of claims about the impact of innovations. Most claims were quite general-they referred to increased achievement of students without specifying which students were impacted, in what ways, or how much their learning improved, and without sufficient supporting evidence. Some projects went further, and specified details of which students improved, how and how much. But, very few provided robust explained claims where there was, in addition, clear, logical and evidence-based explanations about how the innovation was related to improvements.
Successes and Promise
  • Overall there is very high regard for the opportunities presented through involvement in a TLIF project-teachers, project leads, and external experts were grateful for the opportunity to participate, and keen to continue.
  • Comments from teachers often indicated quite profoundly positive attitudes towards their experience of TLIF. Some referred to TLIF, for example, as the best professional learning experience they had ever been involved in, inspirational, a turning point, or an absolutely rewarding opportunity that was a highlight of their professional life.
  • Almost unanimous endorsement of TLIF from both teachers and external experts involved with project teams. When asked if they would recommend to others being involved in a TLIF project, 99.2% of teacher indicated either 'yes, definitely' or 'yes, with some conditions'. All external experts who responded indicated similarly (100% endorsement).
  • The greatest impact on teachers, indicated by questionnaire ratings, was on their dispositions (6.91), practice shifts (6.79), and knowledge and understandings (6.75). Ratings for impacts on students (6.3) and skills (6.22) were slightly lower
  • While the quality of TLIF projects is variable, in most, their efforts have resulted in improved conditions for collaborative inquiry with more regular and routine opportunities for deprivatised practice, professional dialogue about innovations being investigated, and learning with/from colleagues they had not had the chance to learn with/from before.
  • Many teachers reported that TLIF had provided the impetus for them to try new and different practices that were not part of their existing teaching repertoire. In many cases, the notion of 'innovation' was a relative one-the practices being investigated were new to the particular teacher or group of teachers, but not necessarily highly innovative in terms of pushing boundaries or revealing practices new to the field. In some cases, highly innovative approaches were explored, offering new insights that others could learn from.
  • Many teachers described attention to gathering student voice for insights into both teaching and learning that they had not previously paid so much attention to. While student voice data tended to focus mostly on their preferences or satisfaction (rather than data from students about the impact of particular innovations or the reasons for the success or otherwise of an innovation) the move to include data from students is commendable.
  • The TLIF initiative has revealed some project teams with considerable expertise in leading and carrying out inquiries, learning about the impact of innovations, and mobilising knowledge from their inquiries in a way that is relevant to others. These exceptional projects provide a resource for others, and a basis for sharing not only innovative teaching/classroom practices likely to be effective, but productive processes for collaborative inquiry.

Characteristics of Effective TLIF Projects

Insights from the various data sources suggest that the following characteristics are associated with more effective TLIF projects. By effective, we refer to the TLIF purposes of a positive impact on outcomes for teachers (their knowledge and understanding, skills, dispositions and practice), for students (their learning experiences, relationships, and achievement), and the generation of new knowledge, understandings and learning that can be mobilised to others. These characteristics both increase the likelihood that innovations teams try are successful and, equally important, the likelihood that teams will determine, in a timely manner, when and why they are not successful where that is the case.

Figure 1: Characteristics

Enablers and barriers

The extent to which project teams were able to carry out high quality inquiries into their innovation was influenced by a range of barriers and enablers, summarised below:

Barriers Enablers
  • Logistics and project administration (including time, release issues, personnel changes)
  • Resistance
  • Data capability
  • Reporting demands
  • Collaboration time
  • Inquiry orientation
  • Knowledge-building opportunities
  • Contribution of external expertise
  • Leadership support

Summary of Key Messages

With a focus on the purpose of the TLIF evaluation to inform continuous improvement of TLIF's design, implementation and monitoring, the following figure presents a summary of key messages, the current 'story' of TLIF as it were. These messages, derived from the evaluation evidence are key because, if attended to, they hold much promise for next steps in improving the initiative overall. Some of these messages represent problems to be solved, but there are strong grounds and good conditions for solving them, given the widespread endorsement of and regard for TLIF, the establishment (or strengthening) of relational conditions conducive to inquiry and innovation, and the presence of some exceptional projects to learn from.

Figure 2: Key Messages 

Recommendations

The following recommendations, in response to the evaluation findings, are proposed for discussion.

Figure 3: Recommendations

Conclusion

In summary, TLIF is making a difference. It is making a difference to how teachers work with each other, and with their learners, and creating opportunities for trying innovations and exploring practice. The extent of the difference TLIF is making, and the quality of the opportunities to explore practice, varies somewhat across projects. This signals a need for continued attention to building the capabilities teams require, and to mobilising knowledge about how to do this challenging work, well. How can teams be supported to gather and analyse data to inform their inquiry, and strengthen the claims they can make about the impact of the innovations they try? How can teams be supported to strengthen the rigor of the discussions they have as their inquiry is initiated and unfolds? How can the most successful projects be identified and shared with others in ways that help others both learn about innovative practice, and improve the quality of their own projects?

It is hoped that the insights from this evaluation are useful for informing inquiry into how to further improve TLIF, an initiative that has much to offer teachers and students alike

Footnotes

  1. Note, since 2017 TLIF has also been open to early learning services.
  2. Items were scored on an 8-point Likert scale ranging from '1' for 'Strongly Disagree' to '8' for 'Strongly Agree': Strongly disagree (1); Mostly disagree (2); Moderately disagree (3); Slightly disagree (4); Slightly agree (5); Moderately agree (6); Mostly agree (7); Strongly agree (8)

Contact Us

Education Data Requests
If you have any questions about education data then please contact us at:
Email:      Requests EDK
Phone:    +64 4 463 8065