Report on the Evaluation of the Early Childhood Centres of Innovation Programme Publications
Publication Details
This report presents findings of an evaluation of the Early Childhood Education (ECE) Centres of Innovation (COI) programme, which ran between 2003 and 2009. The evaluation focused on centres that entered the programme between 2003 and 2006 (Rounds One to Three). The purpose of the evaluation was to determine the impact of the COI programme within COI services and in the wider ECE sector on teachers practices and children's learning outcomes.
Author(s): Robyn Gibbs and Dr Jenny Poskitt, Evaluation Associates Limited.
Date Published: June 2009
Executive Summary
The Early Childhood Centres of Innovation (COI) programme commenced in 2003. It was established as part of a broad early childhood education strategic plan1. The purpose of COI is to promote self-directed research around teaching practices and aspects of learning in early childhood services and centres2. Tenure in the COI programme is three years. A requirement of the programme is the dissemination of research findings to a broader educational community. This dissemination is usually undertaken concurrently with the research work. To be eligible to be a COI, centres should be actively involved in innovative practice. In the selection process centres outline, in their Expression of Interest, their plans for researching within a broad focus area selected by the Ministry of Education (the Ministry).
COI are supported by an experienced educational researcher (referred to as a Research Associate) and a Ministry appointed Research Leader. The COI and the Research Associate (RA) enter into individual contracts with the Ministry to carry out their respective roles. Accountabilities, in the form of milestone reports, are expected of each participating contracted party. At the time of writing this report COI was into its fifth round. As part of Budget 2009, the Government announced that the COI programme would end on 30 June 2009, with the three Round Four and one Round Five COI ending prior to completing tenure.
In June 2008, the Ministry contracted Evaluation Associates Ltd to undertake an evaluation of the COI programme. This evaluation focused on centres that were involved in Rounds One to Three – cohorts of centres that entered the programme between 2003 and 2006. It sought to address evaluation questions relevant to two main groups; (a) centres who were COI and (b) other centres who found out about the work of COI (referred to in this report as Second Wave).
The overall questions the evaluators were charged with answering were:
- How, and to what extent, have teacher researchers and their teaching practices changed as a result of their participation in the programme? How have children's learning outcomes changed? (effects within COI services).
- What difference is the COI programme, as a whole and elements within it, making to improve quality within the early childhood education (ECE) sector, and to outcomes for children? (effects in wider sector).
Sub questions addressed at the COI level were:
- What factors/conditions are key in facilitating significant change (growth amongst teachers/outcomes for children and their parents) in teaching and learning at the service level?
- How is this significant?
- To what extent is this significant?
- What, if any, differences are discernible 'during COI tenure' compared with 'post COI tenure'?
- How, and to what extent, does COI work contribute to improved outcomes for teacher researchers?
- What are the outcomes for young children, toddlers and infants?
- What are the outcomes for parents and whānau?
- What differences, if any, are discernible between COI rounds?
Sub questions addressed at the wider sector level were:
- What aspects of the COI work (innovative practice, action research and dissemination) have an effect?
- For whom do these effects apply?
- How do these effects apply?
- What has been/is the level of dissemination of COI research findings?
- How well, and in what form are audiences responding to it?
- What is the impact on the body of knowledge about effective teaching and learning practices?
- Is the movement of COI teacher researchers to other employment within the sector significant when considering the impact of the programme within the sector? What are the impacts, if any, of COI teacher researcher mobility on the sustainability of programme benefits within the COI and/or on the sharing of programme benefits with the wider sector?
Across each COI, a number of participants were involved in the evaluation. The views of these participants were collected through interviews and focus group meetings and form the basis of this report. The report is supplemented by the views of Second Wave participants who completed a paper or electronic survey. The number of people involved, and data analysis techniques used, are reported in the methodology section of this report.
Following is a summary of the key evaluative questions together with suggestions for improvements to the programme. Please note that the first three evaluation questions are combined to provide a more coherent summary.
In relation to evaluation questions one, two and three:
Where COI were working well in terms of self reported positive benefits3 for teachers, parents and children, participants and/or evaluators were able to identify a number of factors that enabled these outcomes. Conversely, where teachers were less sure about the outcomes of their COI work, a significant number of corresponding disabling factors were present.
In terms of evaluator judgment, the most significant enablers to successful participation in a COI programme were the knowledge, capability and dispositions that teachers brought. Other factors that contributed included effective leadership by head teachers/supervisors and RAs, and support from regional associations and umbrella groups.
Some COI entered the programme by virtue of their potential to add to a minimal body of local research rather than their proven history of successful innovation. These COI reported difficulties in terms of staying engaged in their COI projects and cited a range of personnel, infrastructural and leadership issues that impacted on them. Nonetheless, at the time of the evaluation all but one COI had completed their tenure. The remaining COI had gained an extension of its project.
Positive relationships are critical within the COI model. Where COI were successful, teachers were aware of the need to foster good relationships with parents and their colleagues. Teachers valued the collaborative relationships they had with RAs, other COI in their cohort, and non-COI organisations with which they disseminated their findings. Some centres were concerned about the lack of support they received from their regional associations/umbrella organisations. Other COI were grateful for the practical and moral support their associations/umbrella organisations provided.
Consistent with the notion of communities of learners (Wenger, McDermott, & Synder, 2002), most COI operated within a culture of distributed leadership, apportioning leadership responsibilities across team members according to the curriculum and individual personality strengths. Teachers valued this approach and believed that it had contributed to their personal and professional growth.
With the exception of a few who felt that the COI work disrupted their children's programme, parents were highly complimentary about the COI work being carried out in their centre. They enjoyed seeing the enthusiasm of their children, and being involved in aspects of the programme.
In relation to evaluation question four:
Teachers reported that they had learned a lot about researching, and were motivated to continue researching their professional practice, but were frustrated by time pressures and inadequate resources post-COI tenure to continue further research. Many teachers found dissemination a time-consuming activity, and engaging in both research and dissemination at the same time was a challenge. These teachers believed that they would be able to do a better job of both if some of their dissemination activity was carried over to a funded fourth year.
Many teachers wanted to continue research activities post COI but were not sure of what alternative funding options were available. The evaluators recommend the Ministry consider ways to support teachers to become aware of these funding sources.
In relation to evaluation question five:
Evaluators concluded that there were many good outcomes for teachers, including opportunities to engage in critical thinking that challenged their previously held assumptions about teaching and learning. As a result of their collaborative professional learning, many teachers reported that their planning, formative assessment processes, and increased use of ICT within the learning and teaching programme were of a higher quality. Teachers reported that RAs played a significant role in exposing them to a range of theories that extended their content and pedagogical knowledge. These comments were confirmed by the Research Leader, RAs and in many of the COI milestone reports.
In relation to evaluation question six:
The findings were less clear. Evaluators concluded that teachers had developed some useful processes around assessment and planning for individual children. However, it appeared that teachers knew little about the impact of their initiatives on children as a whole because they had not established processes for gathering and analysing centre-wide data.
The major recommendation of this evaluation is the tightening of processes for managing programme review so teachers are in a better position to know what practices are most beneficial for children.
To ensure better processes around centre evaluation the COI model would benefit from some structure being applied at Ministry level. The priority of operating in an evidence-based environment should be: (a) articulated in the expectations for COI as they consider entering the programme; (b) supported by the Research Leader through her work with COI; (c) actively promoted through the Participant Action Research (PAR) process by the RAs; and (d) reflected in the milestone and final reporting expectations of COI.
The evaluation focus could be further developed by drawing on the expertise of respected researchers working in the field of assessment and evaluation in the early childhood sector.
In relation to evaluation question seven:
Teachers stated that being a COI provided them with the time to consider ways they could foster meaningful engagement with families. Teachers believed that, as a result of COI, there were stronger learning–focused relationships with parents in which information was exchanged about children's learning. Parents also indicated that they felt more valued and had a greater desire to be involved in the centre programme.
Centres, however, had limited evidence of the impact of the COI programme on parents/whānau. Some services had made a good start by carrying out surveys or observations of parents within the centre. The usefulness of the information was limited; however, because these data were not followed up to find out what changes had occurred over time. This meant that centres did not comprehensively understand the impact of the COI work on parents as a group.
In relation to evaluation question eight:
Some differences were found. These included greater emphasis amongst later cohorts of COI on working collaboratively as they carried out their research and professional learning, and an increasing trend to disseminate in the pre service teacher education sector. Some processes at Ministry level have also improved thereby bringing greater clarity about how to implement the model.
Other key findings related to the wider sector impact that have not already been mentioned above include dissemination and recommendations about how the Ministry could improve the COI model. Dissemination is a fundamentally important component of the COI programme. COI is costly but, by virtue of the highly active programmes of dissemination that many COI engaged in, the cost can be justified. Many other services heard about COI work and have accessed their findings. The findings are valued because they have considerable relevance to the New Zealand early childhood sector. The evaluation data suggests that many Second Wave centres have made changes to their own programmes as a result of engaging with COI ideas.
In the view of the evaluators, dissemination in its many forms is much more effective in stimulating change in the sector than the movement of COI teachers between centres. The evaluators have some suggestions about how to make COI findings accessible to a wider audience, including other centres and the schooling sector that currently do not know about COI or the research they have produced.
For teachers, some valued aspects of the COI model are the financial and property resources that come to their centres through being a COI. They also valued the support they received from the RAs who mentored them through their research projects.
The report concludes with recommendations which include:
- the Ministry facilitating COI work through resourcing more quickly, and providing coordinated administrative support
- developing memoranda of understanding between COI, the Ministry and regional associations/umbrella organisation so that all parties have clear understandings of their role and responsibilities in supporting COI processes and outcomes
- providing better support to COI seeking ethics approval and access to libraries
- extending the pool of RAs to include those who are not currently employed within a tertiary institution, but who have the necessary skills to undertake the RA role effectively. The justification for this recommendation is that the funding for COI has not increased to keep pace with university expectations of staff remuneration.
The eight principles of effective professional learning in ECE4 were used as touchstones in this evaluation. Professional learning is deemed to be effective when it:
- incorporates participants' own aspirations, skills, knowledge and understandings into the learning context
- provides theoretical and content knowledge and information about alternative practices
- involves participants investigating pedagogy within their own early childhood settings
- involves participants analysing data5 from their own settings
- has as its core critical reflection which enables participants to investigate and challenge assumptions and extend their thinking
- supports educational practice that is inclusive of diverse children and their families and whānau
- helps participants to change educational practice, beliefs, understandings and/or attitudes
- helps participants to gain an awareness of their own thinking, actions, and influence.
On the basis of these criteria, the evaluators conclude that COI has provided teachers with some valuable opportunities to enhance their professional practice. The following report describes these opportunities and the outcomes for children, their families/whānau and teachers.
Footnotes
- COI is one of the initiatives undertaken as part of the Pathways to the Future: Nga Huarahi Arataki strategy plan for early childhood education.
- To avoid unnecessary repetition we will use the term centre as a generic term for all early childhood services.
- In a later section, the evaluators comment on the evidence that was available to them in terms of outcomes for teachers, parents and children.
- These principles were sourced from Characteristics of professional development linked to enhanced pedagogy and children's learning in early childhood settings: Best Evidence Synthesis (BES) (Mitchell & Cubey, 2003a)
- Evaluators thinking with respect to data is informed by the theories articulated by Carr, May and Podmore (2000), specifically that action research should be guided by a set of action research tools that allow centres to collect a variety of data. The findings from the analysed data allow teachers to take appropriate responses. The data should be relatively easy to collect, precise, stimulate reflection, challenge assumptions, be capable of analysis and lead to action and change.
Navigation
Contact Us
Education Data Requests
If you have any questions about education data please contact us:
Email: Requests Data and Insights
Phone: +64 4 463 8065