Evaluation of the Sustainability of ECE Services During the Implementation of Pathways to the Future — Ngā Huarahi Arataki

1 - Introduction

The Ministry of Education engaged the New Zealand Council for Educational Research (NZCER) and Te Kōhanga Reo National Trust (TKRNT) to conduct a locality-based evaluation in eight localities. The evaluation focused on the three goals of participation, quality and collaborative relationships. The Ministry engaged Health Outcomes International Pty Ltd (HOI) to work collaboratively with NZCER and TKRNT to inform and complement the research by evaluating the sustainability of early childhood education (ECE) services. This report is the Final Report from the evaluation of sustainability. 

Pathways to the Future: Nga Huarahi Arataki

Pathways to the Future – Nga Huarahi Arataki (“the Strategic Plan”) provides the framework and direction for ECE policy development. It has three main goals:

• To improve participation in quality ECE services;
• To improve quality of ECE services; and
• To promote collaborative relationships.
  

There is a specific focus on communities of low participation.

The Strategic Plan is underpinned by three supporting strategies: the funding framework and funding system, a regulatory review, and the evaluation strategy (Ministry of Education, 2005).

The new funding system for ECE was introduced in April 2005. Under the new system, funding rates are linked to services’ cost drivers, creating incentives for and recognising the costs of improving quality. For example, rates of funding are linked to targets for the percentage of registered and qualified ECE teachers employed at the ECE service. The implementation of the funding system is described in section 2.

A review of the regulatory system commenced in 2003 to ensure that the system is consistent with and supports the Strategic Plan with regard to quality and responsiveness to families’ diverse and changing needs. Some objectives of the Plan also required specific regulatory changes, such as improving adult:child ratios and teacher registration requirements. These changes were legislated for with the passage of the Education Amendment Act 2006.

The evaluation of the Plan is taking place over the ten-year duration of its implementation. Stage 1 of the evaluation (including the current sustainability evaluation) took place during 2004-2007. Stage 1 will feed back into ongoing policy development and implementation, contributing to the continuous improvement of the Plan as its implementation progresses.

The Ministry engaged NZCER and TKRNT to conduct locality-based research to investigate and analyse the actions in the Plan and how the actions contribute to outcomes (focusing on the three goals of participation, quality and collaborative relationships) in the eight localities in the sample. The Ministry engaged HOI to evaluate the sustainability of ECE services during the implementation of the Plan. This complements the work of NZCER and TKRNT, and HOI has worked collaboratively with these groups throughout the evaluation period.

Sustainability

In the context of this evaluation, the term sustainability refers primarily to the financial sustainability of ECE services – that is, the ability of services to cover their costs on an ongoing basis.  However, it needs to be acknowledged that ECE services require more than positive cashflow to remain sustainable. Ultimately, ECE services are only sustainable to the extent that there is demand for the services offered. Thus, sustainability depends on providing a service that families and whanau want (and can afford) to utilise. Sustainability also depends on supply factors such as teacher supply and a cost-effective regulatory framework. These considerations inextricably tie the issue of sustainability to issues of participation, quality and collaborative relationships.

The Plan recognises that increased participation in ECE relies, inter alia, on having ECE services that are sustainable – that is, “if parents, families and whanau are to keep their children attending ECE services, those services need to be sustainable” (Ministry of Education, 2002a). This dependence of participation on sustainability is also articulated in the logic model.

Equally, however, the converse is true: ECE services will only be sustainable if parents, families and whanau keep their children attending those services. The costs and revenues of ECE services are closely related to participation. For example, four-fifths or more of most services’ revenue streams (from both Government and parents) are related to the number of children who utilise the service and their hours of attendance. (Health Outcomes International. 2003) Thus, there is a feedback loop between participation and sustainability which could either reinforce any growth or compound any decline in the sustainability of ECE. This consideration underscores the importance of evaluating the impacts of the Plan on sustainability.

As well, there are relationships between quality, collaborative relationships and sustainability.  Stability is recognised as an important feature of quality ECE as children in stable arrangements are more likely to demonstrate positive educational outcomes (Kershaw, Forer, and Goelman, 2004). ECE services that are financially sustainable are also in a better position to improve quality and develop collaborative relationships than services in financially fragile circumstances. Thus, sustainability is a precondition to all three goals of the Strategic Plan. 

Improving quality and collaborative relationships entails additional costs (e.g. increased teacher costs due to employment of registered and qualified teachers, increased teacher costs per child-hour due to reduced group sizes; time and resources invested in developing and implementing self-review processes; and involving parents and whanau in teaching, learning and assessment). In recognition of this, the new funding system included increased Government funding for ECE services to reflect specific cost increases from the Plan. For example, the Plan (p9) states:

• "Requiring ECE services to employ registered teachers increases the costs to the services.  The Government recognises that for ECE services to pass on too much of this cost to service users would decrease participation.  The Government will therefore fund services so that increased teacher quality does not come at the cost of decreased participation".
  

The Strategic Plan and logic model also identify the following key actions to improve sustainability of ECE services:

• The provision of Equity Funding to promote access to quality community-based ECE services through acknowledging additional costs faced by ECE services in some communities.  In particular, this relates to ECE services that are in low socio-economic and rural communities, cater for children with special needs or who have English as a second language, or provide an immersion service in languages other than English;
• Introduction of funding and regulations that better support quality ECE services to meet the needs of children, parents, families and whanau;
• Working with ECE sector umbrella groups and organisations to ensure that advice and support for governance, management and professional leadership are available to all ECE services; and
• Providing leadership development programmes to strengthen leadership in ECE services.
  

Evaluation Objectives

The objectives for the evaluation of the sustainability of ECE services during the implementation of the Plan were:

• Establish baseline measures in 2004 related to sustainability;
• Describe the implementation of the funding system and the impacts of this, together with the Strategic Plan, on sustainability;
• Determine whether there is any change from baseline measures in 2006;
• Analyse the evidence from each locality to understand, with respect to the logic model and the stated goals (of the Strategic Plan):
  
  
  • How working toward and achieving the goals of participation, quality and collaborative relationships, together with the receipt of increased funding to support the Strategic Plan, are affecting the sustainability of ECE services;
  • How the sustainability of ECE services is affecting their capacity to improve participation, quality and collaborative relationships;

• Analyse similarities and differences in relation to impacts on sustainability in different contexts; and
• Provide evidence of any unintended outcomes in relation to sustainability.
  

The reference to baseline (2004) and follow-up (2006) measures in these objectives reflected an initial expectation that the principal source of data for the evaluation would be the locality-based case studies involved in the evaluative work being carried out by NZCER and TKRNT. However, the study design evolved during the course of the evaluation as new national data sources became available which were more suitable to evaluating the sustainability of the ECE sector. As a result, the evaluation has been broadened from a pre-post analysis of case study data to include trend analysis of national data.

Evaluation Questions

The following evaluation questions were developed for the evaluation of the sustainability of ECE services during the implementation of the Plan:

• What are the changes in patterns of expenditure (due to the implementation of the Strategic Plan)?
  
  
  • By cost component/driver: What are the additional costs related to the implementation of the Strategic Plan, and why do they occur?  What are the main extra activities/investments being undertaken?
  • By source of revenue: To what extent does additional Government funding contribute to meeting the additional costs? Has the proportion of Government to total funding changed?  Has the cost per family changed?
    

• To what extent is ECE service sustainability changing/improving as a result of changes in revenues and costs (due to the implementation of the Strategic Plan), as indicated by:
  
  
  • Surplus/deficit: How well do annual revenues cover annual costs?
  • Cash reserves: Do cash reserves increase or decrease with the implementation of the Strategic Plan and the funding system?
  • Demand/Participation (as indicated by changes in occupancy, utilised child-hours)
  • Supply (as indicated by entries/exits/changes in ownership of services, available child-hours)

• Is the funding system having the intended effects?
  
  
  • Has the additional funding covered the costs of implementing the Strategic Plan (or have parent fees and/or voluntarism had to increase?)
  • Has the additional funding provided incentives to meet the teacher registration targets?

• How are these changes contributing to improved quality of ECE services, increased participation in quality ECE services, and promoting collaborative relationships?
  
  
  • How have changes in financial sustainability impacted on capacity to improve quality, participation, and collaborative relationships?
  • To what extent have the Government’s support interventions under the Plan been accessible?  To what extent have they been taken up?  What effect have they had on the sustainability of the service and/or the capacity of the service to improve participation, quality and collaborative relationships?