Locality-based Evaluation of Pathways to the Future - Ngā Huarahi Arataki:
Stage 1 Report
This is the second and final report from the stage 1 evaluation of the ECE Strategic Plan. It reports on progress towards the three goals of the Plan and identifies changes that have occurred since baseline data were collected in 2004.
Author(s): Linda Mitchell and Edith Hodgen, Report for the Ministry of Education.
Date Published: August 2008
This report is from the first phase of the locality-based longitudinal evaluation1 of Pathways to the Future - Ngā Huarahi Arataki, the strategic plan for ECE. This evaluation provides a baseline picture of how things were in mid-2004 in relation to the participation, quality, and collaborative relationship goals of Pathways to the Future - Ngā Huarahi Arataki and before the major ECE strategic plan policy changes began, and of changes that occurred in services and for parents in the eight localities in the study between 2004 and 2006 as strategic actions got underway. The evaluation complements the evaluation of financial sustainability of ECE services 2004–2006, using the same sample of ECE localities and services, undertaken by Health Outcomes International (HOI) (King, 2008).
The report addresses five evaluation questions based on the main goals of the strategic plan, asking to what extent, in what ways, and how effectively, has the strategic plan increased participation in ECE; supported parents’ ability to engage in work and training; improved the quality of ECE; facilitated the formation of collaborative relationships between ECE services, parent support and development, schools, health and social services; and improved children’s early learning foundations.
Data were collected from a sample of 46 individual services of different ECE service types in each of eight localities, chosen by the MOE and the evaluators to provide some diversity on key relevant variables. Methods included a parent survey, management interview and questionnaire, teacher/educator interview, and observations of process quality in 2004 and 2006. Process quality refers to the environment, interactions, and relationships that occur in an early childhood setting and shape children’s learning opportunities and experiences in that setting. MOE national and locality datasets were used to provide a context for changes occurring at ECE service level.
The logic model that was developed by Patricia Rogers (2003) for the MOE shows the likely paths between the goals or intended outcomes of the strategic plan (increasing participation, improving quality, and promoting collaborative relationships), the strategies and actions to support those intended outcomes, and the intermediate outcomes that would occur along the way (such as more registered teachers in ECE, quality teaching and learning practices, and ECE services are accessible). We developed a set of indicators to measure the nature and extent of changes on each intended outcome from 2004 to 2006. We analysed change according to locality characteristics (rural/urban, socioeconomic levels, and population characteristics), service characteristics (type, teacher-led or parent/whānau-led, sessional or full-day), and child characteristics (proportion of children who are Māori, who are Pasifika, who are under two years old, whether the service receives Equity Funding, and in relation to parental data about the child). Data were reported on intermediate outcomes, and analysis made of changes in intended outcomes in relation to changes in intermediate outcomes. We found out about support and barriers related to change through interview and survey data.
Strategic plan actions
The main strategic plan actions at April 2006 that had been in place between 2004 and 2006 were:
- Improving quality: increasing the proportion of qualified registered teachers in teacher-led services, publishing Kei Tua o te Pae. Assessment for Learning: Early childhood exemplars (Ministry of Education, 2005c), and other professional resources, funding professional development, and establishing Centres of Innovation (COI).
- Supporting participation: analysis and development of the current network of ECE services, Promoting Participation Projects in areas of low ECE participation, land being set aside for ECE services on new school sites, advice and support for new services, and advice and support to help services meet community need.
- Promoting collaboration: MSD parent support initiatives, and Book 5. Assessment and Learning: Community (Ministry of Education, 2005c) early childhood exemplars, and professional development related to working with parents and whānau.
- Supporting strategies: Equity Funding,2 a new funding system based on cost drivers and substantial increases in funding rates. Free ECE for three- and four-year-olds for up to 20 hours per week had been announced and the regulatory review was occurring, but policy changes were still to be implemented.
Many of the government initiatives were fairly recent and we did not expect to see marked changes in overall quality, participation, or collaborative relationships between 2004 and 2006.
“Participation” indicators in this study included whether or not children attend ECE during their preschool years, the duration of attendance, regularity of children’s attendance, children’s starting age in ECE, and their weekly hours of attendance. MOE national and locality-level data set a context for data from the 46 services.
Studies in the US and UK3 that compare the performance of children with and without ECE experience, show benefits during schooling for those with ECE experience. Recent research from longitudinal studies suggests that an early starting age in good-quality ECE before age three is associated with gains for children’s learning and development, but there is mixed evidence about whether starting before age two is more advantageous than starting between ages two and three.4 The longitudinal Competent children, competent learners study, large-scale Effective Provision of Preschool Education study,5 and other US and Swedish research suggest that two or three years attending a good-quality ECE service before starting school is associated with gains for children’s learning.
Changes in ECE participation from 2004 to 2006
At national level:
- A higher proportion of children were experiencing ECE prior to school entry in 2006. The proportion of school entrants who had participated in ECE before they started school rose slightly from 2004 to 2006 from 94.1 percent to 94.5 percent. The percentage of three-year-old ECE enrolments rose as a proportion of the estimated population of three-year-olds from 95.2 percent to 96.6 percent in 2006. The percentage of four-year-old ECE enrolments fell slightly from 103.3 percent to 103 percent, but was still over 100 percent (because some children are counted more than once if they attend more than one ECE service).
At ward-level for the eight localities in this study:
- A slightly higher percentage of younger children were enrolled in ECE in 2006: 18 percent were aged two in 2004 compared with 19 percent in 2006, and 11 percent were aged one in 2004 compared with 12 percent in 2006. This pattern is consistent with national trends.
- Children were enrolled for longer hours in 2006, with 20 percent of children enrolled for 30 hours or more per week in 2004, and 22 percent in 2006. Under-one-year-olds showed the greatest shift towards longer hours, from 27 percent spending 30 hours or more at ECE in 2004 to 43 percent in 2006.
- Income levels affected children’s participation: the two localities where the median family income was the lowest had very low percentages of ECE participation prior to children starting school—74 percent and 78 percent respectively in 2006. This compares with national figures of children attending a decile 2 school of 86 percent.
At service level:
Data on ECE participation at service level were gathered from parents on two dimensions: “Duration of participation” measured by the proportion of a child’s age spent in ECE and starting age, and “Intensity of participation”, measured by weekly hours of attendance.
The main shift in indicators of “Duration of participation” from 2004 to 2006 was for children in the study services to have started ECE at a slightly younger age in 2006 (two years in 2004, and one year nine months in 2006); and a slight decrease in those starting after the age of four. Within these patterns, starting age was still very late for some children who started ECE after the age of four (2 percent in 2006).
- Duration in months of ECE participation was about half the time of the child’s life (on average a child in this study might be predicted to have more than 2½ years of ECE experience before starting school). This amount of time should benefit children if the ECE service is of good quality. Regularity of children’s attendance in ECE, which may also affect children’s learning opportunities, was gathered from parents for the first time in 2006. Eighty-three percent of children were reported to have regular attendance.
- “Intensity of ECE participation” in the study services remained at much the same level in 2006 as in 2004, on average 10–15 hours per week. These hours of attendance are within a range that should benefit children, but perhaps offer less opportunity for cognitive gain than 15–20 hours for children over two (Loeb, Bridges, Bassok, Fuller, & Rumberger, 2005). A fifth of younger children (under three years) in the study services were attending an ECE service for more than 30 hours per week in both 2004 and 2006 compared with 11 percent of three- and four-year-olds.
- Overall, ECE service accessibility remained much the same between 2004 and 2006 , but with some needs for ECE that were not being met by the study services in relation to hours, especially from those using sessional services and parents with children under two. In 2006, 31 percent of parents would like to use more hours of ECE, but were unable to do so, because their current ECE service did not provide the hours wanted (15 percent), additional hours were too expensive (7 percent), or the service did not have places for the times the parent wanted. Over half wanted to use up to four hours more only. Further parental pressure for more hours is likely now that ECE is free for many three- and four-year-olds attending teacher-led services (36 percent of parents wanted to increase their hours when ECE is free).
- Children attending more than one service had increased slightly from 20 percent in 2004 to 24 percent in 2006. Twenty-three percent of these used more than one service to fit into parent working hours, but the most common reason was because parents thought that using two or more ECE services offered benefits for the child, since each service offered different and complementary learning experiences.
- Overall, the main issues related to ECE service sustainability of enrolments, staffing, and finances were staffing sustainability, related to management finding it difficult to meet qualified staffing requirements, especially when relievers were needed. Most managers thought they were more financially sustainable or there was no change from 2004 to 2006. This finding compares with King’s (2008) evaluation finding that the ECE sector as a whole is financially sustainable and perhaps becoming more so. King has examined financial sustainability in much greater detail than in this report.
The playcentres in the study were more likely to be less sustainable than other ECE service types in all three aspects: service enrolment, staffing, and financial sustainability. Sustainability issues for each of these dimensions may compound one another, since funding is based on enrolments, and fewer enrolments or variable enrolments as children leave means lesser or uncertain funding, and a smaller pool of parent educators who may find it hard to sustain staffing requirements in playcentres.
- ECE services were responsive to parent aspirations. In both 2004 and 2006, parents expressed a high degree of satisfaction with their ECE service. Ninety-one percent of parents said their overall satisfaction level was “very good” or “good” in 2004, 92 percent in 2006. Parents were most satisfied with staff qualities and least satisfied with staff:child ratios in both years.
The ward-level data suggest it is in localities where there are planned ECE initiatives underway that positive changes to access and participation are occurring. Most of our participation indicators at service level had not been the target of strategic plan initiatives and since this study is of existing services and current participants, some of the likely impact, e.g. changes to starting age and duration, will not be evident for some years. The evaluation shows some issues in relation to participation, such as the low rates of participation of children in some low-income communities, children enrolled in ECE at a younger age, younger children spending long hours (over 30 hours per week) in ECE, and the desire for parents for more hours at times that suit them. These suggest that it may be valuable for services and the MOE to focus on tailoring changes to community contexts, and to examine wider participation issues, not just whether children attend ECE or not. For example, the findings of a younger starting age and longish hours of ECE for some one- and two-year-olds reinforce the importance of ensuring that good quality all-day provision for this age group is of good-quality, as well as ensuring there is sufficient ECE provision for this age group.
Parents’ ability to engage in work and training
Information on paid employment and training was gathered for the first time in 2006. Key findings were:
Sixty-three percent of parents responding had participated in paid employment and 33 percent had participated in training in the last 12 months. ECE services supported workforce and training participation:
- 54 percent of these parents relied on ECE services combined with non-ECE arrangements
- 40 percent relied only on ECE services.
The free ECE implementation could see a shift from non-ECE arrangements, where parents use these in combination with ECE to more using ECE.
Lack of ECE prevented some parents looking for work or taking part in training. For the group who could not make a suitable arrangement, cost (19 percent), insufficient ECE hours (15 percent), and the desired ECE service not having spare places (10 percent) or being available in the locality (8 percent) were main reasons why appropriate ECE was not available.
Lack of ECE prevented 9 percent of parents from participating in training/study. The reasons were in roughly the same rank order as parents wanting paid employment.
Some parents juggled their work and training arrangements around the opening hours of ECE, many relied on friends and family to provide childcare even for very short time periods where service times did not match work times, and to cope with disruptions to work and training when their child was sick or the service was not available. Some parents wanted flexible ECE arrangements, and longer hours for sessional services, especially for them to be open for the same length as a school day.
Overall, these findings on participation and parent employment and training suggest that main challenges for increasing participation are in provision of accessible good-quality ECE services that are responsive to the circumstances and needs of families, especially as more parents are in or interested in employment and training.
Improving ECE quality
The ECE strategic plan intervention logic model identified six intermediate outcomes related to improving ECE quality: “ECE services meet cultural and language aspirations”, “More registered teachers in ECE”, “Quality in parent-led services”, “Reduced ratios and group size”, “Te Whāriki effectively implemented”, and “Quality teaching and learning practices” that were intended to contribute to the overall outcome “Improved quality of ECE”. We developed indicators for each of these outcomes, and a system to rate these as “very good”, “good”, “fair”, and “poor” for each of the 46 ECE services in the study.
Changes in ECE service quality from 2004 to 2006
Key findings were:
- Overall quality. The percentage of services rated as being of “very good”, “good”, “fair”, or “poor” quality remained much the same from 2004 to 2006. In 2006, 26 percent of children were in services rated as “very good” overall quality, and 46 percent were in services rated as “good” overall quality. Some individual services did change their quality rating from 2004. Seven services had higher ratings, 30 services remained the same, and nine services had lower ratings. Services that were rated as having “very good” overall quality in both years and those that improved their overall quality from 2004 to 2006, had “good” or “very good” ratings on each of the intermediate outcomes: teaching and learning practices, implementing Te Whāriki, and teacher qualifications. Services with low levels of quality in both years or those whose quality declined to “fair” or “poor” quality levels had poor ratings for these intermediate outcomes in 2006, suggesting there are connections between the intermediate outcomes and overall ECE service quality.
- Teacher qualifications. Services rated as having “good” levels of staff qualifications doubled between 2004 and 2006 (44 percent in 2006, compared with 22 percent in 2004). “Good” qualification levels were mainly found in teacher-led services where MOE targets and initiatives have been directed. Sixty-one percent of teacher-led services had “good” qualification levels compared with 17 percent of parent/whānau-led services in 2006.
- Teaching and learning practices improved between 2004 and 2006. Thirty-nine percent of services improved their assessment practices. Fifty percent were at a “good” or “very good” level for assessment in 2006, compared with 28 percent in 2004. Those at a “very good” level had integrated systems of planning, evaluation, and self-review based on Te Whāriki. More services were rated at a “good” level on self-review in 2006 (29 percent) compared with 2004 (15 percent). Planning and evaluation (measured in 2006 only) and self-review were not as well developed as assessment.
- Implementation of Te Whāriki. In 2006, 40 percent of services were rated as “good” or “very good” on teachers’ understanding of Te Whāriki and implementing a bicultural curriculum, compared with 22 percent in 2004. A third of services improved their ratings.
- Ratings on ECE services meeting language and cultural aspirations did not change between 2004 and 2006, but services receiving Equity Funding for the language and culture component did better than others.
- Ratings on intermediate quality outcomes that had not been a focus for MOE actions between 2004 and 2006 did not shift. These were ratios and group size.
Use of strategic plan initiatives and shifts in quality ratings
There was a higher use of MOE professional publications, especially the assessment exemplars, take-up of professional development, and participation in COI and Education Leadership Project workshops in ECE services that improved their ratings on teaching and learning practices, and teachers’ understanding of Te Whāriki. Ninety-two percent of services improving their assessment ratings used the MOE resource, Kei Tua o te Pae. Assessment for Learning: Early childhood exemplars, and most had professional development associated with it. Fifty-two percent of services with low ratings or negative shifts in relation to assessment quality had not used this resource or undertaken MOE-funded professional development.
Teacher-led services that increased their levels of staff holding teacher qualifications and registration made use of MOE initiatives to improve qualifications and become registered. The targets for employment of qualified registered teachers seemed to offer an impetus for these services to encourage existing staff to become qualified and registered. There were no links between shifts in qualification levels and quality shifts. However, one would expect gains for quality from staff improving qualifications to occur over time and to be consolidated with experience.
The use of Equity Funding helped services that were receiving it to improve overall levels of quality. These were services located in low-income communities, services delivering the education programme in a language and culture other than English, and isolated services.
Indicators of intermediate outcomes where policy change did not occur between 2004 and 2006, i.e. reduced ratios and group size, and quality in parent and whānau-led services,6 showed no consistent shifts from 2004 to 2006.
Differences in strategic plan effectiveness
There were some differences in levels and patterns of change on quality dimensions that were associated with locality, service, and child characteristics:
- Locality differences. Minor urban localities, in particular one locality, Kauri, had lower levels on several of the intermediate outcomes, and on levels of overall quality. Differences in levels and shifts seemed to be attributable more to the opportunities within the locality for professional support and staffing rather than the nature of the population of the locality. Features associated with poorer ratings were: isolation from professional support and training opportunities; a limited pool of people to staff ECE services; and limited access to professional support. Localities with high Māori populations did better overall on implementing a bicultural curriculum, and meeting cultural and language aspirations of parents. Some of the services in these localities reported having professional development about biculturalism, and employing Māori staff. The two localities with high Pasifika populations did better overall on levels of qualifications and understanding Te Whāriki. These localities were also main urban localities. The services in these localities had no problems in accessing training provision, and made use of MOE initiatives to support training and registration.
- Service differences. Teacher-led services had higher average quality levels and more made improvements between 2004 and 2006 than parent/whānau-led services on ratings of teaching and learning practices and qualifications. Specific initiatives to raise qualification levels have been targeted at teacher-led services. They also made greater use of assessment resources and MOE-funded professional development than parent/whānau-led services. Playgroup and puna had low levels on all the intermediate outcomes (except the study puna was rated high on meeting language and cultural aspirations), and were not eligible to access MOE initiatives.
- Child characteristics. Services with more than 20 percent under-two-year-olds were rated lower on each of the intermediate outcomes and overall quality ratings. Education and care centres that were rated low all had more than 20 percent under-two-year-olds and had poor teacher qualification levels, and poorer observed adult:child ratios (1:5 and 1:4) than education and care centres that had better quality ratings overall (these averaged 1:3 or lower). These lower quality education and care centres also had poorer ratings of relationships with parents, derived from parent responses to questions about the usefulness of teacher information about the child’s learning, the child’s happiness, the curriculum, whether the parent talked to teachers about home, their satisfaction with information, and the parent’s involvement in assessment and planning. Services receiving Equity Funding had higher ratings on implementing a bicultural curriculum and overall quality ratings, reinforcing the value of Equity Funding for children in isolated services, from low-income families, and in immersion services.
Stronger collaborative relationships with ECE services are a strategic plan goal, with two intermediate outcomes: coherence of education birth to eight, and integrated services for children, parents, and whānau birth to 8. We developed indicators for each of these outcomes, and a system to rate these as “very good”, “good”, “fair”, and “poor” for each of the 46 ECE services in the study.
More integrated services for children, parents, and families
- Ratings of service–-parent relationships improved between 2004 and 2006, with the number of those reaching a “good” or “very good” level almost doubling over the two years, largely due to improvements in parent participation in assessment and planning. Two-thirds of the services had “adequate” or better ratings for their level of service–parent relationship in 2006. The focus within MOE professional resources on engagement with parents (the early childhood exemplars, self-review guidelines, examples from COI of “community of learners” approaches) contributed to strengthening these ratings. These relationships were strongest in some of the parent/whānau-led services which emphasise the roles and responsibilities of all parents and whānau for the ECE service.
- Overall, relationships with health services were “good” or “very good”. Relationships with welfare services were mixed, ranging from “very good” to “poor”. There has been no change since 2004. This intermediate outcome goal is not yet supported with any particular strategic plan actions, and appears highly dependent on individuals, health and welfare organisations’ policies, and available time.
- In 2006, 36 percent of the ECE services had no or limited contact with other ECE services in their locality, the same percentage as in the 2003/2004 NZCER national survey of ECE services (Mitchell & Brooking, 2007). However, some closer relationships with other ECE services were beginning to be made through shared professional development and teacher education offered through strategic plan initiatives, and MOE hui. One of the values of meeting with other teacher/educators in such professional forums is likely to be that the focus on teaching and learning enables service participants to learn from each other within their own locality.
Cohesion of education 0–8 years
- Professional relationships with local schools improved from 2004 to 2006. Twenty-two percent of ECE services had no or limited contact with all the schools/kura in their locality, compared with 33 percent in 2004. Schools and early childhood services were more likely to work together in respect to transition of children than in 2004, but it is unclear why, in relation to strategic plan initiatives. Perhaps the higher level of professionalism and focus on the importance of ECE has contributed to greater awareness of the importance of transition for both ECE and primary teachers.
Barriers to collaborative relationships
The most common barrier to developing close working relationships with external organisations and parents was insufficient time. The most common suggestion for integrating services was to offer family services from the ECE service site.
Overall, the strategic plan has been effective in starting to raise levels on intermediate outcomes linked with children’s learning foundations, such as assessment for learning, self-review, teacher/educator understanding of Te Whāriki (the early childhood curriculum), and teacher qualifications. The value of professional resources and professional development, and of improving teacher qualifications in helping raise levels on intermediate outcomes, was strongly reinforced. There is some way to go to get to “very good” levels of quality throughout and our findings reinforce the importance of MOE continuing the strategic plan initiatives aimed at quality improvements. There are also inequalities in access to ECE services, suggesting that new initiatives around participation and planning could be useful.
The take-up of strategic plan initiatives had some limitations, and some issues particular to some kinds of localities, services, or children were not being addressed. This suggests some targeting of MOE initiatives may be useful.
The main message from this evaluation of the early days of the strategic plan is that the initiatives implemented so far are being used as intended to improve teaching and learning.
- The MOE evaluation strategy for the ECE strategic plan also includes development of a monitoring system designed to provide indicators of progress and identify emerging problems, and targeted evaluations that both contribute to the overall evaluative picture and inform decisions about individual initiatives.
- Equity Funding is a small amount of funding that is additional to bulk funding and discretionary grants, and is intended to reduce educational disparities. It has four components: low socioeconomic, special needs and non-English speaking background: language and culture other than English; and isolation. It is intended to increase participation by providing additional resources, and improve quality through addressing the higher cost for achieving the same educational outcome.
- Bridges, Fuller, Rumberger, & Tran, 2004; Gormley, Gayer, Phillips, & Dawson, 2005; Magnuson, Meyers, Ruhm, & Waldfogel, 2004; Sylva, Melhuish, Sammons, Siraj-Blatchford, & Taggert, 2004.
- Broberg, Wessels, Lamb, & Hwang, 1997; Sammons et al., 2002; Wylie, Thompson, & Lythe, 2001.
- Sylva et al., 2004; Wylie & Hodgen, 2007.
- Note: Parent and whānau-led services other than playgroups and puna had access to professional resources like teacher-led services, and those that were eligible received Equity Funding. Findings related to these initiatives apply to these services too.