An Evaluation of Ministry of Education Funded Early Childhood Education Professional Development Programmes 2003

Executive Summary

Introduction

The Ministry of Education commissioned this evaluation to examine three aspects of professional development (PD) in the early childhood education (ECE) sector: describing the current delivery of PD; identifying barriers and success factors for delivery of the PD programmes; and identifying their impact. The evaluation covered the provision of whole centre PD in the calendar year 2000.

Three factors have been identified by two earlier studies (Allan and Gibson, 1994; Gaffney and Smith, 1997) as central to any evaluation of PD in the early childhood sector. They are: the process of facilitation, including the development of relationships; the structure of the PD programmes; and those features of the PD programmes or the sector itself that impact on the ability to maximise the gains from undertaking PD.

Methodology

Data for this evaluation was collected from three sources:

• milestone reports sent to the Ministry of Education by PD providers
• interviews of the 13 providers that had participated in PD in 2000 (carried out May-June 2001)
• a postal survey, using a questionnaire, of 1130 ECE centres and services that had participated in PD in 2000 (carried out June-July 2001). These made up 38.7 percent of the centres and services in the May 2001 Ministry of Education database of all those licensed. A total of 515 responses were received, giving a response rate of close to 46 percent: 77 from kindergartens, 91 from playcentres, 307 from education and care centres, 26 from home-based services, and 14 from casual education and care centres (including those based in hospitals).

The report was drafted in September 2001 – January 2002.

Professional Development Programmes

The providers outlined the PD programmes they delivered to centres and services in 2000.

Programme strands: Programmes varied according to the content areas which providers were contracted to cover. While all focussed on Te Whärikiandthe revised DOPs, seven of the 13 providers also had a minor focus on quality assurance systems (The Quality Journey - He Haerenga Whai Hua) and special education (Including Everyone – Te Reo Tätaki). Another five had a major focus across all four strands of the programme. One provider omitted quality assurance. Ten of the providers also offered “general professional or advisory support”. All providers said there had been much less interest from centres and services in The Quality Journey - He Haerenga Whai Hua and Including Everyone – Te Reo Tätaki resources than might have been anticipated.

Outcomes: The Ministry adopted a broad set of 11 outcomes to map out the areas where providers were working. As part of their agreement with the Ministry, each PD provider agreed to work toward these. They were of three types: progress outcomes, result outcomes, and mixed outcomes.  The PD providers were asked to report back on the effectiveness of the programmes at the end of years 1 and 2. In addition, all providers had been working on their own evaluation tools as part of their previous PD programme development.

Principles: Provider contracts generally followed current understandings about: developing skills and capacity; acknowledging processes and participant contexts; the importance of relationships; adult learning and development; and adult teaching, facilitation and learning environments. There was general recognition of the importance of responding to diversity in the sector and in the issues that arose.

Formats: Each PD programme was made up of some combination of six formats: individual whole centre development, clustered whole centre development, networking, one-off workshops and seminars, general professional or advisory support, and individual development.

Coverage: Twelve providers gave geographical coverage, and two were service specific. Three providers worked nationally or close to it, while eight operated in very specific regions. 

Professional Development Provider Perspectives

Overall, the factors that providers saw as having the most impact on PD outcomes were to do with facilitation, relationships, educator attitudes, educator commitment and energy, and the way the programme was set up for learners. The importance of some factors, such as the timing of the PD or difficulties in accessing relievers, varied according to sector or geographical area.

Factors supporting change

The factors which providers consistently saw as most important to success were:

• effective facilitation
• good provider relationships with participants
• providing a safe or seccure learning environment.

These factors had an influence in almost all centres and services being worked with.

Factors impeding change

The most prominent barriers impeding change were:

• a low level of commitment to PD in the centre or service
• a low level of educator stability.

According to providers, these factors had a negative influence on about half (40-59 percent) of the centres and services which providers worked with. That influence included both preventing PD goals from being attained, and slowing progress.

Programmes and their outcomes

Providers highlighted two aspects of effective PD programmes: integration, and supporting change. They also discussed monitoring the outcomes of PD.

Integration involved establishing how PD would operate within or alongside a centre or service. A key theme was “flexibility”. Four general approaches to flexibility were: creating a pathway (referred to by 11 providers); needs analysis (9 providers); individualising a programme (5); and working within a suitable time frame (5).

Supporting change through PD involved three important aspects: approach (referred to by 8 providers); content (8); and facilitation (7). These aspects included making expectations and requirements explicit, articulating realistic goals, and having a facilitator who was able to reflect back progress. Another challenge was to find the right match between facilitator and service or centre. Facilitator enthusiasm for the work was seen as very important for motivating educators and management.

Monitoring outcomes involved three aspects: monitoring progress, impact on programme design, and whether PD goal identification is top down as opposed to a bottom up process. All the providers thought that the Ministry of Education’s 11 programme outcomes covered all the potential areas of focus. The major issue was how to match up indicators with the work done, and report on it in a way that related back to the programme outcomes. There was an issue for some of not letting the outcomes drive the PD in a top down process.

Relationships

With centres and services, providers saw the following relationship factors as impacting on PD:

• leadership (or its absence) within centres and services (referred to by 10 providers)
• relationships within the centre or service (10)
• facilitator input (9 providers)
• establishment and maintenance of the relationship (9)
• centre/service reasons for participating in PD (9)
• centre/service commitment to participating in PD (8)
• selection into PD – when demand exceeded availability, all providers had selection criteria in place (6)
• a secure environment for challenge (5)
• management role and reasons for participation (5).

With the Ministry of Education, the range of relationship issues raised by providers included:

• contracting format - the change to a two-year format was welcomed (6 providers referred to this)
• Ministry communications to sector (6)
• milestone reporting and responses, in terms of consistency and feedback (5)
• agency co-ordination (4)
• participatory funding, where providers were keen to eliminate inconsistencies (3)
• professional development for providers (3)
• resource development, especially providers wanting to have some input (3).

ECE sector factors impacting on PD

Educators: All 13 providers thought educators’ own ECE education and training (or in playcentre, parents’ general levels of education) predicted the base level of understanding of PD that might be assumed within a centre. Other important educator factors included turnover (10), understanding the purpose of PD (10), time and energy (10), and attitudes (9). Tired educators, or those who were not paid to meet for PD, were seen as likely to be less motivated than those who had non-contact time through the working day, or were acknowledged for participating in PD.

Organisation and management: Management support (10) and demographics and resources of the centre/service (10) were seen as the most important factors in this area. Although the revised DOPs placed new emphasis on management systems, there was no requirement that educators meet regularly or plan as a group. Where there were no management or curriculum systems in place, the level of support required could be well beyond what the PD formats could offer.

The broader ECE context: The role of ERO (7) and service specific factors (7) were also seen as important. Centres forced into PD as a compliance requirement, following an ERO report, could show resistance to PD. There was not sufficient support for centres with voluntary management committees, especially those who were going through probationary licensing. In the metropolitan areas, providers could not meet the demand for PD. In the remote rural areas, offering a facilitator incurred higher costs. Limited resourcing could be a problem in urban areas where communities had few economic resources.

ECE Centre and Service Perspectives

Centre/service characteristics

Educator training and involvement in PD

Most kindergarten teachers had a diploma, and most had been involved in PD in 2000.

The qualifications most commonly held by playcentre participants were Part 1 playcentre qualifications.

Two thirds of education and care centre staff were involved in PD. Just over half the full-time staff had a diploma or above, and just under half the part-time staff were untrained compared with a fifth of full-time staff.

Nearly all home-based service co-ordinators were involved in the PD, whereas only about one in 5 of the educators were.

Most staff in casual education and care centres were involved in PD in 2000. Over half the full-time staff and the part-time staff had diplomas or above.

Centre and service involvement in PD

Outcome areas and goals: Across all sector types, those areas receiving the most attention in 2000 were management systems and the DOPs, curriculum management, quality environments, and Te Whäriki. The Treaty of Waitangi and Te Reo and Tikanga Mäori were rated highly in kindergarten and in home-based care, but lower in education and care services and in playcentre. The areas of parent partnerships, literacy and numeracy, special needs, and transitions were consistently rated the lowest across all sector types. The highest rating defined goals, across the whole sector, were programme and observation based planning, systems to support DOPs, and assessment. Work on systems to support DOPs was separate from the policy development that might be occurring alongside that work.

Overview of factors impacting on PD

The 515 responses from centres and services held generally consistent views about the major factors impacting on the success of PD. However, there were variations for some sector types.

Provider and facilitator factors supporting change: There was a great deal of consistency among all sector types about the five most important factors. These were having providers and facilitators who:

• looked at centre/service needs
• kept the centre/service focused
• supported centre/service goals
• brought an outside perspective
• provided resources.

These factors were seen as more important than the next set of factors including facilitators being accessible, being flexible, providing intellectual challenge, or articulating practice.

Programme factors impeding change: There was a great deal more variation between sector groups here. For example, kindergartens had far less concern about any of these aspects than the other sector groups did. Evening meetings were an issue for education and care centres, playcentres, and casual centres. Problems with the facilitator relationship were more often identified by home-based services and casual centres than by the other three sector types.

Centre or service factors supporting change: Consistently rated as important were:

• enthusiastic staff (or parents, for playcentre)
• a good relationship with the facilitator
• having systems in place
• supportive leadership.

Having extra meetings paid for was important for 45 percent of education and care centres, but only for less than 10 percent of centres and services of other sector types.

Centre or service factors impeding change: Again, this showed much more variability among sector types. However, respondents in all sector types identified the following barriers:

• lack of time to engage with PD (reported at least twice as often as any other barrier)
• impact of staff turnover (or parent turnover in playcentre)
• limited community resources. 

The importance of other barriers varied by sector type. Low commitment to PD was an issue for many playcentres and home-based services, but far less so for education and care centres and casual centres, and almost negligible for kindergartens. Again, kindergarten appeared to have fewer barriers to change than other sector types.

Outcomes

Centres and services were asked whether they agreed or disagreed that they had improved their ability to implement Te Whäriki and the DOPs, and had met their own PD goals.

Overall, about two thirds of centres and services from most of the sector types reported an improved ability to implement Te Whäriki or the DOPs. However, only half of the home-based services reported improved implementation. Between 5-10 percent of respondents said they did not improve. The remainder were had mixed improvements.

Two thirds of centres and services across the sector types reported meeting their own PD goals, but only half of the home-based services did so. (However, only 26 returns came from this sector type.) While the casual centres agreed they were meeting their goals and improving the implementation of DOPs at the same level as other sector groups, improved implementation of Te Whäriki had a lower level of agreement. (However, only 14 returns came from this sector type.) This suggests that casual centres were working in specific areas not so directly related to curriculum.

Discussion

This chapter is based mainly on a discussion of issues raised by providers, and in responses from centres and services.

Policy implementation: Extensive national consultation seemed likely to have supported the uptake of Te Whäriki and the DOPs. Comments by providers indicated that they would be better placed to support the use of other resources if they had been consulted during their development. However, this does not mean that the low interest in quality assurance and special education is because of the resources. The data for PD in kindergarten, playcentre and education and care centres over the period 1999-2001 showed a definite movement over time to other areas such as quality environments, adult/ child interactions, and curriculum management. To a lesser extent, there was also an increasing interest in areas such as parent partnership, numeracy and literacy, and transitions. This suggests that the longer centres stay involved in PD the more diverse the areas of interest become.

Centre/service development: PD could make participants aware that implementing Te Whäriki and the DOPs had implications for a wider set of practices, which further PD could elaborate, explore and evolve. This appeared to take much longer than a year, and the pace of change was not necessarily continuous. The work of PD might continue in the form of general professional support. Respondents referred to “shopping around” providers to see what was on offer.

Teacher/educator development: The number of pathways available to people to get their diploma qualification created a complicated context around professional development, of which the current in-service PD formed only part. How individual educators learn about and participate in the full range of PD available may prove a worthwhile avenue for study.

PD formats: Both positive and negative comments were made by sector respondents about clustering centres for whole centre development. It is not clear why some described this as working very well, yet others said it did not work. This would take further research to establish.

An average of about 15 hours’ PD was reported across all three evaluations. The two-year contract seemed to have enhanced flexibility in the length of time taken to get through a programme. However, the programme being too short was the most common format factor mentioned by centres and services. Successful centre development seemed to require a longer-term view of what was happening for a centre.

All three studies found that the main centre or service barrier to change was the lack of time, or too many other things happening within a centre or service. This was the only barrier to be mentioned by more than 40 percent of respondents across the sector. Most other barriers were mentioned by fewer than 20 percent.

Monitoring programme effectiveness: Providers had reasons to monitor their own programmes, and did so. Judgements were made not of individual staff, but on the basis of how well a centre was doing overall in meeting certain standards or indicators. Monitoring by facilitators and providers appeared to be taking account of much more complexity than indicator-based reports to the Ministry might indicate. Developing general indicators of change was seen as undervaluing and oversimplifying the work that went into bringing change about.

Needs analysis: Survey respondents rated very highly the importance of a facilitator’s ability to “look at a centre/service’s needs” and “support a centre/service’s goals”, but they did not see this as being “needs analysis”. Further work needs to be undertaken to understand what is happening during this process.

Differences among sectors: Home-based care services and casual education and care services tended to report centre or service barriers more often. These included staff turnover, problems in finding relievers, limited community resources, and lack of management support. While this tendency to report barriers more often could be a feature of the small sample sizes from these two sector types, it is an issue that is worth exploring further. Home-based care services also reported more problems than other sector types in relation to what facilitators and providers brought to the PD experience.

Sustaining change: The two most common approaches to sustaining change were to put systems in place, or to rely on the energy of teachers and educators to maintain the changes as part of their daily routines. While these two approaches are not totally separate, putting systems in place might be more robust, in a context of high staff turnover and change.

Impact on educators: The numerous levels of complexity operating in early childhood services and centres made successful facilitation for individual learning a complex undertaking. Sustainable change appeared to involve the facilitator being able to provide the stimulation and create the opportunities for participants to learn by challenging their own practice.

Further research: Some other useful areas for further research indicated by this evaluation are:

• the role of successful facilitation, and how it supports change in policy implementation, centre/service development, and teacher/educator development
• whether and, if so, why home-based care services and casual education and care centres have more difficulty in engaging effectively with the current PD programmes.
• how all forms of PD, not just Ministry funded programmes, provide an environment to support the professional development of teachers and educators.