Evaluation of Ministry of Education-Funded Early Childhood Education Professional Development Programmes Publications
The Ministry funded ECE Professional Development (PD) Programmes are for a three year period from July 2010 to June 2013. The focus for the PD Programmes is on targeting locations with low rates of ECE participation to support quality provision in those communities and services that are likely to enrol children who have the greatest potential benefit to make from participating in quality ECE (Māori, Pasifika, and children from low socio-economic areas). This report represents the findings from the evaluation of the PD Programmes and was undertaken during 2012.
Author(s): Sue Cherrington, Mary Jane Shuker, Alison Stephenson, Ali Glasgow, Lesley Rameka and Kate Thorndon with Sarah Sears, Nicola Goodman, Alison Barker and Jonine Nager, Victoria University of Wellington.
Date Published: May 2013
This report is available as a download (please refer to the 'Downloads' inset box). Please consider the environment before printing the contents of this report.
The current round of Ministry of Education (MoE)-funded early childhood education professional development (PD) contracts run from mid-2010 until mid-2013. This project was designed to evaluate those PD programmes to assess whether their design and implementation were meeting the intended outcomes,
and to assess the effectiveness of a targeted approach to professional development provision.
Each of the ECE professional development programmes evaluated share commonalities, such as a core focus on valuing identity, language and culture, and an expectation that ongoing and sustainable communities of practice will be developed as a result of engagement in the programme. Participants are therefore not only attempting to transform their thinking and pedagogy in order to enhance learning outcomes for children, particularly Māori and Pasifika children, but are also being supported to work in partnership with their colleagues, with families, and with other centres/services to build new ways of working together. Such shifts in teacher thinking and practice frequently require shifts in attitudes as well as the development of new understandings and skills. Given the scope and range of these PD programmes, this evaluation focuses on a complex array of factors.
The evaluation included nine of the ten PD providers contracted to deliver these programmes in the 2010-2013 contract period. The Te Kōhanga Reo National Trust and kōhanga reo engaged in their programme, were not part of this evaluation.
In order to address the 16 evaluation questions posed by the Ministry for this project, five overarching questions were identified, which clustered the Ministry's 16 questions into logical groups. (Each of the 16 questions is addressed independently in the evaluation report.)
- How effective has the PD been in improving teacher practice to improve learning outcomes for children?
- How effective has the PD been in enhancing teacher practice in relation to the key focus areas?
- How successful has the PD been in supporting teachers to focus more strongly on the impact their practice has on learning outcomes for children?
- How effective has the PD been in supporting services to develop and sustain their own processes for on-going reflection and improvement?
- How have services strengthened their own processes for on-going reflection and improvement as a result of the PD?
- How effective has the PD been in increasing service responsiveness to children's identity, language and culture?
- How have the PD Providers worked with services to increase services' responsiveness to identity, language and culture?
- How effective have the PD Providers been in building and maintaining working relationships with ECE services and other relevant stakeholders to address the PD needs in the targeted community?
- What approaches have PD providers used to successfully involve and engage services in the PD programme?
- How successful have PD Providers been in developing communities of practice in the targeted locations they are working in?
- How are communities of practice being defined?/What are the natures of these communities of practice?
- How are providers working with communities of practice to ensure these are sustainable once the PD is completed?
- How have the rubrics contributed to changes in teacher practice to improve learning outcomes for children?
- How have the PD Providers been using the rubrics?
- In what ways have services been using the rubrics?
- How have the rubrics assisted providers and services to assess the impact of the professional development on services' practices?
Evaluation analytical framework
A multi-dimensional evaluation framework (Guskey, 2000, 2002a) was adopted in order to evaluate both the implementation of the programmes and whether the outcomes were being met. The framework included four levels of investigation:
- First Level: Participant learning, including both attitudinal changes and specific skills and understanding.
- Second Level: Organisational support for change including both wider ecological contexts of the services, and structural aspects embedded in the design of the current PD programmes
- Third Level: Participants' use of their new knowledge and skills.
- Fourth Level: Children's learning outcomes.
A mixed methods approach was adopted for flexibility in addressing the multiple and diverse research questions. Data were gathered in four ways, using multiple qualitative and quantitative methods, which allowed for triangulation of data, and enabled complex issues to be examined from a variety of perspectives. The four data sources used were:
Document analysis of all the PD providers' milestone reports gave an overview of the professional development programmes, and contributed to the development of the provider interview schedule, online survey, and case study protocol.
Interviews with Contract Providers
Semi-structured interviews with the directors of each of the participating PD contracts (and, in some instances, PD facilitators) were conducted. Providers elected whether to be interviewed face-to-face or via Skype and whether to involve some or all of their facilitators. Seven of the nine interviews were conducted face-to-face and two by Skype/teleconference.
Online Survey of Participating ECE Services
An online survey was undertaken of services participating in the three PD programmes. The link to the survey was distributed to 823 ECE services; 260 services (31.6%) proceeded past the opening screen (which provided information about the survey and informed consent) to complete questions. Responding
services were predominately education and care (68%) (including seven Pasifika and three Māori Immersion or bilingual services). Nineteen per cent were playcentre, 11% kindergarten and 2% home-based services. Most services were located in the Auckland (38%), Bay of Plenty (18%) and Waikato (10%) regions.
Twelve per cent of responding services were from the South Island.
Services were asked which MoE-funded professional development programmes they had participated in during the previous 18 months – Programme 1: in-depth or clustered PD; Programme 2: national Leadership (with either of the two providers delivering this programme); or Programme 3: national Education and care for children under two (with either provider) – and could identify more than one programme. Just over three-quarters of responding services (76.2%) had been involved in a single programme: 39.7% in programme 1, 30.1% in the national leadership programme and 6.4% in the national education and care of children under two programme. Fifty-two services had participated in combinations of two or three of the programmes.
Services were asked to respond to questions in the survey using their experiences with the programme they had undertaken most of their professional development in. The high proportion (23.7%) of services that had engaged in more than one PD programme was unexpected so, given that it was impossible to identify which programme this group of respondents based their responses on, analyses of data by programme type used the following groupings:
- Programme one respondents (N=87)
- Programme two and/or three respondents (N=87)
- Programme one plus programme two and/or three respondents (N=45).
Services responding to the survey had engaged in PD across the focus areas: leadership (40%), literacy (36%), social competence (35%), transitions (32%), numeracy (23%); education and care of children aged under-two years (17%).
Case Studies of Participating Services
Sixteen individual case studies of participating ECE centres were undertaken; collectively these covered the range of programmes, providers, service types, geographic regions, and focus areas.
Key findings of the evaluation project
Enhancing children's learning has been referred to as "the ultimate measure of successful professional development initiatives" (Sheridan et al., 2009, p.379). The evidence is that a focus on learning outcomes for children is an emerging focus for teachers; many are still engaged in the process of establishing a culture of reflective practice, and in developing their skills and confidence in assessing and documenting children's learning. In comparison to the many references to those two aspects of teacher practices, references to a focus on learning outcomes for children were relatively sparse. There are, however, many descriptions of learning/teaching episodes which indicate children are learning in a wide variety of ways, both within the nominated focus area(s) and beyond.
Ninety-six per cent of services responding to the survey were 'Very satisfied' or 'Satisfied' with the PD programme in making a difference to children's learning. Both PD providers and services gave qualitative examples that focused on changes in practices, and several PD providers commented on the challenges they are working with in shifting some teachers to a focus on children's learning outcomes]. These are indications that in some services this is still 'work in progress'. Whilst we would expect to see further progress evidence in provider milestone reports over the remainder of the contract period, continued attention to ensuring that teachers' and services' practices are actually improving learning outcomes for children is needed.
Quantitative data from providers indicates that between 57% and 93% of services (across individual rubrics) have shown a rise in their rubric rating as a result of engagement in their PD. Data from services shows that 93% of services believed that their PD programme was 'Very effective' or 'Somewhat effective' in making a difference to their teaching in the focus area(s) and 95.5% were 'Very satisfied' or 'Satisfied' with this aspect of their programme. As a result of their engagement 55% of services assessed their practices in the focus area as now being 'Highly effective'. Both providers and services provide a wide range of qualitative examples that support these figures.
The collective data show successes in supporting teachers to focus more strongly on the impact their practice has on children's learning, but also underline the complexity and challenge for providers in supporting teachers to make that shift. A first step for PD teams and facilitators was that they themselves needed to keep learning outcomes for children to the fore – and there is evidence of this occurring during the phase one of the contracts.
For some teachers, the called-for changes challenged entrenched attitudes and practices. In many services it was necessary to introduce the practice of reflection, and to support teachers in understanding the importance of this within their teaching role. Facilitators frequently found they needed to revisit processes of assessment and documentation of children's learning with teachers; a wide range of qualitative data describes significant shifts in the focus, detail and depth of learning stories. Finally teachers have had to engage with the notion and processes of evaluation.
The data indicate that while significant shifts have been made, in some services these are emerging skills. Overall, however, the indications of change are very positive. The data provide evidence that some teachers are more open to questioning themselves and each other about the purposes behind what they were doing and reflecting on what their learning outcomes for children were. Several services indicated their increased ability to reflect regularly and deeply on their practices, especially in their learning stories, made them more accountable for supporting children's development.
A range of factors which may support or impede providers' success in engaging teachers to focus on the impact their practices have on learning outcomes for children were identified. Among these issues of time, leadership, qualification levels of staff, team involvement, regular meeting times, and a culture of reflective practice appear to be key.
Structural aspects relating to providers that are seen to support or impede teachers' focus on the impact of their practices relate to: sustained involvement with the service, a commitment to building relationships with the service, the qualities, skills and background of the PD facilitator, and the potential flexibility of the PD programme and delivery.
The broader structure and design of the PD contract – for example, the milestones, the focus on self-review, the rubric indicators – are positive factors in keeping children's learning outcomes to the fore. In some contexts, it appeared that a PD delivery model that focused on workshops with little or no facilitator-service contact in between had proved less effective.
In many services facilitators found that introducing teachers to the practice of reflection was an essential first step. There is considerable evidence that many services responded positively to the opportunities for reflection that undertaking the PD had brought. A change in teacher thoughtfulness was often noted in relation to self-review and/or inquiry processes. The role of the PD providers was particularly important as they supported centres through this process. It is clear in some programmes that facilitators took an intensive role in coaching teachers in the processes of inquiry and self-review.
The majority of services (91.8%) indicated in the online survey that that they had found their PD programmes 'Very effective' or 'Somewhat effective' in making a difference to their own ability to critically reflect on their practices and there is also a shift in the percentage of respondents rating themselves 'Highly effective' in their ability to reflect on their practices – from 10% to 59% – as a result of the PD. Utilising video for teacher reflection was found to be an effective tool used by facilitators for teacher reflection and change. The facilitator's role was significant, too, in the process of rating services against the rubrics criteria; providing evidence of those ratings 'in action' resulted in some deep reflection by teachers. Other successful strategies to assist services' on-going reflection include the development of resources to frame the self-review processes so that it could be replicated in the future, and connecting services with provider websites and other relevant early childhood web environments.
The structure and approach of the PD programme appeared to influence the extent to which services were supported in reflecting on their practices. In particular, when PD programmes were strongly focused around the self-review process services were more likely to report adopting reflective practices. The importance of a leader in the centre who was committed to an inquiry process and to leading the team contributed to improved teacher practices. There was also recognition that engaging the whole team in the processes, which had traditionally been the responsibility of management for some services, was important so that all were involved in making effective changes to achieve positive long-term outcomes.
Factors that contributed to services being slower in strengthening their reflective practices included teachers being unfamiliar with the concept of reflective practice, lack of leadership within a service, and teachers adopting 'best practices' from other services but not developing their own theoretical understandings to underpin these. Working with unqualified staff was also an issue as it took longer for teams to use self-review processes.
There was some indication of services strengthening their processes for reflection and improvement, particularly where attention to self-review or inquiry has been a central focus. Comments from the PD providers convey the sense of ownership some services were taking in their PD journey. A similar sense of self-awareness and reflectiveness was indicated by the examples of services recognising they initially overrated themselves on the rubrics and other services who requested an extension of their professional development.
Working in cluster groups proved to be important for many services for their on-going reflection. For some centres grouped together leadership evolved and this assisted them to probe their practices more deeply. Some cluster groups planned to carry on meeting when their PD finished as they wanted to continue with self-review to share ideas and reflect on the information gathered.
Where providers have addressed responsiveness to children's identity, culture and language, participants' learning has often been attitudinal as well as in terms of increased knowledge: services have focused on building relationships and partnerships with families and in strengthening teachers' ability to be culturally responsive in their interactions with children and families.
Extensive examples were given by providers and respondents to the online survey of how services had increased their responsiveness to children's identity, language and culture – these ranged from modest (at times, rather tokenistic) changes to significant shifts in practice. More than three-quarters (78.5%) of services felt that their PD programme had supported them 'A great deal' or 'Quite a lot' to be more responsive – a key feature of shifts in practice is an increase in teachers' willingness to proactively engage with families rather than relying on families to approach them. Several key themes were evident: building stronger relationships with families and children from diverse backgrounds; actively asking parents for information about their family's cultural practices and finding ways to involve families authentically in the service; incorporating their increased understandings into their assessment, documentation and planning processes; using multiple lenses to reflect on their practices; changing environments and resources to reflect cultural and linguistic diversity within the service; increasing their use of te reo me ngā tikanga Māori in the programme; reflecting multicultural diversity within the programme; encouraging children and families to use home languages in the service; and supporting children and families for whom English is an additional language.
Three aspects of service capability appeared to influence whether services became more responsive to children's identity, language and culture: firstly, the impact of leaders within the service who, when committed to developing practice in this area, drove change through in the service; secondly, the need for specific knowledge about New Zealand's bi-cultural heritage and the role of the Treaty of Waitangi, and thirdly, gaps in teachers' existing assessment knowledge to effectively document children's learning and their own practices.
Providers reported using a wide range of approaches and strategies to support services in being more responsive to children's identity, language and culture. Successful approaches included preparing facilitators carefully to work with services on these aspects, using the rubrics as a tool to open up conversations about how the service is responsive to Māori and Pasifika children, incorporating a focus on identity, language and culture within the self-review processes, using a range of resources such as Taitiako, the Pasifika Plan, Te Whatu Pokeka and Ka Hikitia as well as Education Review Office national reports, and supporting services to build links with and knowledge of mana whenua.
However, data from the online survey and case study components of the evaluation suggest variation in the extent to which providers actively addressed how services could increase their responsiveness to children's identity, language and culture: collectively just over three-quarters of responding services (77.8%) felt that their programme had supported them 'A great deal' or 'Quite a lot' to be more. Case study data revealed that some programmes appeared to have paid little attention to these aspects, some programmes had helped services to strengthen existing practices to a moderate extent, and some programmes had combined with services' commitment and existing practices, resulting in some powerful outcomes. Where providers did actively work with services to increase their responsiveness, the degree of shift in participants' learning was influenced by their existing knowledge and openness to new learning in these areas.
Learning for participants from Immersion Māori and Pasifika was influenced by the ability of their provider to deliver the programme in culturally appropriate ways. The structure of the current PD contracts does not differentiate between the PD needs of Māori and Pasifika Immersion services and those services who may have high Māori and Pasifika participation but which do not operate from a kaupapa Māori or Pasifika philosophy. There was evidence from the case studies and online survey that Māori and Pasifika Immersion services do not always feel that their specialised PD needs are met, despite the best efforts of providers and, thus, were not always able to engage with the PD work in a way that encouraged development of professional knowledge. Different skills and knowledge are required for PD that supports non-immersion services to develop in areas such as te reo me ngā tikanga and cultural responsiveness versus the specialised expertise required for working with kaupapa Māori and immersion services.
Data from this evaluation has highlighted the need for more targeted PD provision for kaupapa Māori and Pasifika immersion services built on appropriate protocols and processes that set the scene for professional growth. Given that there are such Māori and Pasifika services located across the country, the provision of PD programmes for these services is needed across the country, not just in specific geographical regions. Thus, we suggest the Ministry consider structuring future PD programmes differently, providing two parallel strands of PD:
- One for kaupapa Māori services focusing on areas of importance for Māori (e.g., Te Whatu Pokeka) or for Pasifika services focusing on areas of importance for Pasifika communities
- One for non-Immersion services focusing on te reo, tikanga, and Mana Whenua.
A complex set of factors influence, positively and negatively, providers' effectiveness in building and maintaining relationships with services and other stakeholders. Factors contributing to success included the targeted approach to PD which has enabled less proactive services to have the opportunity to engage in a programme. Given that findings from the evaluation of ECE PD programmes undertaken in 2005–06 (Cherrington & Wansbrough, 2007) showed many services' slow internal organisation processes resulted in them missing out on PD, this is a positive outcome of the targeted approach. Professional leaders within services were instrumental in services' initial engagement; progress within the programme was greater when there was internal leadership (either from the designated leader or through distributed leadership) that maintained momentum.
Services with particular characters or philosophies value highly undertaking PD with providers who have in-depth and first-hand knowledge of the service types they are working with. In addition to Immersion Māori and Pasifika services discussed above, playcentre and home-based services clearly prefer working with providers and facilitators who have this expertise and can tailor programmes to fit their philosophy.
A significant feature of the current PD contracts has been the closer working relationship developed between providers and local MoE offices that have enabled targeted services to be identified and, almost always, a unified approach to be taken with services causing concern to either the Ministry or providers. Providers have been effective in engaging with umbrella organisations that generally report feeling informed and listened to.
Challenges in building relationships and engaging services included providers working in new areas where they had little existing knowledge of the ECE and wider community and had to build new relationships, travel demands for providers working over large geographic regions, activities such as re-licensing and ERO reviews diverting services' focus away from engaging in PD, internal service communication blocks that resulted in information not being passed on to all staff, staff changes slowing engagement and progress, and management not providing the structural support (such as non-contact time or recompense for working afterhours) to help staff actively engage in their programme. The three factors rated by services as most likely to impact on the development of an effective relationship with their provider were time for the facilitator to get to know the service, the geographical area covered by the facilitator, and access to the facilitator.
Few services declined to engage when invited to participate in PD programmes. Reasons for non-engagement included: umbrella organisations refusing permission for their services to participate, services having their PD needs met through their umbrella organisation, services engaging in other MoE or Ministry of Health-funded PD programmes, teachers undergoing initial teacher education programmes or advanced study, services feeling that the identified focus areas did not meet their PD needs, and services in 'survival mode' who did not have the capability to take on a PD programme. Particular issues of engagement and relationship building evident for some Māori and Pasifika Immersion services were noted in the previous section.
The opportunities providers created for services to connect with and work with others were actively taken up by teachers with significant levels of increased communications with groups beyond the service, most especially with other ECE services and with schools. Services reported engaging mostly with other ECE services and schools but also with a range of special education, health and social services and with local iwi and community cultural groups. Where services have developed relationships with stakeholders beyond ECE as a result of their PD programmes, these have led to qualitative shifts, particularly in teacher attitudes and confidence.
It is clear that the management and leadership of ECE services can have a significant influence on the success or otherwise of a PD programme, either through the support that is put in place for teachers or the barriers that are erected. In an environment where access to PD programmes is targeted and therefore some services are receiving a greater investment from government than others – who are still expected to "take all reasonable steps to provide staff employed or engaged in the services with adequate professional support, professional development opportunities, and resources" (Ministry of Education, 2009) – it is reasonable to expect that the management of services receiving MoE-funded PD actively support their staff to effectively engage in their programme. Such support should address structural elements such as ensuring that staff have access to sufficient non-contact time to be able to collect and analyse data and that MoE-funded PD undertaken afterhours is adequately and appropriately acknowledged and recompensed.
The commitment of providers to engage with ECE services and to build relationships with them and other stakeholders has been exemplary. An extensive set of strategies have been employed by providers to engage eligible services in PD, with considerable time put into building relationships and tailoring the delivery of the programmes to meet services' needs and contexts.
Factors contributing to providers' success in engaging services included: targeting management personnel to build their commitment to the PD process, using consultative approaches with services to determine PD focus areas, using an appreciative inquiry model that began from services' strengths, and having skilled facilitators who, in particular, were culturally competent and listened carefully to services' needs.
Mixed success in developing communities of practice beyond the individual service is evident. Where PD clusters have been established and are working successfully, participants' learning has been considerable. In these situations, participants have had opportunities to break down barriers, address misconceptions about others' roles and work practices (particularly in those clusters addressing transitions to school), and to learn from others' perspectives. Having opportunities to visit other services and develop stronger relationships between services has contributed to participants' learning, both about the focus of the PD and as members of the community of practice.
Providers use a wide range of strategies and tools to support the development of communities of practice, and most services felt that their provider was 'Very effective' (23%) or 'Effective' (61%) in building communities of practice. A key element of success in the cluster group models, and thus the learning of participants, has been the identification and mentoring of key teachers who take on leadership roles within the cluster or their service. The provision of leadership clusters and mentoring programmes is supporting services to develop their communities of practice and grow into self-sustaining communities.
Factors that impacted on how effectively cluster groups developed into communities of practice included logistical challenges such as geographical isolation and distances that providers and/or services had to travel for meetings; time constraints; and coordinating meeting dates across different services and organisations such as schools. The influence of relational aspects – teacher confidence; existing relationships and levels of trust or distrust between teachers and services – impacted on the development of clusters into communities of practice.
There is evidence that the homogeneity of services and providers involved influences the effectiveness of cluster groups. Playcentre (both services and provider) highlighted the importance of their shared philosophy; in contrast some Pasifika and Māori Immersion services identified that they did not make as much progress in their PD when their provider did not share cultural understandings and felt frustrated when their progress was slowed by working in clusters with non-Immersion services. Heterogeneous cluster groups appear to work more effectively for non-Immersion services that are able to gain additional benefit from working alongside services that are steeped in, particularly, Māori or Pacific Nations cultures and languages.
Multiple and somewhat confused definitions of communities of practice are evident across both providers and services. Services tend to define, as a minimum, that their community of practice includes parents/whānau along with children, and may extend to include other ECE services, schools and other organisations and personnel such as health and social services. Providers tend to define communities of practice by their purpose, rather than by their membership, suggesting that they are groups with a common purpose.
Given these multiple definitions, it may be helpful to develop clarity within the ECE sector about communities of practice, along with other commonly used descriptors such as learning communities and professional learning communities. Whilst Lave and Wenger's (1991) initial use of the term, communities of practice, focused on the situated nature of learning, particularly in workplaces, it also emphasised the concept of legitimate peripheral participation whereby newcomers were inducted into the community and were allowed a reduced role whilst they learnt how things were done in that particular community. In expanding this definition of a community of practice when he argued that learning is fundamentally a social phenomenon that occurs when people actively participate in the practices of social communities, Wenger (1998) also identified specific indicators that suggest a community of practice had formed.
In contrast, the literature on professional learning communities specifically positions these entities in a more defined way. Stoll, Bolam, McMahon, Wallace and Thomas (2006) suggest that professional learning communities are:
Such a definition suggests that professional learning communities are likely to be established and maintained in a more deliberate manner and for a more specific purpose that would necessarily be the case with Wenger's communities of practice. The two concepts do, however, share some fundamental similarities such as the development of shared understandings and beliefs, active participation of members, the mutual influence of members on the community and vice versa, and a collaborative approach to negotiating meaning, and thus learning, within the community (Hipp, Huffman, Pankake & Olivier, 2008; Stoll et al., 2006; Wenger, 1998).
A third term found in the literature is learning communities. Multiple definitions abound and parallels are drawn in the literature with Wenger's communities of practice; Kilpatrick, Barrett and Jones (2003, p. 11) offer the following definition:
These definitions suggest that when services include children, parents/whānau and wider community organisations and agencies that they are, in fact, referring to a learning community, whereas the communities established for the purposes of professional development (whether cluster groups or whole-service teams) are professional learning communities. Adopting such definitions is more than just a matter of semantics as the expectations of and demands on a professional learning community are considerably greater than for a learning community. Establishing expectations that teachers will engage in critical interrogation and reflection of their practice in an on-going and collaborative manner is an important feature of effective professional development (Mitchell & Cubey, 2003, and using the research evidence on effective professional learning communities will support providers in setting these expectations from the outset of their programmes.
The timing of the evaluation, midway through the contract period, meant that limited data was able to be collected on this aspect. Although providers had moved into phase two of their programmes, a significant number of services from phase one were continuing their programmes into phase two; those services who had completed their programme at the end of phase one had only recently finished and thus had little experience of being involved in self-sustaining communities. The available data does suggest, however, that services find it hard to, or are anxious about sustaining either their communities of practice or their new practices without input from their facilitator to keep them on track.
Providers are using a number of strategies to support services to maintain their gains and momentum following their intensive programmes, including establishing network groups (several of which were becoming increasingly independent of facilitator input), regular programmes of seminars and workshops available to services beyond the in-depth or cluster programmes, and open access to online resources and webinars through their websites. The success of these initiatives will need to be monitored over the remainder of the contract period.
The introduction of the rubrics reflects the growing emphasis on finding effective ways of evaluating PD programmes and teacher development (Demimonde, 2009; Early, 2010), and the recognition that evaluation needs to be considered from the beginning of a PD programme (Early, 2010; Kuijpers, Houtveen & Webbels, 2010). As a new and iterative statement, effective use of the rubrics is clearly a work in progress. Variability in how the rubrics are being introduced and used by providers with services is evident, reflective perhaps of providers' journeys in understanding and integrating the rubrics into their PD programmes. The evidence of effective strategies reported in this evaluation for introducing and integrating the rubrics into PD may assist in more consistent use by providers.
Working with the rubrics has contributed to attitudinal changes for teachers through 'raising the bar' on what effective practice looks like, heightening teacher awareness of areas where their practices may be less effective, and provoking reflection on current practices. Including rubrics A, B and C re-emphasised for services that culture, language and identity, working with parents/whānau, and leadership should always be important elements within their programmes and practices.
Learning for teachers from participating in the process of using the rubrics and initiating a subsequent self-review includes understanding: the process of evaluation, of establishing a goal, developing an action plan, and of identifying how they might find the evidence to assess /demonstrate their increasing effectiveness. For many services these were new concepts.
Initial resistance to engaging with the rubrics reflects the fact that they were a new and unfamiliar tool for teachers – and also for PD providers. Therefore there was little initial awareness of the scope of the task or the level of thoughtfulness required to engage in a realistic evaluation, and some services were given the task of evaluating themselves against the rubrics with little information and/or support.
The contracts required that providers used the rubrics at the start and completion of the PD programme with each service in order to assess programme effectiveness. The fact that around a third of services responding to the online survey were either not aware of the rubrics or had not used them at all indicates some services or provider/s may not be prioritising teachers' engagement with them. It is also not always clear which rubrics providers are using with services.
Factors which support providers' effectiveness in using the rubrics with services include: PD providers' support for their facilitators' own engagement with the rubrics; thoughtful introduction and use of the rubrics with teachers during the sensitive relationship-building phase of the PD process; joint engagement in the process with the service; and addressing the rubrics at a time when all/most of the team can be present. Some providers have adapted the rubrics to make them more user-friendly, useful, and appropriate for the particular context.
Several factors impeding providers' effectiveness in using the rubrics were identified: lack of facilitator familiarity with the document and/or lack of experience in introducing and using them effectively with services; pressure of time – wanting to get on with the PD itself; facilitators not being accessible, or being inflexible in meeting times; working only with clusters and therefore less able to provide service-specific support.
The willingness of the MoE to respond to critique about the rubrics has been helpful; providers are positive about the changes incorporated in the second edition. Further suggestions from teachers and providers for improvements to the rubrics documentation focus particularly on it being 'difficult' to understand and to details of the grading system which make it challenging to use as an evaluation tool.
The rubrics are most effective when thoroughly integrated into the PD programme. In-depth discussions between the facilitator and service personnel that draw on evidence of practice, akin to what Earl and Timperley (2008) describe as learning conversations, provide more opportunities for reflection and professional growth than when either the service or the facilitator complete the rubrics ratings on their own. Allowing sufficient time for these conversations to take place, particularly when working with services where new relationships are being developed concurrently, is an important consideration and one that may need to be more explicitly factored into programme hours in future. Inclusion of time to review the rubrics formatively during PD programmes should also be considered.
The depth of teachers' attitudinal change is reflected in the number of services that have not altered – or may even have moved backwards—on their rubric rating post PD. Factors leading to this apparent lack of progress include: teachers' increased awareness of what constitutes 'Highly effective'; a more realistic recognition of their own developing effectiveness through the PD journey; a greater awareness of the process of evaluation; and a willingness to engage with the process more honestly with their now-familiar facilitator.
Service-related factors that support engagement with the rubrics are: management involvement and support, allotted time for the task; strategies for involving the whole team; and a person who understands the task and is willing to take leadership of the process.
Potential barriers within the service are: lack of support from management; rubrics evaluation completed with no/very limited teacher involvement; no allotted time; evening meetings which clash with other commitments for staff; staff members not seeing such tasks as part of their teaching role – particularly when it involves attending in their own unpaid time; high numbers of unqualified staff who may lack pedagogical knowledge; and high numbers of relievers who may be less committed.
There is considerable evidence that the rubrics made a significant contribution to the direction and depth of the PD journey for many services. They provoked discussion and reflection among teachers; were a tool with which to evaluate their current programme and practices and identify the gap between their own and the 'highly effective' model; they often influenced a team's choice of focus and goal for their PD programme; and they were the tool with which teams could measure and reflect on progress and affirm achievements. The requirement to engage with rubrics A, B and C re-emphasised for teachers the underlying importance of culture, language and identity, working with parents/whānau and leadership.
Although outside the scope of this evaluation, the development of the rubrics statement within a targeted PD environment raises the issue of how services outside the targeted communities are able to access and be supported to use the rubrics as a tool in their own self-review practices. Knowledge of the rubrics is likely to spread beyond services in the targeted communities and the MoE may wish to consider the development of a resource around the rubrics that provides appropriate guidance for services wanting to use the rubrics independently. Given the evidence to date, that services engaging with the rubrics frequently overrate themselves in their initial assessment, such guidance is important so that services don't misuse the rubrics as confirming existing practices.
The delivery of effective professional development programmes for diverse ECE services across multiple focus areas (literacy and numeracy, social competence, transitions, education and care for children under two, and leadership) together with underpinning themes (identity, language and culture; engaging parents in their children's learning; developing sustainable communities of practice) is highly complex. Determining the effectiveness of these PD programmes has also been complex, given that the methodology used for this evaluation did not enable the 'effect size' of programmes to be readily measured in comparison to other influences on teachers' and services' improvements in practice. Rather, the generally qualitative evaluation methodology has enabled current ECE service practices to be identified and some of the influences on those practices to be teased out.
Multiple influences on services' success within their PD programmes are identifiable. These can be broadly grouped as provider, service or policy influences and may, in many cases, have a positive or a negative impact on progress. For example, issues concerning leadership within services indicate that effective, proactive leadership supports success with PD whilst a lack of leadership or ineffectual leadership can slow down or derail progress. Previous evaluations (e.g., Cherrington & Wansbrough, 2007; Gaffney, 2003) have highlighted many of these issues which still exist as barriers today: leadership, changes in staffing, structural issues such as time and support for PD, travel (for both providers and isolated services), unqualified staff, and gaps in professional knowledge. Similarly, enablers such as an emphasis on relationship building (between provider and service; provider, service and community; and between service and parents/whānau) and facilitator characteristics (such as a willingness to listen to services' needs, cultural competence, knowledge of the service, and knowledge) have also been documented in previous evaluations and were evident in this evaluation. Two additional enablers are more evident in this evaluation: 1) providers' commitment to the services and their flexibility in tailoring programmes to meet services' needs, and 2) providers' increased attention to supporting services to gather evidence through the rubrics and self-review processes and to engage in critique of practice using that evidence is an aspect that has been strengthened in the current contract round.
Policy issues have also influenced the success of these PD programmes: the requirement that providers work closely with local MoE officials to determine priorities within the targeted communities has helped strengthen relationships between MoE and providers, resulting in a more unified engagement with services. The rubrics, whilst not universally popular or understood, are beginning to make a positive impact as providers and services engage with them as reflective and evaluative tools. As noted by one provider, they create opportunities for some of the hard questions to be asked and force services to really look for evidence to support their ratings.
The delivery of multiple professional development opportunities by the MoE (such as the Incredible Years and quick response programmes) together with programme offered by other government departments such as the Ministry of Health – often to the same targeted communities – seems, at times, to be overloading services and impacting on their rate of progress. There also appears potential to strengthen communication and liaison between different Ministries and between local and head offices, to support services to prioritise which programmes they engage with. Finally, funding changes appear to have had a negative impact on some services' ability to engage in and progress their programmes – reductions in the availability of non-contact time, increased numbers of unqualified staff, and absent staff not being replaced when the service is still able to maintain regulatory requirements were all identified as barriers to making progress with PD programmes.
Differences between targeted programme and national programmes
The prime focus of this evaluation has been on the largest component of the MoE-funded ECE PD programmes – whole-centre/service and cluster PD – although both the national leadership and education for children under-two programmes were integrated into the evaluation process. Data collected from providers has focused more heavily on programme one, given that seven of the nine providers participating in the evaluation only delivered programme one. Similarly, 14 of the 16 case study services were involved in programme one. However, more than half of the survey respondents had been enrolled in either programme two and/or three or in programme one plus programme two and/or three, enabling analysis of the results by these groupings of respondents. As the data reported in Chapters 4 through 8 indicate, overwhelmingly the differences in the responses by programme groupings are small with few clear trends emerging. The one key area where significant differences were noted was in the strategies used by providers of different programmes to sustain communities of practice after the completion of PD programmes: respondents who had participated in a national programme were significantly more likely to report their provider(s) used online strategies whilst those in programme one reported much greater use of face-to-face strategies.