Evaluation of the nature and efficacy of support for English language learners Publications
This report presents the findings of an evaluation of the nature and efficacy of the Ministry of Education’s (the Ministry’s) funding and support for English Language Learners (ELLs) in primary, intermediate and secondary schools in Aotearoa New Zealand. The evaluation covers the Ministry’s funding mechanisms for English for Speakers of Other Languages (ESOL) in schools and related supports (including teacher education), the sufficiency of ESOL funding, and the nature of the resulting outcomes for schools, teachers, and ELLs. The Evaluation Report is accompanied by a Literature and Evidence review.
Author(s): Dr Calvin Scott, Anne Alkema, Natalie Piesse, Allen + Clarke
Date Published: August 2022
We conducted the evaluation throughout 2021. It began in March 2021 with the development of a programme logic in relation to ESOL that served as a thinking, planning and implementation tool for setting out causal relationships and correlations between all the resources, systems, activities and outcomes involved in the Ministry’s support for ELLs. In conjunction with the programme logic, we also developed an evaluative rubric that provided an evaluation description of what performance quality looks like for:
- the efficiency and accessibility of the Ministry’s support
- the relevance (or sufficiency) of resourcing
- outcomes for schools, staff, ELLs and their families.
In April 2021 we completed an extensive literature and evidence review of existing published research and documentation in relation to learning the language of instruction at school as a migrant or former refugee student. Our collection of primary data involved an online questionnaire open from mid-June to July 2021 for all 1,600 schools that receive ESOL funding as well as practice site visits to schools (via Zoom or in person) in August and September. Our analysis of these collected data has been guided by the criteria and dimensions of the evaluation rubric and the components of the programme logic.
We are mindful that, while the Ministry allocates ESOL funding based on a set formula, schools are autonomous in determining how this funding is used to support their ELLs. As a consequence, it is the individual school’s decision-making and teaching practices that ultimately shape outcomes for ELLs. Therefore, a direct or immediate attribution between the Ministry itself and these outcomes cannot be made. Similarly, beyond a school’s chosen use of their ESOL funding, other contingencies impact on outcomes for ELLs, such as the variety pedagogical approaches applied in schools, the atmosphere in the classroom, home and community environments, and the character and disposition of the individual ELL.
Efficiency, accessibility, and relevance of resourcing
Based on the evaluative rubric, our criteria for the dimensions of efficiency, accessibility, and relevance of resourcing are determined quantitively from the results of our online survey. In relation to the criteria of efficiency and accessibility of the Ministry’s systems (i.e., funding application and verification processes, timely and responsive provision of advice and assistance) the rating level is excellent: approximately 95 percent of survey respondents indicated that what the Ministry does works very well for them. In terms of schools knowing what supports are available, and how to apply for funding, the rating is good, with approximately 85 percent stating they are aware of all available support for their ELLs through the Ministry.
While the current funding mechanisms work well, our research indicated that there are alternative approaches to the distribution of funding that would make it easier for schools, for example using schools census data or a rolling (rather than a biannual) application process.
Our evaluation revealed some areas for improvement: firstly, the relatively low extent to which schools use funding to access professional learning opportunities; and, secondly, the extent to which there is insufficient funding to meet the needs of ELLs. Because fewer than 60 percent of all participants indicate that this is the case in both of these categories, the rating in both of these dimensions is poor.
From a teaching perspective, the lack of access to, or use of, professional learning opportunities is a particular cause for concern given that specialist ESOL teacher capability is crucial to the educational progress of ELLs.
The sufficiency of funding was the largest area of concern for everyone who contributed their insights to this evaluation. While not all of them queried the amount per ELL, they did query the amount of time for which ELLs can be funded. New Zealand-born ELLs are of particular concern as they are funded for only three years and may come from families where English is not spoken in the home, in their families or communities. Also of concern are those who have come from full-immersion kura and are not eligible for funding.
Outcomes for schools, staff and ELLs
Despite the insufficiency of funding, data collected through the online survey and practice sites showed that approximately 95 percent of respondents and interviewees involved in identifying students’ language-learning needs (and who, in the main, have ESOL roles in schools) are confident they do so accurately. This is accomplished through a variety of mechanisms including testing, observations, conversations with teachers and senior staff, and interviews with ELLs and their families on enrolment. While the rating for this dimension is ‘excellent’, based on the ‘confidence’ expressed by respondents in the survey, supporting data from practice sites show schools do receive funding for the majority of ELLs for whom they apply.
In relation to staff we have given a rating of adequate: while schools do have staff with formal ESOL qualifications, they are few in number. Although the majority of staff require support to develop their ESOL capability, for a variety of reasons (e.g., competing priorities, few opportunities) they are not accessing professional learning. A tension then arises where ESOL experts in schools may assess ELLs’ language-learning needs and then the actual ESOL teaching is left to mainstream teachers or teacher aides who may not possess the necessary specialist ESOL knowledge or experience.
The school leaders who participated in the evaluation emphasised how their schools are culturally responsive and have policies and plans for fostering and supporting cultural diversity. However, because they still face challenges in enacting these policies and plans, we have rated this outcome as good.
The extent of a given school’s ESOL provision and practice cannot be solely attributed to the Ministry’s ESOL funding: data show that schools variously top up ESOL funding from their operational grants or from international fee-paying students. Giving a definitive rating for outcomes for learners is difficult as we feel we have insufficient data from the ELLs’ perspective (particularly in relation to wellbeing) and the variance we noted between schools. Therefore, the rating of good/adequate is probably the fairest. Although the National Certificate of Educational Achievement (NCEA) results provided by the Ministry do show that ELLs achieve educational parity, more needs to be known about these ELLs in relation to the overall retention of ELLs at school; their ethnic affiliations; the time spent at school New Zealand prior to NCEA; the subjects and credits they are achieving; and their school leaver destinations. In terms of wellbeing, while most schools are working hard to develop culturally inclusive environments more still needs to be known about the extent to which ELLs really have a sense of belonging.
Although the following recommendations are based primarily on the findings of our evaluation, they also reflect successful international practice as documented in this report’s companion document, Evaluation of the Nature and Efficacy of Support for English Language Learners - Literature and Evidence Review (April 2021).
- Reconsider the policy setting of ‘special educational needs’ through which ELLs are currently funded and align it with government policies and funding mechanisms that relate directly to learning, diversity and settlement outcomes.
- Review and adjust current ESOL funding levels to allow ELLs to access increased funding support for longer periods, particularly those who are New Zealand born, and consider the extent to which there should be funding for ELLs who have come from full immersion kura.
- Conduct further research on the processes and funding levels in the refugee specific funds, and for the Bilingual Assessment Service and Bilingual Support Workers.
- Review the funding application approach schools are required to use and test the efficacy of alternative models.
Building workforce capability
- Review Professional Learning and Development (PLD) needs and how a greater mix of provision could support the capability-building of mainstream and ESOL teachers (for example, qualifications, internal and external face-to-face workshops, and online resources). Consider ways of increasing the number of relevant, flexible options such as funded workshops and modules, taking blended and online approaches to staff participation.
- Provide targeted training and more employment opportunities for bilingual ESOL teachers, support workers, and teacher aides.
- Review the extent to which Professional Learning Communities (PLCs) are working and how these could better be supported.
- Review and update ESOL resources, including what is provided on ESOL online.
Educational parity and inclusion
- Interrogate NCEA data (school and qualification) to find out more about those ELLs who are achieving educational parity and the time period during which they achieve this.
- Interrogate NCEA data to find out the credits that are being achieved and by whom, and the extent to which these credits allow for progression to further education or meaningful employment.
- Investigate ways in which more can be found out at a national level about wider wellbeing outcomes for ELLs.
Education Data Requests
If you have any questions about education data then please contact us at:
Email: Requests EDK
Phone: +64 4 463 8065