Summary

Background

The external evaluation and review (EER) reports published by the New Zealand Qualifications Authority (NZQA) provide a rich source of information on the quality of non-university tertiary education organisations and ways in which they can improve. In this report, the results from 43 EER reports for Institutes of Technology and Polytechnics (ITPs) were analysed to identify patterns and trends in the quality ratings of ITP educational performance and self-assessment.

With the creation in 2020 of a single institute of technology for New Zealand, Te Pūkenga, ITPs no longer exist as independent entities.^[1] However, an analysis of historical EERs can provide insights into what the quality was like in a large part of the New Zealand vocational education sector over more than a decade.

The EER system was introduced in 2009 and was designed to contribute to an environment where non-university Tertiary Education Organisations (TEOs) use self-assessment to improve performance. The EER measures the capability of self-assessment by TEOs as well as looking at the nature and quality of educational performance. In an EER report, ratings are assigned to the level of quality in various areas of outcomes (such as achievement by learners) and input processes (such as support for learners). Based on these, each of the two dimensions of educational performance and self-assessment receives a statement of confidence. The TEO can receive a statement of confidence at the following levels: ‘Highly Confident’, ‘Confident’, ‘’Not Yet Confident’ or ‘Not Confident’. From these statements of confidence an overall rating or ‘Category’ (1 representing the highest quality and 4 the lowest) is assigned to a TEO. The statements of confidence are published in the EER, along with the ratings for the underlying key evaluation questions (KEQs) and focus areas (areas/programmes that were selected to be evaluated). Recommendations and requirements are also published in the EER reports.

The results of an analysis of Category and the ratings published in the 43 EERs released between 2010 and 2021 are presented below.

Category of ITPs

Between 2010 and 2021 the level of quality of ITPs was generally high. Of the 43 full EERs carried out during this time, 19 (44 percent) resulted in an ITP being assessed as Category 1, 19 (44 percent) as Category 2, four (9 percent) as Category 3, and one (2 percent) Category 4.

Of the five EERs which resulted in a Category 3 or 4 rating, issues with moderation^[2] in the time leading up to the EER were mentioned in three of those cases. Significant structural change and/or turnover of staff was also mentioned in four of the EERs.  Generally, mergers of ITPs did not appear to have a detrimental impact on the Category assigned to the remaining institution.

The proportion of ITPs with a Category 1 rating declined over time. As at December 2015 just over 60 percent of ITPs were Category 1, but by March 2021 this had declined to 38 percent. Circumstances unique to individual ITPs played a role in this trend, but it also sits within a broader context of falling enrolments generally and deteriorating financial positions for some ITPs. It may also reflect the evolving nature of the EER process, where changes in indicators and the bedding in of the evaluation approach may have impacted the evaluation judgements.

Figure 1 - Distribution of Category of Institutes of Technology and Polytechnics, as at 31 December in each year

Notes: 1. The number of brackets refers to the number of ITPs. 2. N/A refers to Toi Ohomai, which began in 2016 but did not have an EER report published until 2018. 3. * The as at date for 2021 is the end of March.

An analysis of statements of confidence help identify whether it was educational performance or self-assessment that led to the decline in Category 1 ITPs.

Statements of confidence judgements

The drop in Category 1 ITPs followed from a decline in the level of confidence for the educational performance dimension. As at December 2013, 61 percent of ITPs were at the ‘Highly Confident’ level in educational performance, but this dropped to 38 percent by 2021.

The opposite trend was observed in the statement of confidence in self-assessment. Here there was an improving trend, albeit off a low base. The proportion of ITPs at the ‘Highly Confident’ level increased from 11 percent in 2013 to 44 percent in 2021.

Figure 2 - Distribution of statements of confidence for Institutes of Technology and Polytechnics, as at 31 December in each year

Notes: 1. The number of brackets refers to the number of ITPs. 2. N/A refers to Toi Ohomai, which began in 2016 but did not have an EER published until 2018. 3. * The as at date for 2021 is the end of March.

Key evaluation question ratings

The ratings assigned to the underlying KEQs provide more context for the trends in the statements of performance. In terms of educational performance, the ‘Achievement’ KEQ generally had the lowest proportion of ‘Excellent’ ratings and displayed a declining trend over time. As at December 2013, 39 percent of ITPs had an ‘Excellent’ rating, compared with 13 percent in 2021. Similarly, the ‘Outcomes’ KEQ showed a decrease in ‘Excellent’ ratings from 72 percent in 2013 to 44 percent in 2021. This was the main contributor to the decrease in ITPs at the ‘Highly Confident’ level in terms of the statement of confidence, as the other three KEQs that were analysed did not exhibit a consistent downwards trend over time. The ‘Support’ KEQ generally had the highest proportion of ‘Excellent’ ratings.

A closer look at the ‘Achievement’ KEQ ratings in educational performance suggested that the evaluation system evolved over time, with rising expectations and increasing attention on disparities for priority learner groups such as Māori and Pacific Peoples. This coincided with a stronger focus on equity for priority learner groups in Tertiary Education Strategies.

In terms of self-assessment, the KEQs generally showed a lower proportion of ‘Excellent’ ratings than in their related education performance KEQ. Apart from the ‘Achievement’ KEQ, there was an improvement over time in the proportion of ITPs with an ‘Excellent’ rating in the self-assessment KEQs, albeit off a lower base in some cases. This improvement took place in the first half of the analysis period with the ratings generally remaining unchanged between 2017 and 2021.

Ratings for focus areas

The analysis of the ratings assigned to qualifications within focus areas showed that for educational performance the highest proportion of ‘Marginal’/’Poor’ ratings were in qualifications at Level 5 (10 out of 49 qualifications). Level 7-10 qualifications had the highest proportion of ‘Excellent’ ratings (eight out of 116 qualifications). However, there was no distinct pattern of higher levels of qualification having higher quality, or vice versa. In self-assessment, Level 1-2 qualifications had the highest proportion of ‘Marginal’/’Poor’ ratings (16 out of 53 qualifications) while Level 7-10 qualifications had the lowest proportion (15 out of 116 qualifications). Once again, there was no clear pattern of higher levels having higher ratings, or vice versa.

The broad field of study of qualifications in the focus areas was also analysed. This showed that for educational performance, the field of Information technology had the highest proportion of ‘Marginal’/’Poor’ ratings (five out of 22 qualifications) with the field of Food, hospitality and personal services having the lowest proportion (one out of 35 qualifications). For self-assessment, the field of study with the highest proportion of ‘Marginal’/’Poor’ ratings was Management and commerce (22 out of 56 qualifications), while the field of Agriculture, environmental and related studies (two out of 24 qualifications) had the lowest proportion of these ratings.

Summary of recommendations in EER reports

Analysis of the recommendations listed by evaluators showed there were some common themes or ways to improve quality. One of the major themes was a recommendation for ITPs to increase capability to measure student achievement and graduate outcomes, as well as to analyse this data to inform improvements in future performance. Another theme was for assessment and moderation to be reviewed and for standards to be applied consistently across the organisation. This last theme sits within a wider overarching theme towards more consistency across an ITP in terms of policies, approaches and practice.