Assessing skills of adult learners in 2011 Publications
This is an initial statistical report on the Literacy and Numeracy for Adults Assessment Tool in terms of the extent of its use in 2011, the first full year of implementation, the reading and numeracy profiles of learners when first assessed, and the extent to which learners can be seen to have increased their reading and numeracy skills over the course of the 2011 year.
Author(s): Chris Lane, Tertiary Sector Performance Analysis, Ministry of Education.
Date Published: February 2013
- In 2011, 77,000 learners were assessed using the Literacy and Numeracy for Adults Assessment Tool, and over 200,000 individual assessments were carried out, across the skills of reading, writing, vocabulary and numeracy.
- More than three quarters of assessments were for reading and numeracy.
- Some of the learners enrolled at each qualification level were assessed, but the focus was on New Zealand Qualification Framework Levels 1 to 4: the percentage of learners assessed in these levels ranged from 16 per cent at Level 4 to 31 per cent at Level 3 (in programmes funded by the Student Achievement Component or Youth Guarantee).
- Youth Guarantee learners were the most thoroughly assessed group, with over 70 per cent assessed at least once, and over 35 per cent assessed at least twice for one or more skills.
- Among learners enrolled in SAC- or YG-funded programmes at NZQF Levels 1 to 3, the rate of assessment was higher among learners enrolled at Institutes of Technology and Polytechnics (36 per cent) than at Private Training Establishments (20 per cent) or at wānanga (15 per cent).
- Assessed learners covered a wide age range, but there was a focus on young learners: 31 per cent of assessed learners were aged 16 to 19 on first assessment in 2011.
- Assessment Tool scores can be converted to Steps on the Tertiary Education Commission's Learning Progressions for Adult Literacy and Numeracy, which range from lowest skill at Step 1 to highest at Step 6, although there is not a direct correspondence between the Steps for literacy and for numeracy.
- The differences between the Learning Progressions for literacy and for numeracy were reflected in the fact that learners overall were assessed lower on the reading than on the general numeracy steps, with approximately half of learners assessed for reading scoring in the top three steps (Learning Progressions Steps 4, 5 and 6), while for general numeracy approximately half of learners scored in the top two steps (Steps 5 and 6).
- Although the groups of learners assessed using the Assessment Tool were clearly not representative of the adult population, the variations in Assessment Tool reading and general numeracy scores followed the same patterns as the total adult population according to the Adult Literacy and Life Skills (ALL) Survey 2006, in terms of age, gender, ethnicity, first language, and educational participation.
- Among learners assessed at least twice for reading or general numeracy, more than half of those who scored at the bottom of the scale (Step 1) on their first assessment recorded statistically significant gains in skill during 2011. Among others, the rates of significant gain were progressively less the higher their first assessment scores. Learners whose first assessment scores were at the top of the scale (Step 6) had little scope to show significant gain.
Use of the Assessment Tool in 2011
The picture of use of the Assessment Tool in 2011 is highly dynamic, with education providers and other organisation beginning to use the Tool either during the pilot year of 2010, or some time during 2011. The Assessment Tool was still undergoing development, with a short 'Snapshot' version of the Tool becoming available in February 2011 for reading and numeracy. By November this had become the most common type of assessment for those skills. It remains to be seen whether the patterns of results reported here continue to be evident in the use of the Assessment Tool, or whether these patterns represent an initial phase before settling into a more stable state.
In 2011, 77,000 learners were assessed at least once using the Assessment Tool, and over 200,000 individual assessments were undertaken, with separate assessments for reading, vocabulary, writing, general numeracy and two specific sub-skills of numeracy. More than three-quarters of those assessments were either for reading or for general numeracy.
The Assessment Tool was used by 262 organisations in 2011. More than three-quarters of assessments were completed either by private training establishments (PTEs) or by institutes of technology or polytechnics (ITPs).
Learners assessed in 2011 covered a wide range of ages, but there was a concentration on young learners, with learners aged 16-19 accounting for about 31 per cent. Fifty-six per cent of learners assessed were male. The ethnic distribution of assessed learners largely followed that of learners enrolled in Level 1 to 3 qualifications, which meant that Māori and Pasifika were over-represented in comparison with the general population, together accounting for over 40 per cent of learners assessed, while people who identified as European or Asian were under-represented in comparison with the population. The proportion of assessed learners whose first language was not English was 17.5 per cent: this is close to the estimate for the population aged 16 to 65 from the Adult Literacy and Life Skills (ALL) Survey 2006.
Some of the learners enrolled at each qualification level were assessed, but the focus was on New Zealand Qualification Framework Levels 1 to 4: the proportion assessed at least once was between 16 per cent at Level 4 and 31 per cent at Level 3 (in programmes funded by the Student Achievement Component or Youth Guarantee). In contrast, less than 5 per cent of learners enrolled at Level 5 or above (in SAC- or YG-funded certificate, diploma or degree programmes) were assessed.
This focus is in line with the aims of the government's Tertiary Education Strategy for the development of learners' literacy and numeracy skills, and in line with the Tertiary Education Commission's guidelines for embedding literacy and numeracy learning across all fields of study. Both of these are directed in particular towards learners at Levels 1 to 3.
The greatest use of the Assessment Tool among NZQF Level 1 to 3 programmes was in Youth Guarantee fees-free tertiary places, in which 76 per cent of learners were assessed during 2011.
Of learners studying in SAC- or YG-funded programmes at NZQF Levels 1 to 3, 36 per cent of those enrolled at ITPs were assessed, and 20 per cent at PTEs were assessed, while 15 per cent of those enrolled at wānanga were assessed.
Learners' reading and numeracy profiles on first assessment in 2011
The differences between the Learning Progressions for literacy and for numeracy were reflected in the fact that learners overall were assessed higher on the general numeracy than on the reading Learning Progressions, with the distribution of learners' assessments for general numeracy bunching around a peak at Step 5, while Steps 3 and 4 were nearly equal as the peak steps for reading.
Consequently, approximately half of learners assessed for reading scored in the top three steps (Learning Progressions Steps 4, 5 and 6), while for general numeracy approximately half of learners scored in the top two steps (Steps 5 and 6).
Reading and numeracy profiles according to demographic characteristics of learners were in line with previous research based on the Adult Literacy and Life Skills Survey 2006. Specifically, both skills increased with age from relatively low at 16-17 up to a peak among those aged 25-34, and then declined somewhat for older age groups. Men tended to score higher for general numeracy, but there was little gender difference in reading. Learners identifying as European or Māori scored higher in reading than those identifying as Pasifika or Asian, while European and Asian learners scored higher in general numeracy than Māori and Pasifika. Learners whose first language was English scored higher than learners with other first languages in reading and in general numeracy.
For learners whose Assessment Tool results could be matched to enrolments in SAC- or YG-funded programmes, the proportion of learners scoring in the upper part of the scales (Steps 4, 5 and 6 for reading, and Steps 5 and 6 for general numeracy) rose steadily with each higher NZQF level.
Although the groups of learners assessed using the Assessment Tool were clearly not representative of the adult population, the variations in Assessment Tool reading and general numeracy scores followed the same patterns as the total adult population according to the Adult Literacy and Life Skills (ALL) Survey 2006, in terms of age, gender, ethnicity, first language, and educational participation. This study was not designed to test the validity of the Assessment Tool, but these correspondences between the Assessment Tool and ALL Survey results do tend to confirm that the Assessment Tool measures the same skill constructs as the assessments used for the ALL Survey.
For learners enrolled in programmes at NZQF Levels 1 to 3, those in Youth Guarantee tended to have lower skills than those enrolled in Student Achievement Component (SAC) funded programmes. The proportion of learners enrolled in SAC- or YG-funded programmes at NZQF Levels 1 to 3 scoring in the upper part of the scales (Steps 4, 5 and 6 for reading, and Steps 5 and 6 for general numeracy) was higher for learners enrolled at ITPs and wānanga than in PTEs.
The extent of learner gain in reading and numeracy skills in 2011
Approximately a third of those learners who had been initially assessed at some time during 2011 were assessed again for the same skill in the course of the year.
Approximately 22,000 learners were assessed two or more times in 2011 for reading, and about 16,000 for general numeracy, including about 14,000 learners who were assessed more than once for both skills.
The extent of learner gain was explored by comparing the first and last reading and the first and last general numeracy assessments in 2011 for these learners. This approach was adopted for the purpose of researching learners' experiences, and is different from the approach proposed by the Tertiary Education Commission for measuring the performance of tertiary education organisations in literacy and numeracy.
Learners whose scores were low on first assessment were more likely to be assessed again than those who scored high on first assessment.
The literacy and numeracy scales have maximum and minimum values; so learners who scored near the maximum on their first assessment were not likely to show significant gain because there was little scope for them to improve their score on the Assessment Tool (even though they may have in fact improved their skills). On the other hand, learners who scored near the minimum had plenty of scope to improve their scores and tended to do so.
The main finding concerning learner gain was that a large proportion of learners (over 50 per cent for reading and for general numeracy) who scored near the minimum (i.e. at Step 1 on the Learning Progressions for adult literacy and numeracy) showed statistically significant gain in scores between first and last assessments in 2011. For learners in the middle parts of the scale (i.e. whose first assessment score corresponded to Steps 2 to 5 on the Learning Progressions), the proportion who showed statistically significant gain was not as great as this and was smaller for learners at the higher steps.
The extent to which learners who scored low on their first assessments showed statistically significant gain in scores varied somewhat according to demographic characteristics of the learners and characteristics of educational provision.
For learners whose first assessment scores were the lowest (i.e. Step 1), the proportion showing statistically significant gain, in general, increased with age, with the greatest proportion among learners aged 55 and over. The proportion was higher for men than women for both skills. Learners who identified as Asian showed the greatest rate of significant gain for general numeracy, while for reading, Māori and European learners showed greater rates of significant gain. Learners whose first language was English showed greater rates of significant gain for reading, but lower rates of gain for general numeracy.
Among learners studying in SAC- or YG-funded programmes at NZQF Levels 1 to 3, over 35 per cent of those who were initially assessed for reading or general numeracy were reassessed within the year. Of learners who scored at Step 1 on first assessment, those studying at higher NZQF Levels tended to experience higher rates of significant gain.
Learners in Youth Guarantee programmes were most likely to be reassessed for reading and general numeracy: in fact, over 50 per cent of Youth Guarantee learners were reassessed. Given that Youth Guarantee learners were most likely to be assessed at least once, this group of learners was clearly the most thoroughly assessed and reassessed. For learners whose first assessment was at Step 1 for either skill, Youth Guarantee learners also showed a greater rate of significant gain than learners in SAC-funded programmes.
NZQF Level 1 to 3 learners enrolled in SAC- or YG-funded programmes at PTEs and wānanga, whose first assessment for reading or for general numeracy was at the bottom of the scale (Step 1), had a somewhat higher rate of significant gain than corresponding learners, initially assessed at Step 1, who were enrolled at ITPs.