Comparing literacy and numeracy scales in the Assessment Tool and Survey of Adult Skills (PIAAC) Publications
This report is intended to help educators and researchers using the Literacy and Numeracy for Adults Assessment Tool understand the extent to which results from the Assessment Tool can be compared with the results from the Survey of Adult Skills.
Author(s): Ministry of Education
Date Published: March 2019
The Survey and the Assessment Tool were set up for different purposes and are administered in different contexts. However, they share similar concepts of literacy and numeracy. Comparing results helps educators and researchers consider where different groups of learners fit in terms of overall population patterns.
This report looks at the scores for people who were assessed in both the Survey and the Assessment Tool (see methodology section for more detail). It updates a previous paper which compared the scales in the 2006 Adult Literacy and Life Skills (ALL) Survey and the Assessment Tool (Earle, 2014).
Comparing the scales
The literacy and numeracy scales in the Survey and the Assessment Tool are moderately correlated. This suggests that the scales measure similar underlying concepts. However, there is a range of variation in scores of people who were assessed in both.
The variation in scores means it is not possible to provide an exact mapping. However, it is possible to map the range of scores that people are likely to have on the other instrument (as shown below).
People who scored at Step 1 for reading in the Assessment Tool are likely to score at Level 1 or below for literacy in the Survey. Those who scored at Step 4 or above are likely to score at Level 2 or higher. People who scored at Step 6 are likely to score at Level 3 or higher.
People who scored at Step 1 or 2 for numeracy in the Assessment Tool are likely to only score at below Level 1 for numeracy in the Survey. Those who scored at Step 5 or 6 are likely to score at Level 2 or higher in the Survey.
Based on these mappings, it is not possible to say that someone needs to achieve a specific step on the Assessment Tool in order to participate in employment, further study or everyday life.
The Literacy and Numeracy for Adults Assessment Tool
The Assessment Tool measures the literacy and numeracy skills of individuals. It was developed by the New Zealand Council for Educational Research, on behalf of the Tertiary Education Commission. It is widely used in lower-level tertiary education, schools and other settings.
The Assessment Tool is designed to show what individuals need to learn to improve their skills. Assessments are conducted in a wide range of contexts, including homes, classrooms, assessment centres, and prisons.
The Assessment Tool provides reading and numeracy scores for each individual on a 1000 point scale. The scores are divided into six steps, from Step 1 to Step 6. The steps are mapped to the learning progressions for adult literacy and numeracy (Tertiary Education Commission, 2008a, 2008b). The steps describe the kinds of literacy or numeracy tasks learners are able to do.
The Survey of Adult Skills
The Survey of Adult Skills estimates the distribution of literacy and numeracy skills in the population. It is an international survey coordinated by the Organisation for Economic Cooperation and Development. It was conducted in New Zealand in 2014/2015, as a household survey, drawing on a nationally representative sample of 16 to 65 year olds. The respondents were tested across literacy, numeracy and problem solving in technology-rich environments. The assessments were undertaken at home with an interviewer present, and included a background questionnaire.
Each respondent was tested in two of the three domains in order to reduce respondent burden. Scores in the other domains were imputed. This approach is valid for estimating the literacy and numeracy skills of population groups, but is less accurate in estimating the skills of each individual.
The Survey provides literacy and numeracy scores for each individual on a 500 point scale. To interpret the scores, they are divided into six levels (from below Level 1 to Level 5). At each level individuals can successfully complete certain types of tasks. These levels describe the kinds of tasks adults can typically complete and do not define the skill required for particular purposes, such as participation in modern society (Organisation for Economic Cooperation and Development, 2016).
Figure 1: Mapping of Literacy
Figure 2: Mapping of Numeracy
Interpreting the mapping
The purpose of the mapping is to help educators and researchers understand where groups of learners may sit in the overall skills distribution of the population. The mappings provide an approximate range over which the Assessment Tool steps and Survey levels can be compared.
For example, learners who are assessed at Step 1 or 2 on the numeracy assessment are likely to share characteristics with people who were assessed at below Level 1 in the Survey of Adults Skills. Educators and researchers can then look at published results from the Survey to learn more about adults with this level of skill.
The Survey of Adult Skills does not provide any benchmark for the level of skills that adults need to participate in work or society. The levels in the Survey describe the kinds of tasks people can do successfully. How well people at each skill level can participate will depend on the numeracy and literacy demands of their everyday life and work. This contrasts with the earlier surveys which made claims that people needed skills at Level 3 and above in order to fully participate in modern society.
Similarly, it is not possible, based on these mappings, to say that someone needs to achieve a specific step on the Assessment Tool in order to participate in employment, further study or everyday life. The Assessment Tool steps provide the learner with a description of what they are able to do. This can then be compared with what they need to be able to do for specific situations, such as work or study.
This report uses the same approach as in the previous paper (Earle, 2014). It looks at the scores of people who were aged 25 and over in the Survey and spoke English as their main language at home, who also had one or more assessments on the Assessment Tool. People under 25 were excluded as their skills are more likely to be changing over time. This may also be the case for people with lower English proficiency. The people who had scores in both the Survey and the Assessment will not be representative of all people in the Survey. This was explored in more depth in the previous paper.
The scores in the Survey are recorded as 10 plausible values. The mean of the plausible values was used to calculate the level for each person. Where a person had more than one assessment on the Assessment Tool, the mean value of their scores was used to calculate their step. Assessment Tool results with a standard error of greater or equal to 50 scale points were excluded. Otherwise, all assessment for each individual were used, covering a time period from 2010 to 2017.
When the Survey of Adults Skills was conducted, New Zealand’s results from the 2006 ALL Survey were rescaled to provide comparable literacy and numeracy scores. The analysis was also run using the rescaled ALL Survey. Both surveys produced very similar results. The results of the two surveys were combined to inform the mapping. This provided a total of 285 individuals with matched numeracy scores, and 325 with matched reading and literacy scores.
The Survey of Adult Skills was matched to the Assessment Tool in Statistics New Zealand’s Integrated Data Infrastructure (IDI). Only 2,950 of the 6,117 respondents in the Survey have a reliable match in the IDI. There were 123 matched individuals with Assessment Tool results in numeracy and 141 matched individuals with Assessment Tool results in reading.
In the ALL survey, around 70 per cent of participants were matched to education records using their National Student Number. This enabled the Ministry to directly match them to their Assessment Tool results. There were 162 individuals matched to Assessment Tool results in numeracy and 184 matched to Assessment Tool results in reading.
Results for the Survey of Adult Skills and the ALL survey were each cross tabulated by level and step. The table percentages for each were then added together and divided by two, to give equal weight to each survey.
The table below shows the correlation coefficients between both Surveys and the Assessment Tool. A 0.6 correlation coefficient indicates a moderate correlation, while 0.8 or higher is generally considered to be strong.
|ALL Survey||Survey of Adult Skills|
|Literacy / Reading||0.69||0.70|
The correlation statistics are Pearson’s coefficient.
The graphs below show the distribution of results in both Surveys and the Assessment Tool. Figures 3 and 5 compare the Assessment Tool steps to the Survey levels. Figures 4 and 6 compare the Survey levels to the Assessment Tool steps.
Figure 3: Literacy - Comparing Assessment Tool with Survey of Adult Skills
Figure 4: Literacy - Comparing Assessment Tool with Survey of Adult Skills
Figure 5: Numeracy - Comparing Survey of Adult Skills with Assessment Tool
Figure 6: Numeracy - Comparing Survey of Adult Skills with Assessment Tool
- Earle, D. (2014). Measures of Adult Literacy and Numeracy: comparing the Adult Literacy and Life Skills Survey and the Literacy and Numeracy for Adults Assessment Tool (Tertiary education occassional paper No. 2014/01). Wellington.
- Organisation for Economic Cooperation and Development. (2016). Skills Matter: further results from the Survey of Adult Skills. Paris.
- Tertiary Education Commission. (2008a). Learning progressions for adult literacy. Wellington.
- Tertiary Education Commission. (2008b). Learning progressions for adult numeracy. Wellington.
Use of data in the IDI
Access to the anonymised data used in this study was provided by Statistics New Zealand in accordance with security and confidentiality provisions of the Statistics Act 1975. The findings are not Official Statistics. The results in this report have been confidentialised to protect individuals from identification.