PIAAC measures of qualification, field of study and skill mismatch

Publication Details

This technical note looks at the three measures of mismatch in the Survey of Adult Skills: qualification level, field of study and skills. It provides general comment on the strengths and weaknesses of each measure and how robust they are for policy purposes. It concludes that the qualification level mismatch is the most robust measure. The field of study measure has considerable methodological issues and should be used with caution. The skills mismatch measure is not suitable for use in analysis or international comparisons.

Author(s): Amapola Generosa, MBIE and David Earle and Paul Satherley, Tertiary, Ministry of Education.

Date Published: December 2018

Overview

The Survey of Adult Skills provides three different measures of qualification and skill mismatch. The measures vary in definition and robustness. This note provides general comment on the strengths and weaknesses of each measure and how robust they are for policy purposes. The first section summarises the strengths and weaknesses of each measure. Further details are then provided on each measure.

The Survey of Adult Skills is part of the OECD’s Programme for the International Assessment of Adult Competencies (PIAAC).

This note has been developed from discussions between the Ministry of Education, the Ministry of Business, Innovation and Employment and Treasury.

Strengths and weaknesses of each measure

Qualification level mismatch

This measure compares the level of education a person has achieved with his/her self-assessment of what level of education would be needed to get his/her job.

This is the strongest and most robust of the three measures.

This measure can be used for international comparisons, although it does need to be interpreted carefully. Differences between countries may be due to differences in qualification and occupation structures, as well as labour market differences.

In interpreting this measure, regard needs to be given to age group and level of qualification. Other variables that may be useful to explore are differences by field of study and length of time the person has been in employment.

Field of study mismatch

This measure compares the field of study of the highest level of qualification attained with the occupation that the person is employed in. For each broad level of study there is a list of occupations that are considered to be 'matched'.

There are considerable methodological issues with this measure and it should be used with caution.

There are a number of issues with the matching methodology that are set out in the accompanying notes. As with qualification level mismatch, regard needs to be given to age group and level of qualification in any interpretation of the measure. In particular, if it is being used to assess the performance of a particular part of the education sector (e.g. school, vocational or university level), then the aggregate statistic is an inappropriate measure, and the field of study mismatch should be calculated for that level alone.

International comparisons based on this measure should be treated with extreme caution, as differences are likely to reflect the structure of qualifications and occupations in each country, as well as any real differences in labour market efficiency.

Looking at qualification level mismatch by field of study (that is, without also attempting to match to occupation) is likely to provide a better measure of over- and under-supply of qualifications.

Further developments of this measure are planned for the next cycle of PIAAC.

Skills mismatch

This measure attempts to identify people whose literacy, numeracy and/or problem solving skills are clearly much higher or much lower than required for their job.

There are considerable methodological issues with this measure, as set out in the accompanying note. It should be regarded as a developmental measure and not suitable for use in analysis or international comparisons.

Work is underway to significantly improve this measure in the next cycle of PIAAC.

Strength of the measures in explaining wages across countries

Figure 1 compares the effect of the three mismatch measures on explaining wage differences. It compares the wages of a person who is overqualified, overskilled or with field of study mismatch with the wages of a well-matched person with the same level and/or field of study. Age, experiences and other factors have been controlled for in the models.

The effect of qualification level mismatch is shown by the height of the bars. The results show that people who are overqualified have lower wages than people with the same level of qualification in well-matched jobs. This effect varies across countries. New Zealand is around the OECD average. All results are statistically significant.

The effect of field of study mismatch is shown by the circles. The grey circles indicate that the effect is not statistically significant. For most countries, including New Zealand, the effect of field of study mismatch on wages is not statistically significant, once qualification level mismatch is taken into account. This suggests that the field of study mismatch on its own does not affect worker productivity.

The relationship between the skills mismatch measure and wages varies considerably across countries. It would appear that in some countries it is beneficial to be in a job for which one is overskilled, compared to a job where one is well matched. This reinforces concerns about the credibility of the measure.

Figure: 1 Effect of qualification, literacy and field of study mismatch on wages

Qualification mismatch measure

Qualification mismatch, as used in the OECD Skills Matter 2016 report, is based on a sample of employed participants in the Survey of Adult Skills (PIAAC), excluding those who were in self-employment or in multiple jobs.

To measure this type of mismatch, the OECD compared the level of education a person effectively achieved with his/her self-assessment of what would be the current usual level of education needed to get his/her job.

This approach aggregates the highest qualification level attained and the education level needed to get the job into comparable categories to define over-qualification, under-qualification and well-matched qualification.

A person is classified as overqualified if his/her qualification level is higher than that required to get the job; he/she is classified as underqualified if lower; well-matched if the same.

Question D_Q21a (If applying today, what would be the usual qualifications, if any, that someone would need to GET this type of job?) and the EDCAT6 from the PIAAC dataset were used to replicate the qualification mismatch rates presented in the OECD report. They confirm the methodology the OECD used to estimate the prevalence of qualification mismatch in the participating countries (refer to Table 1).

We find that the OECD approach is consistent with measures used in the literature on over-education, and has the advantage of being job-specific.

Based on how qualification mismatch is defined and measured in the OECD report, the phrase persons who are qualification mismatched refers to persons in employment who are in jobs where their qualification requirements do not correspond to their highest qualifications.

Overall, we find that OECD’s qualification mismatch indicator is useful and informative when we refer to people in employment who are mismatched by level of qualification. When defined and measured this way, this indicator is readily implementable, and allows for international comparison. Compared to the other indicators of mismatch (skills and field of study), the qualification mismatch indicator is more robust and can provide insights for policy analysis and development.

Analysis

We investigated the qualification mismatch indicator reported for New Zealand in the OECD Skills Matter 2016 report. The investigation aimed at replicating the prevalence of qualification mismatch reported in the OECD report, and testing whether the prevalence is sensitive to different measures used in the calculations.

We compared three qualification mismatch measures:

  • Mismatch by Qualification levels (QM1) : Compares the level of education a person effectively achieved with his/her self-assessment of the level of education needed to get his/her job. The OECD adopted this approach, which aggregates the highest qualification level attained and the education level needed to get the job into comparable categories.
  • Mismatch by imputed years of education (QM2): Compares the years of formal education a person achieved with the years of formal education required to get his/her job.
  • Self-report (QM3): This measure is similar to what was used in the Skills and Work 2016 report, and is based on the question: Thinking about whether this qualification, or level of schooling, is NECESSARY for doing your job satisfactorily.

Given that mismatch compares the characteristics of the job with individual characteristics, the above methods were applied to an analytical sample (n=3,328) that included those who were employed, and excluded those who were self-employed or in multiple jobs.

Table 1 summarises results from applying the three approaches to the analytical sample of paid employees. The second column lists the PIAAC variables used to estimate the rates of qualification mismatch for the New Zealand sample.

Table 1: Prevalence of qualification mismatch (% of paid employees, n=3,328)
Note:
  1. Yrsqual: highest level of education obtained imputed into years of education (derived); yrsget: imputed years of formal education needed to get the job (self-reported – derived)
Qualification
mismatch measures
PIAAC
Variable
Used
Under
Qualified
Over
Qualified
Qualification
-matched
Qualification level (QM1)
(OECD’s methodology)
EDCAT6 10.8 (0.65) 33.6 (0.91) 55.6 (0.87)
D_Q12a    
Imputed years of formal education (QM2) YRSQUAL 17.1 (0.75) 53.6 (1.05) 29.3 (0.92)
YRSGET    
Self-reported (QM3) D_Q12b 8.9 (0.62) 28.9 (1.21) 62.2 (1.33)  

Field of study mismatch measure

How field of study is collected and coded

Where a person’s highest qualification is equivalent to NCEA level 1 or higher, they are asked for its main field of study. The New Zealand question text ran: What was the main subject for your highest qualification? If there was more than one, please choose the one you consider most important.

The international version of the questionnaire had nine response options – the broad levels of the Eurostat field of education and training classification:

  • General programmes
  • Teacher training and education science
  • Humanities, languages and arts
  • Social sciences, business and law
  • Science, mathematics and computing
  • Engineering, manufacturing and construction
  • Agriculture and veterinary
  • Health and welfare
  • Services

Countries were allowed to ask an open question in order to code responses to a national field of study classification, or the detailed levels of the Eurostat one. In New Zealand, we took the open question option and coded the text strings to the detailed level of Statistics New Zealand’s NZCED Field of Study Classification. These codes were mapped to the above nine categories for the OECD to create our international variable.

How field of study match and mismatch are derived

For those working, the OECD has listed sets of occupation codes (three-digit International Standard Classification of Occupations (ISCO) codes) that are deemed matched occupations for each of the field of study codes 2-9. If a person’s occupation isn’t in the list for their field of study they are deemed mismatched. Some occupations are matched under more than one field of study.

Where any of the following conditions are met, field of study match/mismatch is not derived and the variable for field of study match/mismatch is set to missing.

  • no school qualifications
  • field of study is General programmes
  • employment status is self-employed
  • occupation is armed forces, legislators, senior officials, refuse workers and other elementary workers.
Differences between countries

Countries’ field of study mismatch rates differ by level of qualifications. The graph below compares New Zealand with the United Kingdom (UK), Korea and Slovenia.

Figure 2: Field of study mismatch rate by highest qualification for selected countries

Some comments on this analysis are:

  • New Zealand, the UK and Korea have similar overall field of study mismatch rates, with Slovenia lower.
  • For NZ and the UK, the higher someone’s qualification, the less likely they are to have a field of study mismatched occupation.
  • Compared to NZ, the UK has a much higher proportion of people with a lower secondary qualification who report General Programmes as their main field of study; therefore their field of study mismatch is set to missing. Despite this, NZ and the UK have similar rates of mismatch for lower secondary qualifications.
  • Korea has much less variability across qualification level compared to NZ and the UK. (Korea has zero rates of mismatch for lower secondary and tertiary less than degree qualifications. This appears to be because it has no qualifications at these internationally defined levels.)
  • Slovenia has a contrasting pattern – an overall lower mismatch rate which is highest for upper secondary qualifications.
  • If we look at field of study mismatch rates for the subgroup with post-school highest qualifications, compared to the total mismatch rate, we see similar differences in New Zealand and the UK. About 40% compared to a half. However, Korea and Slovenia have similar rates of mismatch for those with post school qualifications compared to everyone.
Further comments

Validity of the OECD’s occupation/field of study mapping

The OECD deems a wide range of occupations to be matched under each broad field of study category. Therefore, it includes many field of study mismatched individuals. For example, the field of study category: Humanities, languages and arts maps to the occupations: librarians, archivists, authors, journalists, architects and teachers. It’s a mapping that is more mismatch-not-proved than match-proved.

On the other hand, it’s the same for all countries, so a difference in match rate across countries might still represent a real difference in their education systems or labour markets.

Someone’s occupation may be matched to a secondary field of study for their highest qualification rather than their qualification’s main field of study. Or their occupation could be matched to the main field of study of a qualification lower than their highest qualification. PIAAC collects no data to check this.

Countries have different proportions of workers for whom field of study match is missing

New Zealand PIAAC respondents with highest qualification NCEA Level 1-3 or equivalent have field of study quite spread out across the nine categories. Eighty percent of these people got their highest qualification at school. They appear to have reported their field of study as one of their school subjects – frequently English – which seems reasonable enough. English field of study is mapped to Humanities, languages and arts which has a wide range of occupations that are treated as matched.

A review of other countries’ data shows wide variability in the proportions of people with school qualifications whose field of study is General programmes. For example, in the UK, 33% of these people have field of study General programmes compared with 4% in New Zealand. Several reasons could account for this:
(a) This reflects a real difference in countries’ education systems.
(b) Countries coded open responses to a local classification then mapped back to the nine international responses in a different way from us.
(c) Where countries just used the closed question, General programmes might seem a logical response for a school qualification.

This means that mismatch rates are not very comparable across countries. The small proportions of New Zealanders with only school qualifications who are coded to field of study General programmes is a case in point. It means that a relatively small proportion of our school qualifications people are set to missing, whereas a much larger proportion in the UK are missing.

In addition, countries with lower employment rates, or higher proportions of self-employed people will have greater proportions of their population missing from their field of study match rates.

The four selected countries have large and variable proportions of employed people for whom field of study mismatch is not derived. New Zealand and Slovenia have much smaller rates than Korea, where six in 10 people have field of study mismatch set to missing. The rates are therefore calculated for widely different sized subsets of the employed.

Table 2: Proportions of the employed who are neither field of study (FOS) matched not mismatched
CountryProportion of the employed population wh
are neither FOS matched nor FOS mismatched %
New Zealand 31
England & Northern Ireland 38
Korea 57
Slovenia 29
Analysing qualification mismatch by field of study

This may be more useful for understanding mismatch in the labour market than looking at field of study mismatch according to the current OECD definition. It may also be more useful than attempting a finer- grained ANZSCO/field of study mapping.

Skill mismatch measure (literacy, numeracy or problem solving)

How is skill mismatch defined?

Two questions in the background questionnaire are used:

"Do you feel that you have the skills to cope with more demanding duties than those you are required to perform in your current job?"

"Do you feel you need further training in order to cope well with your present duties?"

People who answered "No" to both questions were considered to be well matched. The minimum and maximum observed proficiency for these people was considered to be the minimum or maximum skills required for the job. To avoid outliers, the 5th and 95th percentile were used to set the minimum and maximum. This range is set by occupation and country, using the first digit level of ISCO.

When a worker's proficiency is above the maximum they are considered overskilled. When a worker's proficiency is below the maximum they are considered underskilled.

How was occupation collected and coded?

Respondents were asked to describe their occupation and what their main tasks were. The survey company coded these responses directly to the lowest levels of both the ISCO and ANZSCO classifications.

Comments

The identification of the 'well matched' group relies on self-completed questions which are open to interpretation. In particular, the questions do not require participants to think about 'how much'. Also, their interpretation of 'skills' in these questions is likely to be quite broad and not specific to literacy and numeracy.

As Table 3 shows, the proportion of 'well matched' workers in each occupation is small, ranging from 2.7% for service and sales workers to 8.5% for elementary workers. A much higher proportion in each occupation answered 'yes' to both questions – which appears to be a contradictory response (from 16 to 32%, or between two and nine times the number answering 'No' to both).

Figure 3 shows that the range of proficiency levels for 'well matched' workers is wide and does not appear to clearly differentiate the skill requirements of different occupations. For example, the literacy skill range for crafts and trades is almost the same as that of elementary workers.

Looking at the range of proficiency for each question combination within occupations, the questions do not appear to show consistently differentiated groups by skill levels. And, in some occupations, these groups are not much different from the overall 5th and 95th percentile for that occupation.

The use of this measure for international comparisons is questionable. It is very dependent on the way in which people understand the questions within countries and occupations. It provides little real difference from simply using the overall skill distribution for each occupation. It is not surprising, therefore, that the amount of differentiation across countries is limited (most countries being in the 10-20% range for total mismatch).

Table 3: Frequency of observations who were 'well matched' and range of literacy and numeracy scores
Note:
  1. associate* = Associate Professionals.
  FrequencyLiteracyNumeracy
No%MinMaxMinMax
1 Managers and professionals 122 7.7% 228 364 207 369
3 Technicians and associate* 36 5.8% 179 355 151 364
4 Clerical 25 5.9% 182 321 171 323
5 Services and sales 20 2.7% 159 318 138 292
7 Craft and trades 23 5.8% 120 310 120 323
8 Plant and machinery 18 7.8% 170 306 146 291
9 Elementary 25 8.5% 112 294 113 281
Figure 3: 5th to 95th percentiles of literacy proficiency across all 4 question combinations and total, for occupation groups

Alternative approach

An alternative approach to this measure would be to simply look at those who are below or above the 5th and 95th (or 10th and 90th) percentiles for their occupations. This would focus on people with unusually high or low skills compared to their occupational peers. It may be possible to do this at level 2 of the classification (with some grouping).

However, this approach would not provide any meaningful international comparisons, or comparison over time. It would only be useful as a way of understanding the characteristics of high and low skilled workers by occupation.

Footnotes

  1. The two classifications are dissimilar. For example, some services and science fields of study are spread across the NZ classification but collected together in Eurostat’s. And the reverse is also true for some categories of management. The New Zealand classification’s broad categories are:
    • Natural and Physical Sciences
    • Information Technology
    • Engineering and Related Technologies
    • Architecture and Building
    • Agriculture, Environmental and Related Studies
    • Health
    • Education
    • Management and Commerce
    • Society and Culture
    • Creative Arts
    • Food, Hospitality and Personal Services
    • Mixed Field Programmes
  2. In New Zealand, we also coded occupation to the Australia and New Zealand Standard Classification of Occupations (ANZSCO).
  3. In the UK, only England and Northern Ireland participated in PIAAC.
  4. For New Zealand, the level of lower secondary schooling for which field of study mismatch is derived is NCEA Level 1. Upper secondary is NCEA Level 2 or 3.
  5. The first level groups for ISCO are:
    1. Managers
    2. Professionals
    3. Technicians and Associate Professionals
    4. Clerical Support Workers
    5. Services and Sales Workers
    6. Skilled Agricultural, Forestry and Fishery Workers
    7. Craft and Related Trades Workers
    8. Plant and Machine Operators and Assemblers
    9. Elementary Occupations
    10. Armed Forces Occupations
      (Due to sample size, groups 6 and 0 were dropped, and 1 and 2 were merged).
  6. The horizontal axis shows, for each occupation group, each combination of yes (Y) and no (N) answers to the two questions about coping with more demanding duties and about needing more training for present duties. The horizontal axis also includes the total (T) for each occupation group.

Contact Us

Survey of Adult Skills
For queries about the Survey of Adult Skills please contact:

Paul Satherley
Senior Research Analyst
Tertiary
Ministry of Education
P O Box 1666
Wellington 6140

Email:   Adult Literacy & Life Skills Mailbox