Skills and education: How well do educational qualifications measure skills?

Appendix A: Technical notes

Scores in the Adult Literacy and Life Skills Survey

Satherley and Lawes (2007) gives an account of the scoring approach in the ALL survey:

To each individual, and for each of the domains, a score from zero to 500 is assigned. Zero indicates extremely low proficiency, and 500 extremely high. In addition, based on this score, one of five ‘cognitive levels’ is assigned. These cognitive levels are used in national and international comparison, essentially as a benchmark. The following list provides descriptions of typical tasks associated with each cognitive level.

Level 1 (Scores 0–225):
Tasks at this level require the ability to read simple documents, accomplish literal information-matching with no distractions, and perform simple one-step calculations.

Level 2 (Scores 226–275):
This level includes tasks that demand the capacity to search a document and filter out some simple distracting information, achieve low-level inferences, and execute one- or two-step calculations and estimations.

Level 3 (Scores 276–325):
Typical tasks at level 3 involve more complex information-filtering, sometimes requiring inferences and the facility to manipulate mathematical symbols, perhaps in several stages.

Level 4 (Scores 326–375):
A level 4 task might demand the integration of information from a long passage, the use of more complex inferences and the completion of multiple-step calculations requiring some reasoning.

Level 5 (Scores 376–500):
Level 5 tasks incorporate the capability to make high-level inferences or syntheses, use specialised knowledge, filter out multiple distractors, and to understand and use abstract mathematical ideas with justification.

OECD and Statistics Canada (2000) provide the following characterisation of Level 3:

Level 3 is considered a suitable minimum for coping with the demands of everyday life and work in a complex, advanced society. It denotes roughly the skill level required for successful secondary school completion and college entry.

Using variables from the ALL dataset

Highest qualification

The highest qualification was derived from questions A4B and A4C of the ALL questionnaire which asked respectively “What is the highest level of primary or secondary school that you have ever completed?” and “What is the highest level of formal education or training that you have ever completed?” The New Zealand coded responses were used and grouped in two different ways, as follows:
 

Table 2: Detailed classification of educational qualifications

Detailed    Grouping	Responses (from A4B & A4C)	Frequency	Weighted Frequency
Up to Year 10	Up to Standard 3/Year 5 Standard 4/Year 6 Form 1, 2 or 3/Year 7, 8 or 9 Form 4/Year 10	838	265,997
Year 11	Form 5/Year 11	1,151	402,329
Year 12 or 13	Form 6 or 7/Year 12 or 13	1,202	483,386
Level 1, 2 or 3 certificate	Level 1, 2 or 3 certificate	952	337,523
Level 4 certificate	Level 4 certificate	630	233,745
Level 5, 6 and 7 certificate/diploma	Level 5, 6 and 7 certificate or diploma	880	310,251
Bachelors degree	Bachelors degree Professional degree	845	348,860
Postgraduate	Bachelors degree with honours or    postgraduate diploma Masters degree Doctorate	629	251,587
Not stated		4	765

Table 3: Broad classification of educational qualifications

Broad Grouping	Responses (from A4C)	Frequency	Weighted    Frequency
Lower secondary	Up to Form 5/Year 11	1,989	668,326
Upper secondary	Form 6 or 7/Year 12 or 13 Level 1, 2 or 3 certificate	2,154	820,909
Tertiary	Level 4 certificate Level 5, 6 and 7 certificate or diploma Bachelors degree Professional degree Bachelors degree with honours or postgraduate diploma Masters degree Doctorate	2,984	1,144,443
Not stated		4	765

 

Immigrant status

Immigrant status was derived from ALL question A1, which asked “Were you born in New Zealand?”, and question A2, which was addressed to those who answered “No” to A1 and asked “In what year did you first immigrate to New Zealand?” No further information relating to immigration is available from the survey. All those born overseas are thus classified as ‘immigrants’ whether they are skilled migrants or business investors or their families, or have been accepted under humanitarian (eg refugee quota), family reunification or other policies.

Survey respondents were selected from people living in permanent private dwellings who had “no usual place of residence elsewhere”, thus excluding short-term visitors. However, some temporary migrants may have been included in the survey sample.

Immigrants have been subdivided by date of first immigration so that those who had been in New Zealand five years or less are treated as a distinct group of ‘recent’ immigrants.¹²

 

Table 4: Classification of immigrant status

Group	Responses (from A1 and A2)	Frequency	Weighted  Frequency
Non-immigrant	Born in New Zealand	5,336	1,924,363
Established immigrant	Born overseas, immigrated between 1946 and 2000	1,260	451,602
Recent immigrant	Born overseas, immigrated between 2001 and 2006	530	256,820
Unclassified	Born overseas but date of arrival not    stated	1	892

First language

First language was derived from ALL questions B1A and B1B which asked “What is the language that you first learned at home in childhood and still understand?” Respondents could nominate two languages if they were learned at precisely the same time. For a binary classification (English or other) respondents who provided English as a response to one of these questions (5910 for B1A and 20 for B1B) were categorised as having English as a first language, while the rest were categorised as having English as a second or other language.

Classification of first language as English, Asian or Other was based solely on responses to B1A. Respondents were classified as having an Asian first language if their responses to B1A were coded as one of the following: Chinese, Korean, Japanese, Vietnamese, Malay, Indonesian, Hindi, Punjabi, Tamil, other Indian language. The residual category of ‘Other languages’ may include some Asian languages, such as Thai and Tagalog, which were not specifically identified in the coding of the survey responses.

Industry

The ALL background questionnaire contains a series of questions (D18 to D43) about the most recent job that the respondent had worked at. This is termed the “main job”. Industry and occupation all relate to this job.

Industry is coded in the ALL data set using the International Standard Industry Classification. For this project, the international classification has been concorded to the Australia and New Zealand Standard Industry Classification 2006. For the purposes of the analysis, several industries in the top level of the classification have been grouped together, as follows:

 

Table 5: Classification of industries

Group	ANZSIC06	Frequency	Weighted Frequency
Construction	E Construction	437	163,856
Retail and wholesale trade	F Wholesale trade G Retail trade H Accommodation and food services	1,362	545,153
Community, social and personal services	M Professional, scientific and technical services N Administrative and support services O Public administration and safety P Education and training Q Health care and social assistance R Art and recreation services S Other services	2,597	945,014
Other industries	A Agriculture, forestry and fishing B Mining C Manufacturing D Electricity, gas, water and waste services I Transport, postal and warehousing J Information media and telecommunications K Finance and insurance services L Rental, hiring and real estate services	2,108	767,858
Not stated		19	7,878

Occupation

Occupation was coded to the International Standard Classification of Occupation in the ALL data set. This has been concorded to the New Zealand Standard Classification of Occupation 1999. The level one classification was used in this project, as follows:

 

Table 6: Classification of occupations

Group	NZSCO99	Frequency	Weighted Frequency
Managers	Administrators and managers	620	231,878
Professionals	Professionals	1,076	417,491
Technicians	Technicians and associate professionals	774	296,175
Clerks	Clerks	988	353,255
Other occupations	Service and sales workers Agricultural workers Trades workers Plant and machinery operators and assemblers Labourers Military	3,053	1,126,254
Not specified		12	4,707

Definitions of level in the New Zealand Register of Quality Assured Qualifications

The New Zealand Register of Quality Assured Qualifications establishes 10 levels of qualifications and qualification titles that can be used at each level. For each qualification there is a statement of learning outcomes that includes what the whole qualification represents in terms of the application of knowledge, understanding, skills and attitudes, as well as the components of the qualification.

Each qualification has a credit value that represents the amount of learning and assessment that is typically required to achieve the qualification.

Level 3 is the level required for achievement in year 13 (the final year) of secondary school. Level 7 is the level required for completion of a bachelors degree and level 8 for a bachelors degree with honours or a postgraduate diploma, while level 9 is the level for completion of a masters degree and level 10 a doctoral degree.

The levels are described in the table below, drawn from the website of the New Zealand Qualifications Authority:

                                                                                                                                                                   

Table 7: Description of levels of qualifications in the New Zealand Register of Quality Assured Qualifications

LEVEL	PROCESS	LEARNING    DEMAND
1	Carry out processes that: are limited in range are repetitive and familiar are employed within closely defined contexts	Employing: recall a narrow range of knowledge and cognitive skills no generation of new ideas
2	Carry out processes that: are moderate in range are established and familiar offer a clear choice of routine responses	Employing: basic operational knowledge readily available information known solutions to familiar problems little generation of new ideas
3	Carry out processes that: require a range of well developed skills offer a significant choice of procedures are employed within a range of familiar contexts	Employing: some relevant theoretical knowledge interpretation of available information discretion and judgement a range of known responses to familiar problems
4	Carry out processes that: require a wide range of technical or scholastic skills offer a considerable choice of procedures are employed in a variety of familiar and unfamiliar contexts	Employing: a broad knowledge base incorporating some theoretical concepts analytical interpretation of information informed judgement a range of sometimes innovative responses to concrete but often unfamiliar problems
5	Carry out processes that: require a wide range of specialised technical or scholastic skills involve a wide choice of standard and non-standard procedures are employed in a variety of routine and non-routine contexts	Employing: a broad knowledge base with substantial depth in some areas analytical interpretation of a wide range of data the determination of appropriate methods and procedures in response to a range of concrete problems with some theoretical elements
6	Carry out processes that: require a command of wide-ranging highly specialised technical or scholastic skills involve a wide choice of standard and non-standard procedures, often in non-standard combinations are employed in highly variable routine and non-routine contexts	Employing: specialised knowledge with depth in more than one area the analysis, reformatting and evaluation of a wide range of information the formulation of appropriate responses to resolve both concrete and abstract problems
7	Carry out processes that: require a command of highly specialised technical or scholastic and basic research    skills across a major discipline involve the full range of procedures in a major discipline are applied in complex, variable and specialised contexts	Requiring: knowledge of a major discipline with areas of specialisation in depth the analysis, transformation and evaluation of abstract data and concepts the creation of appropriate responses to resolve given or contextual abstract problems
8	Involves skills and knowledge that enable a learner to: provide a systematic and coherent account of the key principles of a subject area; and undertake self-directed study, research and scholarship in a subject area, demonstrating intellectual independence, analytic rigour and sound communication
9	Involves knowledge and skills that enable a learner to: demonstrate mastery of a subject area plan and carry out - to internationally recognised standards - an original scholarship or research project. Demonstrated by: the completion of a substantial research paper, dissertation or in some cases, a series of papers.
10	Involves knowledge and skill that enable a learner to: Provide an original contribution to knowledge through research or scholarship, as judged by independent experts, applying international standards.

 

 

Footnote

12. Five years is the time generally considered as representing the initial period of settlement, according to International Migration, Settlement and Employment Dynamics Research, Department of Labour.