3. Data and methods

Key Points

• Previous data on field of specialisation has been limited, and in many cases was not able to accurately represent a graduate's field(s) of specialisation, in particular, for more general qualifications such as arts, science and commerce degrees.
• New field of specialisation data has been developed based on a student's course enrolments rather than on their qualification.
• A field of specialisation has been defined as any field successfully studied for the equivalent study load of two or more full-time courses at the final-year level or higher of a qualification.
• For graduates' field of specialisation, the new method currently permits analysis for the years 2002 to 2006 only. Any environmental changes in the last two years, such as any effects from the implementation of recent tertiary sector reforms or any effects from the current economic recession, therefore, will not be reflected.
• For participation (ie enrolments and equivalent full-time students), data on field of study using this new approach is available for the period 2001 to 2008.
• A range of statistical tables on participation and achievement in tertiary education using this new and more detailed field of study data are available on the Ministry of Education's Education Counts website.

Rounding

Because of issues with data quality described below, and because of the nature of the derivation methods used, all field of specialisation counts for students completing qualifications have been rounded to the nearest 10.

Counting multiple fields of study

The methods used allow for more than one specialisation or main field of study. For example, 83 percent of bachelors graduates in 2006 had one specialisation at the broad level of NZSCED, while almost all of the remainder had two. At the narrow level of NZSCED, 69 percent had one specialisation, 29 percent had two, and two percent had more than two. At the detailed level of NZSCED, 60 percent had one specialisation, 33 percent had two, and seven percent had more than two.

All statistics produced in this report count each graduate once for each specialisation they had. Therefore, the sum of rows or columns usually add to more than the stated total.

Definitions

Field of Study

Each qualification and each course are assigned fields of study using the New Zealand Standard Classification of Education (NZSCED). NZSCED has three levels.  There are 12 broad fields, 71 narrow fields and 376 detailed fields. Course-level field of study data has been collected since 2001, when the New Zealand Standard Classification of Education (NZSCED) was first introduced. The classification list is included in Appendix A, while the full classification, including definitions and examples can be found on the Ministry of Education's Education Counts website under the technical information section.¹

Graduate

This report relates to anyone who gained a bachelors degree between 2002 and 2006. Gaining a degree means completing the academic requirements for that degree. It does not necessarily require that the student has ‘graduated’ or that the actual degree has been conferred or awarded. In this true sense then, students in this report may not all be ‘graduates’. However, for convenience, the term ‘graduate’ is used in this report.

Bachelors degree

This report relates to bachelors degrees only, regardless of the length of the degree. They include conjoint and double degrees, but exclude Honours degrees, with the exception of the two degrees; the Bachelor of Engineering (Honours) from the University of Auckland, and the Bachelor of Engineering with Honours from the University of Canterbury, which are included in this report. This report also excludes other bachelors-level qualifications, such as graduate certificates and diplomas, or certificates of proficiency.  Statistical tables covering all levels of study can be found on the Ministry’s Education Counts website.

Domestic

Most of the analysis in this report relates to domestic students. These are New Zealand citizens, New Zealand permanent residents, and any students who were Australian citizens.

Current field of study data availability

As discussed in the introduction, national information on tertiary education participation and achievement by field of study has been very limited. Earlier Ministry of Education statistics, for example, were based on a field of study coding assigned at the qualification level and so in many cases – especially in multi-year qualifications and at higher levels – failed to accurately capture the main subjects studied in the courses taken as part of that qualification. For instance, all Bachelor of Science graduates were shown as graduates in natural and physical sciences because science degrees are assigned to that broad NZSCED field.  This meant that we weren't able to separate out, those in computer science from those in mathematics or those in biological science from those in physical science.

In addition to NZSCED, a separate 'main subject' classification and collection has existed since the early 1990's. This data allows for up to three subject majors to be recorded for every student completing a qualification. Unfortunately this data is not considered suitable for analytical purposes. It is required to be reported only for generic degrees (such as Bachelors of Science, or Bachelors of Arts). As a result, the data is subject to very low response rates (60 percent for degrees, and less than 20 percent for certificates and diplomas. It was felt that for more vocationally-specific qualifications, ie those which don’t require a major to be reported, the qualification-based NZSCED would be sufficient. However, subsequent analysis has shown this not to be true. Further, this 'main subject' collection uses a classification that is different from the current New Zealand standard (NZSCED). While 'main subject' data was not considered for use directly in this project, it was used in the development of new data, as an independent means to assess the quality of the new derived field.

As a result, previous statistics on field of study tended to be reported at the broad level only (12 categories), and had significant shortcomings. Over a third of the fields studied by Bachelor of Science graduates are not in the natural and physical sciences, while over one in ten Bachelor of Arts and Bachelor of Commerce graduates specialised in fields outside of society and culture, and management and commerce respectively. In essence then, existing data collections do not directly provide a robust measure of graduates' field of study. This project investigated whether, indirectly, they could.

Method

A better picture can be seen by looking at the fields of study for the various courses a graduate was enrolled in as part of their qualification, independent of the field or name assigned to the qualification. This approach attempts to assign to each graduate a combination of fields that reflects in a more precise way the depth and breadth of study undertaken.

The method links completed qualification records to individual course enrolments over the duration of a graduate's study towards that qualification. It uses the level of study, field of study, year of study, and study load of each course to determine what best constitutes their main field(s) of study – or specialisation(s). In this way, it is similar in concept to a major. However, the term ‘major’ often has a precise meaning and definition depending on which institution the student is completing at, and these may differ across institutions. While this project does not set out to fairly represent these differences, it does attempt to develop a single basis upon which students’ specialisations could be identified and compared nationally.

The approach assigns as a main field or specialisation any field involving successful study of at least two equivalent full-time courses at a level equivalent to the final-year level or higher of the qualification. The ‘or higher’ condition was included to recognise a significant minority of students who study in a higher-level qualification, but who exit with a lower-level qualification. For example, Masters students who exit with a second Bachelors degree. The final-year level papers do not need to have all been done in the student's last year. Courses are measured in terms of equivalent full-time study load, or EFTS. One EFTS equates to a full-time study load over a full academic year. On average, two full-time courses equate to 0.3 EFTS, or roughly one-third of a full-time full-year's study. Because of the differences in setting full-time course loads across institutions, a threshold of 0.25 EFTS was set beyond which, a field of study was considered as a main field.

In practice, it is not always easy to link the qualification a student completes to all his or her course enrolments back across time. Sometimes the student will take out a different qualification from the one they have enrolled in, through course changes, cross-credits and exemptions, etc. When a student has enrolled in multiple qualifications, it is sometimes difficult to discern which path of learning has led to which qualification. Around 6 percent of students complete more than one qualification at the same provider during a year.

A hierarchy of logic is applied in order to track qualifications back to individual course enrolments. First, the method seeks to find all course enrolments for the last three years in the same qualification as the one completed. Around 75 percent of resulting specialisations can get set in this way. If this fails to produce sufficient EFTS at the required final-year level then the method attempts to supplement the final-year EFTS with all course enrolments at the same level as the qualification completed. The method progressively relaxes which course enrolments it associates with the qualification until a main field can be set. A further 5 percent are set in this way. In a small number of cases, less than 1 percent, no matching enrolments of any kind can be found for a qualification. In these cases, the method resorts to the field of study assigned to the qualification.

After this linking, there are around 20 percent of records which, even though they can be linked back to all course enrolments, have no single field with 0.25 EFTS or more. This situation arises, for example, for certificates of less than 0.25 EFTS in duration. This can also occur when there is no single dominant subject – ie, six courses of 0.16 EFTS each, or for those qualifications for which there are less than 0.25 EFTS of matching course enrolments.

The method can also initially fail to find a specialisation as a result of missing NZSCED values in the course enrolment data. However, non-response levels are very low, just 1 percent in 2001, reducing to less than 0.2 percent in 2006. NZSCED was first introduced in 2001, so is largely missing for 2000. This will affect results slightly for 2002 graduates, as only two years of historical course-based NZSCED data is available to find appropriate final-year level enrolments for this group.

Of the 20 percent with no single field with 0.25 EFTS or more, 90 percent are certificate-level qualifications. A further 5 percent relate to diplomas. For these cases, the qualification-based NZSCED assigned to the certificate or diploma is taken as the student’s main field. The remaining 5 percent of cases relate to degree-level qualifications or higher. For these records, the linked course-based NZSCED with the highest study load (EFTS) is taken.

The method has procedures to recognise and cater for all the main fields for those graduating with conjoint degrees, double majors or where there is otherwise more than one equivalent full-time year of study at the final-year level.

For graduates' field of specialisation, the new method currently permits analysis for the years 2002 to 2006 only. Data for later years will be added as it becomes available. Therefore the trends presented in this report will not reflect any environmental changes in the last two years, such as any effects from the implementation of recent tertiary sector reforms or any effects from the current economic recession. Therefore, care may be needed when applying these past trends to the present or possible future environments. For participation (ie enrolments and equivalent full-time students), however, data on field of study using this new approach is available for the period 2001 to 2008.

Data and method quality

The quality of the resulting fields of specialisation produced by this method, as well as the quality of the coding of fields of study in the source data was assessed extensively during the project using a range of methods. These assessments were also used to refine and improve the quality of the derivation methods.

One of the data quality issues involved the coding of course level. The method relies on correctly identifying courses equivalent to the final-year level or higher of a qualification. For example, for most bachelors degrees, it requires third-year courses (ie, level = 7)² to be identified from first-year (level = 5), or second-year (level = 6) courses.

For some institutions, courses were coded entirely to the level of the degree (eg a level of 7), so it was not directly possible to distinguish first-year courses from final-year courses. A significant evaluation was undertaken of all degree-level course coding and an exercise was undertaken to reassign incorrect course levels to their likely correct levels. This largely used course codes, which in many cases indicate the level of study. For example, ‘STAT101’ is likely to represent a stage one course, or course level of 5. Of those institutions where course level codes were an issue, only one provider did not use a system which included reference to the level or typical year of study. These were sent to the institution and they provided correct levels. Around half a dozen distinct naming structures were able to be discerned for the remaining providers, from which correct course levels were able to be reasonably estimated.

Much of the focus on course level for this project has been on bachelors degrees, and it is felt that the resulting quality at this level is sufficiently robust for aggregate-level statistics to be produced. For postgraduate and non-degree qualifications, some course level errors may still remain. However, it is generally more likely that such qualifications have a narrower range of fields studied over a narrower range of levels than is the case for bachelors degrees.

Results were checked using a number of assessments. The first assessment involved a comparison of results with the existing 'main subjects' collected on qualification completion data, where available. 'Main subject' data is available for about 65 percent of degree level and above qualifications, but only for 10 to 20 percent of certificates and diplomas. Because 'main subject' uses a different classification to NZSCED, a mapping between the two codes was developed to facilitate this comparison.

For those records where 'main subject' was available, this comparison showed only around 70 percent consistency, although at doctorate level it was over 90 percent. While specific patterns of inconsistency enabled improvements to the methods, the matching in fact revealed a range of issues with the existing 'main subject' data.  For example, in the area of teacher education (the most common field of study at bachelors level) providers have been requested to report the 'main subject' as specialist teaching subjects, such as English, maths, geography, etc, instead of the field of teacher education (for teacher supply modelling purposes). Other common sources of inconsistency in 'main subject' occurred through confusion with similar-looking codes. For example, between mechanical engineering ('ENGM') and mineral/mining engineering ('ENME'), or between Finance ('FINA') and Fine Arts ('ARTS'). Main subject fields in the areas of accountancy, economics, business studies, and business administration were often used interchangeably. The results of this comparison check found that in over 98 percent of cases, where 'main subject' was available, the new method provided a better, or the same result as the 'main subject'.

The second assessment compared the results of the new method with the NZSCED assigned to the qualification. The main reason for developing the new method was to move away from the shortcomings associated with using qualification-based NZSCED. However, for shorter qualifications, such as one year certificates, or for those with a more specialised or vocational focus, one might expect a higher level of correspondence.

This comparison showed the following level of correspondence across levels: 85 percent for certificates, 75 percent for diplomas, 55 percent for postgraduate certificates and diplomas, 40 percent to 50 percent for bachelors and masters degrees, and just 2 percent for doctorates.

As multi-year qualifications were expected to have low correspondence, this check was really directed at single-year, or vocationally specific qualifications.  These comparisons highlighted a large range of coding issues, affecting both the quality of coding of NZSCED course enrolments as well as the quality of the NZSCED codes assigned to qualifications. While these did not usually necessitate a change to the methodology, they did require manual correction to the data in order for the method to produce correct results.

Often commonly occurring inconsistencies highlighted issues with the NZSCED classification itself. This included gaps in the classification or areas of the classification where it was ambiguous as to which code to allocate. An example of this is low-level office computing packages, such as the introduction to word processing or spread-sheeting packages, which belong in the broad field of management and commerce, but were often assigned information technology codes. Another area is computer-based arts and design fields, such as multi-media design, web-site design, computer animation, media digitisation, and computer-based design in general. These are relatively recent and growing areas of study. They are not well addressed in NZSCED, resulting in inconsistency in coding of these fields between information technology and creative arts.

In most cases, where the comparisons were inconsistent, the correct coding was readily discernable, as either the course-based NZSCED, or the qualification-based NZSCED code. However, in a few areas, especially where the NZSCED classification is ambiguous, it was less clear which, if any, is the better code.

A third check involved an analysis of outliers. Specifically, this involved checking graduate numbers in fields of specialisation otherwise not expected. For example, bachelors degree graduates in social skills development or office skills. Occurrences of suspected outliers were investigated to assess whether they were real or due to NZSCED miscoding.

A similar check was carried out on trends. Large changes to graduate numbers in certain fields between one year and the next year, may be real, but may also indicate an issue with coding.

The final check involved the use of 'not elsewhere classified' or 'N.E.C.' codes. Most of the narrow fields in NZSCED have a code for fields of study that are genuinely not elsewhere classified. However, in a number of cases, enrolments can get incorrectly assigned here. This can occur by error or with courses that span two or more detailed fields (ie, involve 'mixed' fields). It can also occur where there is otherwise not enough information to assign a more detailed code. For example, some courses are structured as 'research topic' or 'dissertation', where the field of study may vary for different students over several fields of study. Large numbers of graduates in an N.E.C. code may indicate that this code has been used inappropriately, or as a ‘dumping’ code.

N.E.C. codes are present in 10 of the 12 broad fields. Each of these was checked to estimate which graduates could have been reassigned to more specific fields, and which were genuine cases.  The broad fields most affected were health, engineering and information technology. It was often, but not always, possible to reassign miscoded courses back to a more specific detailed field. Some residual effects remain in engineering and health. The major area where there remains an issue is information technology where, the other N.E.C. narrow field 0299 (other information technology) is less likely to represent genuine N.E.C. cases, but rather cases which should have be allocated to 0201 (computer science) or 0203 (information systems).

This issue of miscoding is important when inferring the significance of the size and trends in these ‘other’ codes. In particular when N.E.C. codes at the narrow level also contain genuine cases of emerging fields of study at the detailed level (such as biotechnology).

A single general catch-all code exists in NZSCED for 'mixed field programmes', code '129999'. At the qualification level this code is heavily used, especially for doctorates, and bachelors degrees. At the course level, it is less relevant. One of the benefits of developing course-based fields of study has been the removal of this catch-all code, with graduates now able to be reallocated across more specific and relevant NZSCED fields.

Because of the coding issues described above, these N.E.C. codes have been altered to reflect actual practice. The labels and meanings for all residual N.E.C. codes (ie all codes ending in 99 or 9999) are now labelled to reflect the fact that graduates in these fields include all three cases of response – ie N.E.C, mixed, and not further defined (or N.F.D.). This applies not just to the analysis in this report, but to all tables and statistics produced from the project.