Monitoring Youth Guarantee 2017: Secondary-Tertiary Programmes Publications
This report provides results for young people who started in Secondary-Tertiary Programmes up to 2014, and their outcomes and destinations to 2015 and 2016.
Author(s): David Earle, Tertiary Sector Performance Analysis, Ministry of Education.
Date Published: April 2018
Section 1: Summary
Section 2: Secondary-Tertiary Programmes
Section 3: Monitoring and Evaluation
Section 4: Who Participates?
Section 5: Education Outcomes
Section 6: Destinations
Section 8: Appendix: Methodology
Who participates in Secondary-Tertiary Programmes?
Secondary-Tertiary Programmes were first established in 2011 and fully implemented in 2013. The number of young people in Secondary-Tertiary Programmes has increased from 2,580 in 2012 to 4,190 in 2014.
More males than females participated in each year, although the proportion of females gradually increased. Just under 40% of participants were Māori and 14% were Pasifika in each year. The programmes have generally been established in lower decile schools. As a result, the programmes attract a higher proportions of students with low school achievement and/or at risk of poorer outcomes.
How effective are Secondary-Tertiary Programmes?
Secondary-Tertiary Programmes have been effective in increasing the number of young people who attain NCEA Level 2 or equivalent and providing a pathway to employment.
Participants in the programmes were more likely to remain in education during the period of the programme. The programmes have also engaged a small group of young people who would otherwise have been not in employment, education or training (NEET).
While participants were more likely to achieve NCEA Level 2 or equivalent than young people with a similar background, participants were no more likely to progress to Level 4 and above tertiary education (including through industry training). Secondary-Tertiary Programmes have been particularly effective in providing pathways to full employment. This has resulted in higher average earnings for participants, compared to young people with a similar background.
Following the programme, participants have similar rates for being NEET and/or receiving a benefit as comparable young people. There is some indication that the proportion of participants receiving a benefit three years after the programme was lower than similar young people.
Who are Secondary-Tertiary Programmes more effective for?
Analysis of progression to Level 4 and above by groups of students shows only small or no differences in rates between participants and similar young people. Analysis of progression to employment found that some groups of participants had much higher employment rates than the same groups of young people with similar background characteristics. These included:
- those with no prior NCEA qualification
- those with higher education performance
- those at lower risk of poor outcomes.
There was no additional employment gain for females and Māori.
2. Secondary-Tertiary Programmes
Secondary-Tertiary Programmes, most of which are also known as trades academies, were implemented from 2011, as part of the Youth Guarantee policy. During 2011, 11 programmes were established. This was increased to 22 programmes during 2012.
The Youth Guarantee policy provides opportunities for young people to achieve education success, and progress into further education, training and employment. It supports schools, tertiary education organisations and employers to work together.
Secondary-Tertiary Programmes allow students to remain enrolled with a secondary school, while participating in various forms of education delivered by tertiary education providers. Each has a lead provider, which can be a secondary school, a tertiary education provider or an industry training organisation. There is no single model of provision. Each programme has developed its own approach. The results in this report show the average outcomes across all of the programmes.
3. Monitoring and Evaluation
The purpose of monitoring and evaluating the Youth Guarantee programmes is to understand the extent to which the desired outcomes of the programmes are being met.
The education outcomes of interest are:
- improved retention in school and/or tertiary education (including industry training)
- more students achieving NCEA Level 2, or equivalent
- increased progression to tertiary study at level 4 or higher (including industry training).
The employment outcomes of interest are:
- obtaining sustained employment
- reduced incidence of not being in employment, education or training (NEET)
- reduced incidence of welfare benefit receipt.
This report includes results for young people who started the programme up to 2014. It uses data in the Statistics New Zealand Integrated Data Infrastructure (IDI). At the time of analysis, 2015 was the latest year for which most data was available. Employment data was available up to 2016, so an extra year is included for the employment indicator.
The outcomes are explored for the group of young people who started Secondary-Tertiary Programmes in each year. We look at the effect of the programme for those who participated, compared with young people who had matching characteristics in the year prior to the programme but did not participate in Secondary-Tertiary Programmes (the comparison group). The comparison group were similar to participants in terms of age, gender, ethnicity, education achievement and social background. Many of them also participated in school and/or other tertiary education settings. The matching was undertaken separately for each starting year within each programme, in order to control for changes in the composition of participants in the programmes over time. The participants and the comparison group had the same level of outcome on each measure in the year before the participants started the programme. A full description of the matching is set out in the appendix.
By comparing the outcomes of the participants to those of the matched comparison group, it is possible to estimate how much of the outcome was likely to be due to programme participation rather than the characteristics of the young people.The main area that we remain uncertain about is the effect of selection into the programmes. For example, those participating in the programmes may be more interested in gaining employment rather than going on to further study, than those in the matched comparison group. There is a possibility that further unmeasured factors (not available in the IDI) influenced both the choice to participate in a programme and the outcomes of participants. These factors could include family and peer influence, career and study preferences, as well as personality and other individual attributes. It is not possible to know if including these unmeasured factors would increase or reduce the apparent differences between the participants and the comparison group, or have no effect. However, given the wide range of variables used to match the individuals, we can be reasonably confident that the difference in outcomes, is to a large degree, related to the effect of programme participation.
4. Who Participates?
Age, gender and ethnicity
The number of people starting in Secondary-Tertiary Programmes has increased from 2,580 in 2012, to 4,190 in 2014. The largest age group in each year was 17 year olds. In 2014, around 3% of all 17 year olds in New Zealand started in Secondary-Tertiary Programmes.
Figure 1. Participation in by age
Note: The height of the bars shows starters as a percentage of the population in each age group, and the labels show the number of starters.
The majority of starters in each year were male. The proportion of females has slowly increased from 28% in 2012 to 36% in 2014.
Just under 40% of starters in each year were Māori and around 14% were Pasifika. In 2014, just over 4% of Māori 17 year olds and just over 3.5% of Pasifika 17 year olds started in Secondary-Tertiary Programmes.
Figure 2. Proportion of age groups starting for Māori and Pasifika in 2014
Note: The height of the bars shows participants as a percentage of the population in each age group.
4.1 School achievement
Just under half of people starting in Secondary-Tertiary Programmes had no school qualifications at the time. Around 12 to 15% had already achieved NCEA Level 2 before starting.
Figure 3. Distribution of starters by school-level qualification
Note: The height of the bars shows the proportion of participants with each level of qualification within each year. School level qualifications include qualifications at the same level gained through tertiary education before starting.
Prior education performance provides a measure of how well students achieved in their Level 1 NCEA standards compared with all other students undertaking NCEA Level 1. Across all three years around 60% of starters were in the lowest 30% of prior education performance.
Figure 4. Distribution of starters by prior education performance
Note: The height of the bars shows the proportion of participants with each decile of performance. The lowest 10% of all students by performance is represented by 1, and the highest 10% of all students by performance by 10.
Risk of poor outcomes
A risk score was calculated for each young person. The score was developed using the 1991 birth cohort, and used the characteristics of the young people at age 15 to predict if they had had at least two years of being long-term NEET and not achieved NCEA Level 2 or above by age 20. The same weighting was then applied to subsequent birth cohorts. This provides a consistent risk rating of poor outcomes across birth cohorts.
Across all three years, just over half of the participants were in the highest 30% of risk.
Figure 5. Distribution of starters by risk of poor outcomes
Note: The height of the bars show the proportion of participants with each decile of risk. The 10% of the population with the highest risk is represented by 10, and the 10% of the population with the lowest risk by 1.
5. Education Outcomes
Figure 6. Education outcomes by starting year
Note: Orange lines are the results for the participant group and grey lines are for the comparison group.
Secondary-Tertiary Programmes were effective in improving retention during the year of starting the programme. Nearly all participants remained in education during that year, compared with just over 90% of the comparison group. This result indicates that up to 10% of participants might not have been in education if Secondary-Tertiary Programmes were not available to them.
Young people in Secondary-Tertiary Programmes had higher achievement of NCEA Level 2 or equivalent after starting the programme than the comparison group. Across the years, the results indicate that around 7 to 8% of participants might not have gained NCEA Level 2 or equivalent if these programmes were not available to them.
Education activity of the comparison group
The people in the comparison group are drawn from the population of young people who had not started in Secondary-Tertiary Programmes as of the starting year. A few of them do start in in subsequent years. They are matched on education and other characteristics on the year before starting the programme. This means the participant and comparison groups have the same pattern of education participation before starting the programme.
In each year, 82 to 85% of the comparison group were enrolled in school for at least part of the starting year. A further 10% of the comparison group were enrolled in tertiary education, but not at school, during the year.
Young people starting Secondary-Tertiary Programmes in 2012 and 2013 were less likely to progress to Level 4 and above than other young people with similar characteristics. This includes participation in industry training. The difference was not statistically significant for 2012 starters but was for 2013 starters.
Note: "P" denotes participants; "C" denotes comparison group; "R" denotes the ratio of participants over the comparison group (risk ratio).
NCEA Level 2
Progression to Level 4 or higher
Figure 7. Destinations by starting year
Note: Orange lines are the results for the participant group and grey lines are for the comparison group.
Secondary-Tertiary Programmes have been particularly effective in providing pathways to full employment. For each year, around 35 to 40% of participants were in full employment two years after starting, compared to around 25% of the comparison group.
Secondary-Tertiary Programmes were successful in engaging a small group of young people who might otherwise have been NEET during the period of the programme. Around 6% of the comparison group were NEET during the starting year. However, following the programmes, there were similar proportions of participants and the comparison group who were NEET.
Participants on Secondary-Tertiary Programmes had similar rates of benefit receipt in the years immediately following the programme. However, there is some indication from the 2012 starters that the participants were less likely to receive a benefit three years after the programme. This may be related to the higher proportions in full employment.
"P" denotes participants; "C" denotes comparison group; "R" denotes the ratio of participants over the comparison group (risk ratio). Green shading indicates participants had a better outcome than the comparison group, orange shading indicates that the difference was not statistically significant, and red shading indicates that participants had a poorer outcome than the comparison group
Not in education, employment or training
Receiving a benefit
Figure 8 shows the average annual earnings from wages and salaries for people in Secondary-Tertiary Programmes and the comparison group. It is shown just for years when the person was in employment and not in education, by the number of years in employment since the starting year.
Participants in Secondary-Tertiary Programmes had very similar starting earnings to the comparison group in both full and part employment, with earnings increasing a similar rate with each year of employment. When looking at all people in employment, participants had higher earnings. This is due to the higher proportion in full employment.
Figure 8. Average real earnings by type of employment (in 2016 dollars)
Note: This figure shows earnings for people in employment and not in education. Time period refers to the number of years in employment, while not in education, since the starting year.
7. Who Are Secondary-Tertiary Programmes More Effective For?
This section explores which groups of learners Secondary-Tertiary Programmes are more effective for. The graphs show the proportions enrolled at Level 4 and above within two years of starting the programmes, and the proportions in full employment two years later. These measures are not mutually exclusive.
All results are based on those starting in 2012 and 2013. The rates have been derived from regression models. The graphs show the differences for each subgroup, when all other factors in the model are held constant. So they show if there is an advantage from the programme for that group having taken into account other differences associated with the group. Differences between groups are statistically significant when the error bars are not overlapping.
In general, 17 and 18 year olds are more likely to have enrolled at Level 4 and above or be in full employment two years later than 16 year olds. There was no statistically significant difference in the proportion of participants and the comparison group enrolling at Level 4 and above by age group. However, young people starting the programme at ages 16 and 17 were more likely to be in employment two years later than people of the same age in the comparison group.
Male participants were slightly more likely than female participants to have enrolled at Level 4 and above. Female participants were slightly less likely than females in the comparison group to enrol at Level 4 and above, once other differences were accounted for. Male participants were much more likely to be in full employment two years later than female participants. Employment rates for males was one of the few areas where participants had a significant advantage over the comparison group. There was no difference in employment rates between female participants and females in the comparison group.
Māori participants were less likely to have enrolled at Level 4 and above than non-Māori participants, once other differences were accounted for. They were also less likely to have enrolled at Level 4 and above than Māori in the comparison group. Māori participants were also less likely to be in full employment two years later than non-Māori participants. There was no statistically significant difference in outcomes for the Māori participants and Māori in the comparison group on this measure.
There were no statistically significant differences in enrolment at Level 4 and above between Pasifika and non-Pasifika, or between Pasifika participants and Pasifika in the comparison group. Pasifika were less likely to be in full employment two years later than non-Pasifika, having controlled for other factors. Pasifika participants had a small improvement in employment rates compared to Pasifika in the comparison group, but their rates of employment were still lower than non-Pasifika participants.
Figure 9. Outcomes by age, gender and ethnic group
Note: Orange bars are the results for the participant group and grey bars are for the comparison group. This figure shows the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals.
Figure 10 shows outcomes by the level of NCEA achieved prior to starting Secondary-Tertiary Programmes, controlling for the other differences.
Participants who started with a higher level of NCEA achievement were more likely to go on to enrol at Level 4 and above within two years, than those starting with lower levels of NCEA. However, participants who already had Level 2 NCEA were less likely to enrol at Level 4 and above than people in the comparison group who already had Level 2. For those with less than Level 2 to begin with, there was no significant difference between participants and the comparison group.
Similarly, participants who started with a higher level of NCEA achievement were more likely to be in full employment two years later. At each level of NCEA, participants were more likely to be in employment than similarly qualified young people in the comparison group. The largest relative difference was for those who had no NCEA achievement in the year prior to starting.
Prior education performance shows how well young people did in their NCEA Level 1 assessment standards. Figure 11 shows that higher levels of prior education performance are associated with greater rates of enrolment at Level 4 and above, and slightly lower rates of full employment, once other factors are controlled for. Participants with higher levels of prior education performance were less likely to enrol at Level 4 and above than people in the comparison group with the same level of performance. Participants with higher levels of prior education performance were more likely to be in full employment than people in the comparison group with the same level of performance.
Figure 10. Outcomes by NCEA attainment in year before starting
Figure 11. Outcomes by prior education performance
Note: Orange bars/lines are the results for the participant group and grey bars/lines for the comparison. Both figures shows the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals. Prior education performance is shown by deciles of the overall population. The graph show the range that covers most of the participants in the programme. Those with the lowest performance are represented by 1.
Risk of poor outcomes
A risk score was calculated for each young person based on their likelihood of having been NEET in at least 2 years and not achieving NCEA Level 2 by age 20. This score was calculated from factors not included in this model. Further information is available in the appendix.
A lower risk of poor outcomes is associated with higher rates of enrolment at Level 4 and above. There was no statistically significant difference between the participants and the comparison group on this measure.
Participants who had a lower risk of poor outcomes were more likely to be in employment two years later than people in the comparison group with the same risk level.
Figure 12. Outcomes by risk of poor outcomes
Note: Orange bars/lines are the results for the participant group and grey bars/lines for the comparison. This figure shows the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals. Risk of poor outcomes is shown by deciles of the overall population. The graph shows the range that covers most of the participants in the programme. Those with the highest risk are represented by 1.
Estimated resident population
The data for monitoring the Youth Guarantee programmes is sourced from the Statistics New Zealand's Integrated Data Infrastructure (IDI).
A dataset was constructed of the estimated resident population aged 12 to 24 from 2007 to 2016. This identifies everyone in the IDI who has evidence of presence in New Zealand during these years and ages.
This data set was then turned into a period person dataset with a record for each person from age 12 to 24, irrespective of whether they were in New Zealand in each year. A wide range of indicators were then added to the data set, covering education, travel in and out of NZ, employment, Ministry of Social Development benefit receipt, health and interactions with the justice system.
The outcomes are explored for the group of young people who started in each year. There have been various changes to operational policy and scope from year to year. Looking at annual starting groups controls for these differences.
We look at the effect of the programme for those who participated compared to a matched group of young people who did not participate. Each participating young person was matched to another non-participating young person using a propensity score, based on their characteristics in the year prior to starting. The propensity score is the probability of participating based on a range of demographic, education and social background characteristics. People with the same score have a similar likelihood of being in the programme as well as similar likelihood of education and employment outcomes in the absence of the programme. By comparing the outcomes of the participants and the comparison group, we can estimate how much of the outcome is likely to be due to programme participation rather than the characteristics of the young people
The main area that we remain uncertain about is the effect of selection into the programme. For example, those participating may be more interested in gaining employment rather than going on to further study, than those in the matched comparison group. There is a possibility that further unmeasured factors (not available in the IDI) influenced both the choice to participate and the outcomes of participants. These factors could include family and peer influence, career and study preferences, as well as personality and other individual attributes. It is not possible to know if including these unmeasured factors would increase or reduce the apparent differences between the participants and the comparison group, or have no effect.
Over the period of monitoring Youth Guarantee, the number of variables used to create the comparison group has been increased. The first monitoring report only used education variables for matching. As a wider range of variables have been added, the differences between the participants and comparison groups have generally reduced.
Comparing results across the reports shows that adding more variables to the matching reduced the estimated impact on retention, while increasing the estimated impact on progression to Level 4. It made only small differences to the estimates for the other destinations. Across each report, the direction of the programme impact has remained the same.
Given the wider range of variables now used to match the individuals, we can be reasonably confident that the difference in outcomes is to a large degree related to the effect of participation.
Propensity score models
Propensity score models were run for each starting-year cohort. Running models for each starting year takes account of differences in both the programme and the target group across years. These models identify factors associated with the likelihood of participating. The following variables relating to the year before starting were identified for all of the models:
- being in the Māori, Pasifika, Asian or other ethnic groups (tested separately)
- having been truant, suspended or stood down at school (tested separately)
- having received special education funding
- having been involved in notification to Children and Young Persons
- having been the subject of a police investigation and/or been charged in court
- having been sentenced to prison, home detention or community service
- having received treatment or services for mental health issues, including drug dependency
- having changed addresses since the age of 12
- having changed schools since the age of 12
- been the dependent child of a beneficiary
- number of credits achieved on the New Zealand Qualifications Framework
- prior education performance in NCEA Level 1
- highest NCEA qualification achieved
- NZ Deprivation index of address
- whether enrolled in education
- whether in employment
- have at least one child
- whether main activity was NEET
- whether received a welfare benefit.
Forward selection was used to identify the variables that were statistically significant for each starting year. Interactions of each variable with age were tested, with the exception of the ethnic variables.
The models produced a probability of participation for each young person, which is known as the propensity score. These scores are assigned to both programme participants and non-participants. After assigning the scores, a group of programme participant and non-participant records are identified as being within the "zone of common interest". These are records which range from the 5th percentile of the distribution of participants to the 99th percentile of the distribution of non-participants. This removes the outliers from the dataset, i.e. non-participants with a very low likelihood of participation, and participants with a very high likelihood of participation.
Records for participants and non-participants were then put in random order. Participants were matched to the first non-participant of the same age with a close score. A logistic regression was run to see if there were statistically significant differences in the composition of participants and the matched comparison. While a few statistically significant differences were found, the size of the difference was very small. It was decided further controls on the matching would be unlikely to correct these differences.
There are two improvements in this methodology compared to the last monitoring report. One is to run separate models and matching for each starting cohort. The other is to include a wider range of variables in the models. The results presented in this report will differ from those presented in the earlier monitoring reports as a result of the new methodology.
The report compares the outcome measures for participants with their matched individuals. The participants are grouped by the year in which they started the programme. In the graphs, this year is labelled as year zero.
The 90% confidence intervals for each indicator were calculated using the standard formula for the standard error of proportions.
Only participants and comparison group members who are in New Zealand during the year of observation are included.
Comparing outcomes for subgroups
A further analysis was undertaken comparing the outcomes for subgroups. This analysis used a logistic regression model to estimate the impact of the programme for each outcome. The model included the age, gender, ethnicity, school achievement and risk of poor outcomes as covariates. These covariates were each interacted with being in the participant or control group. The value of the interaction terms provides an estimate of the impact for each subgroup. This allows impacts for subgroups to be examined while holding all other variables constant.
The regression used PROC GENMOD in SAS, with the matched case numbers identified as repeated measures in order to produce GEE estimates of the standard errors.
The following are the definitions of the indicators used in the reports. These measures were agreed at the start of the monitoring process. These definitions are specific to this monitoring project and may differ from other definitions used by the Ministry of Education and the Tertiary Education Commission.
Retention in education
The retention rate is the number of young people who were retained in school and/or tertiary education, as a proportion of the population of interest.
Being retained in education is defined as being enrolled in one or more education programmes for a total period of at least 75 weekdays during a year. This equates to 15 weeks of education. In school terms, it is an enrolment of one and a half terms. In tertiary terms, it falls just below the minimum time required to complete a 40 credit certificate through full-time study. Students who are enrolled for less than 75 days in a year are unlikely to make substantial learning progress during that time.
Enrolment time is counted from the administrative records. For schools, the first and last date of attendance entered on the ENROL database are used. These dates are then adjusted for secondary school holidays. For tertiary education providers, the start and finish dates supplied with course enrolments are used. No adjustments have been made for breaks within courses, as these vary between courses and providers. For industry training, the start and finish dates of training programmes were used.
NCEA Level 2 or equivalent
The NCEA Level 2 or equivalent attainment rate is the number of young people who have attained NCEA Level 2 or equivalent as a proportion of the population of interest. It includes those who attained it during that year and those who attained it in preceding years.
In this report, NCEA Level 2 or equivalent includes:
- being awarded NCEA Level 2
- completing 80 credits on the New Zealand Qualifications Framework, with at least 60 at Level 2 or higher (the requirement for award of NCEA Level 2)
- being awarded an equivalent level in an international school qualification, such as Cambridge or International Baccalaureate
- being awarded another Level 2 New Zealand Qualification Framework qualification; or
- being awarded a Level 3 or higher New Zealand Qualification Framework qualification, including NCEA Level 3.
The option of restricting the definition to exclude Level 2 qualifications on the New Zealand Qualifications Framework that require less than 80 credits to complete was investigated. This was not adopted as the credit information for qualifications completed through tertiary providers is of variable reliability. Qualifications with the same title can have different credit requirements depending on where and when they were offered. It appears to be only a very small number of young people who have attained a Level 2 qualification on the New Zealand Qualification Framework without also meeting the credit requirements for NCEA Level 2 (as set out in the second bullet above).
Enrolling at Level 4 and higher
The Level 4 and higher progression rate is the number of young people who have had an enrolment in a New Zealand Qualifications Framework qualification at Level 4 or higher after leaving school, as a proportion of the population of interest.
Level 4 on the New Zealand Qualifications Framework represents the lowest end of qualification that leads to skilled employment. A Level 3 certificate provides training for specific roles within an area of work and/or preparation for further study. A Level 4 certificate qualifies individuals to work or study in a broad or specialised area.
Enrolments in bachelors degrees and university qualifications are included in this indicator. All these qualifications are on the New Zealand Qualifications Framework and above Level 4. Enrolments through industry training organisations are included, as well as at tertiary education providers.
The indicator counts whether students have ever enrolled in a Level 4 or higher qualification after leaving school. So if a young person enrolled in a tertiary vocational qualification in the first year after a Youth Guarantee programme and then withdrew, that person will still be counted as having enrolled at Level 4 or higher in subsequent years. The indicator does not count young people who enrolled in a course within a Level 4 or higher tertiary qualification while still enrolled at school.
The IDI data provides information on employment spells per person and employer and their taxable income from these spells. It does not provide any information on contract type or hours worked. From the length of the spell and the income, we can identify employees who are being paid very low amounts relative to their period of employment. It is likely they are working part-time or irregular hours. For this study, a threshold was set at having an income pro-rated over a 30-hour week that averages to more or less than the minimum wage.
People can have more than one employer during the year, or multiple spells with the same employer under different employment arrangements. This means they can have some spells that fall below the threshold and some above. Furthermore, the length of these spells can vary. For example, they could have a short spell above the threshold and a longer spell below the threshold.
To identify people who are more fully engaged in employment, they were counted as being in full employment in the year if they had spells above the threshold that added up to at least 182 days (six months) in a year. All other people with employment were counted as having part employment for that year.
Earnings is the total gross annual earnings from wages and salaries, as recorded by Inland Revenue. It does not include benefits, Accident Compensation Corporation payments, self-employment or other sources of income. Earnings have been adjusted to 2016 dollars using the consumers price index.
Not in employment, education or training (NEET)
The NEET measure used in this report looks at the total number of days each person is in education, employment, NEET or overseas during a year. The activity with the largest number of days is assigned as the main activity.
Benefit receipt is calculated from the benefit spell data produced by the Benefit Dynamics Project. This provides exact start and end dates for receiving income from the main benefits. It excludes superannuation and retirement benefits. Only primary beneficiaries have been included in the analysis in this paper. Partners and spouses of beneficiaries have not been included.
In New Zealand or overseas
Young people are counted as being overseas during a specified year if they spend more than 9 months of that year out of the country. Once they are counted as overseas, they are counted as being back in New Zealand if they spend 9 months or more of a subsequent year in the country.
The age at 31 December in the year of starting the programme.
Ethnicity has been identified from the source ranked ethnicity table in the IDI.
Highest level of NCEA
The highest level of NCEA achieved in the year prior to starting the programme. The level includes equivalent qualifications, on a similar basis to NCEA Level 2 equivalence discussed above.
Prior education performance
Using NCEA Level 1 results, it is possible to calculate a performance score for each student based on the proportion of assessment standards with results of not achieved, achieved, merit or excellence relative to their peers. This provides a performance score for each student for each level of NCEA that they have attempted achievement standards in.
The scores for students who undertook no achievement standards were imputed using background characteristics that are predictive of the score. This included students with no credits at Level 1 or who were only assessed on unit standards.
In the reports on effectiveness by groups of participants, the score is presented as quartiles of the distribution for participants.
Risk of poor outcomes
A model was developed to predict the risk of young people having at least two years NEET and no attaining NCEA Level 2 or equivalent by age 20. The model used the following factors observed at age 15:
- had been truant * had changed schools since age 12
- had been stood down * had been suspended
- had received special education funding
- had been dependent child of a beneficiary * had a Children and Young Persons notification
- decile of school attended at 15 * whether in NZ at age 15
- NZ Deprivation Index of address at 15 * whether changed address since age 12.
Ethnicity and school achievement were deliberately excluded from this model so that the measure could be used in regressions alongside these factors.
The model was calibrated for the 1991 birth cohort and then same model weightings applied across subsequent birth cohorts. This means that the risk calculations are constant across cohorts and not influenced by prevailing changes in school achievement or labour market conditions.
In the reports on effectiveness by group of participants, the score was converted to deciles for the total population.
All standard errors presented in the reports have been calculated using the standard formulas for proportions and risk ratios. These formulas provide a close approximation to the real standard error.
All graphs display the 90% confidence intervals. Where these intervals do not overlap there is approximately a 95% or higher chance that the results are statistically significantly different.