Monitoring Youth Guarantee 2017: Youth Guarantee Fees-Free places Publications
Publication Details
This report provides results for young people who started in Youth Guarantee Fees-Free places up to 2014, and their outcomes and destinations to 2015 and 2016.
Author(s): David Earle, Tertiary Sector Performance Analysis, Ministry of Education.
Date Published: April 2018
Page Contents:
1. Summary
Who participates in Youth Guarantee Fees-Free places?
Youth Guarantee Fees-Free places were established in 2010. The number of young people starting Fees-Free has increased each year, from 1,930 in 2010 to 9,000 in 2014.
Youth Training was amalgated with Youth Guarantee Fees-Free places in 2012, resulting in shifts in the types of providers and programmes. Following the amalgamation, q larger proportion of starters in Fees-Free Places were in private training establishments, at Level 2, and/or in non-trades programmes, as well as a greater proportion of learners entering the programme with a high risk of poor outcomes. The proportion of Māori starters increased from 35 to 50%.
Slightly more males than females started in Fees-Free places each year. The proportion of Pasifika starters has remained constant at around 20% each year. Around 60% of participants in each year had no school qualification prior to entry.
How effective are Youth Guarantee Fees-Free places?
Youth Guarantee Fees-Free places have been effective in keeping more young people in education during the period of the programme. The programme has also engaged some young people who would otherwise have been not in employment, education or training (NEET).
Fees-free participants were more likely to achieve NCEA Level 2 or equivalent than young people with a similar background. However, across all years, only around half of starters passed most of their courses, although the proportion failing most of their courses has decreased.
Fees-free participants were no more likely to progress to Level 4 and above tertiary education, or to gain full employment, than young people with a similar background. There is no evidence of any earnings premium from participation in the programme for those in employment.
Following the programme, participants were more likely to be NEET and/or receiving a benefit than young people with a similar background.
Who are Youth Guarantee Fees-Free places more effective for?
Analysis of employment and progression to Level 4 and above by groups of young people shows:
- Māori and Pasifika participants are no more likely to be in full employment or progress to higher level study than Māori and Pasifika with a similar background.
- Participants who had low or no school qualifications before they started, had lower education performance at school and/or had higher risk of poor outcomes were no more likely to progress to Level 4 and above or to full employment than similar young people.
- Participants who had a Level 2 qualification before they started, had higher education performance at school and/or had lower risk of poor outcomes were less likely to progress to Level 4 and above and more likely to progress to full employment than similar young people.
2. Youth Guarantee Fees-Free Places
Youth Guarantee Fees-Free places were implemented in 2010, as the first part of the Youth Guarantee policy. The Youth Guarantee policy provides opportunities for young people to achieve education success, and progress into further education, training and employment. It supports schools, tertiary education organisations and employers to work together.
Youth Guarantee Fees-Free provides full-time, fees-free tertiary study at New Zealand Qualifications Framework Levels 1 to 3 for 15 to 19 year olds who have left school with low or no qualifications. It is intended to re-engage young people in education and provide a pathway into further study, training and employment.
The programme was initially targeted to 16 to 17 year olds. The number of places and providers were expanded in 2012 by transferring funding from the Youth Training programme. The Youth Training programme had been established in 1999 and was aimed at providing training for employment and further education for 16 and 17 year olds who had become disengaged from education. The programme focused on providing short foundation skills courses. By moving the funding to Youth Guarantee Fees-Free places, a greater emphasis was put on providing programmes aimed at improving education outcomes, and pathways towards higher-level qualifications.
From 2012, 15 year olds with an early leaving exemption were eligible to participate in Youth Guarantee Fees-Free places. From 2014, the age eligibility was further widened to include 18 and 19 year olds.
Youth Guarantee Fees-Free places have been implemented in different ways by different providers. Some providers offer dedicated and tailored programmes for funded learners, others offer places within existing programmes, and many offer a mix of both. The results in this report show the average outcomes across all of the providers and programmes.
3. Monitoring and Evaluation
The purpose of monitoring and evaluating the Youth Guarantee programmes is to understand the extent to which the desired outcomes are being met.
The education outcomes of interest are:
- improved retention in school and/or tertiary education (including industry training)
- more students achieving NCEA Level 2, or equivalent
- increased progression to tertiary study at Level 4 or higher (including industry training).
The employment outcomes of interest are:
- obtaining sustained employment
- reduced incidence of not being in employment, education or training (NEET)
- reduced incidence of welfare benefit receipt.
This report includes results for young people who started the programme up to 2014. It uses data in the Statistics New Zealand Integrated Data Infrastructure (IDI). At the time of analysis, 2015 was the latest year for which most data was available. Employment data was available up to 2016, so an extra year is included for the employment indicator.
The outcomes are explored for the group of young people who started Youth Guarantee Fees-Free places in each year. We look at the effect of the programme for those who participated, compared with young people who had matching characteristics in the year prior to the programme but did not participate in Fees-Free places (the comparison group). The comparison group were similar to participants in terms of age, gender, ethnicity, education achievement and social background. Many of them also participated in school and/or other tertiary education settings. The matching was undertaken separately for each starting year within each programme, in order to control for changes in the composition of participants in the programmes over time. The participants and the comparison group had the same level of outcome on each measure in the year before the participants started the programme. A full description of the matching is set out in the appendix.
By comparing the outcomes of the participants to those of the comparison group, it is possible to estimate how much of the outcome was likely to be due to programme participation rather than the characteristics of the young people.The main area that we remain uncertain about is the effect of selection into the programme. For example, those participating in the programme may be more interested in gaining employment rather than going on to further study, than those in the comparison group. There is a possibility that further unmeasured factors (not available in the IDI) influenced both the choice to participate in a programme and the outcomes of participants. These factors could include family and peer influence, career and study preferences, as well as personality and other individual attributes. It is not possible to know if including these unmeasured factors would increase or reduce the apparent differences between the participants and the comparison group, or have no effect. However, given the wide range of variables used to match the individuals, we can be reasonably confident that the difference in outcomes is, to a large degree, related to the effect of programme participation.
4. Who Participates?
Age, gender and ethnicity
The number of people starting Youth Guarantee Fees-Free places has increased from 1,930 in 2010 to 9,000 in 2014. The majority of starters were aged 17 or 18 at the end of the year they started. In 2013 and 2014, around 5 to 6% of all 17 year olds in New Zealand and 4% of all 18 year olds started in Fees-Free places.From 2012, 15 year olds with an early leaving exemption were eligible to participate. Less than 100 have started in each year. From 2014, the age range for entry to the programme was extended to include 18 and 19 year olds. In that year, 22% of starters were aged over 18. Around 2% of 19 year olds and 1% of 20 year olds started in 20141.
Figure 1. Participation by age
Note: The height of the bars shows starters as a percentage of the population in each age group, and the labels show the number of starters.
In each year, there have been slightly more males than females starting. The proportion of female starters has increased from 41% in 2010 to 46% in 2014.
In 2010 and 2011, around 35% of starters were Māori. This increased to 50% in 2012, following the merging with Youth Training, and it reduced to 45% in 2014. In 2014, 10% of Māori 17 year olds and 7% of Māori 18 year olds started in Fees-Free places. This is nearly twice the rate of the total population. The proportion of starters who are Pasifika has been constant since 2010 at just under 20%. In 2014, 7% of Pasifika 17 year olds and 6% of 18 year olds started in Fees-Free places.
Figure 2. Proportion of age groups starting for Māori and Pasifika in 2014
Note: The height of the bars shows participants as a percentage of the population in each age group.
School achievement
From 2010 to 2014, around 60% of starters had no qualifications in the year prior to starting. In 2012 and 2013, 14% of starters had already achieved NCEA Level 2 and above. In 2014, 27% of starters had already achieved NCEA Level 2 and above and 7% had already achieved NCEA Level 3.
Figure 3. Distribution of starters by school-level qualification
Note: The height of the bars shows the proportion of participants with each level of qualification within each year. School-level qualifications include qualifications at the same level gained through tertiary education before starting.
Prior education performance provides a measure of how well students achieved in their Level 1 NCEA standards compared with all other students undertaking NCEA Level 1. Across all years, around 30 to 35% of starters were in the lowest 10% of performance and around 20% in the second lowest 10%. However, between 8 and 15% of starters in each year had above average performance (in the top 50% of all students).
Figure 4. Distribution of starters by prior education performance
Note: The height of the bars shows the proportion of starters with each decile of performance. The lowest 10% of all students by performance are represented by 1 and the highest 10% of all students by performance by 10.
Risk of poor outcomes
A risk score was calculated for each young person. The score was developed on the 1991 birth cohort and used the characteristics of the young people at age 15 to predict if they had had at least two years of being long-term NEET and not achieved NCEA Level 2 or above by age 20. The same weighting was then applied to subsequent birth cohorts. This provides a consistent risk rating of poor outcomes across birth cohorts.In 2010 and 2011, around 25% of starters were in the 10% of the population with the highest risk score. Following the amalgamation with Youth Training, over 40% of starters were in this highest risk group, reducing to 34% in 2014. In each year, a further 20% were in the second-highest risk group. However, in 2014 18% of starters were in the half of the population with lowest risk.
Figure 5. Distribution of starters by risk of poor outcomes
Note: The height of the bars shows the proportion of starters with each decile of risk. The 10% of the population with the highest risk is represented by 10 and the 10% of the population with the lowest risk by 1.
Provider and programme characteristics
This section looks at the characteristics of the providers and programmes that participants were enrolled in. It is based on the last Youth Guarantee Fees-Free programme that each person was enrolled in and is shown by starting year. The 2012 results are omitted due to inconsistencies around the data collection for that year.
The results contrast the composition of the programmes before the amalgamation with Youth Training (2010 and 2011) and afterwards (2013 and 2014). The graphs show the proportions by year. It should be kept in mind that the actual numbers of starters has increased substantially across years, so the numbers of students in each category may be increased even if the proportions have decreased.
Fees-free places are delivered by polytechnics, private training establishments and one wānanga. Figure 6 shows the distribution of participants between polytechnics and private training establishments. The wānanga has been included with the polytechnics due to confidentiality requirements of the IDI. In 2010 and 2011, more than two-thirds of starters were in polytechnics. In 2013 and 2014, over two-thirds of starters were in private training establishments. This is partly a result of the amalgamation with Youth Training, which was predominantly delivered through private training establishments.
In 2010 and 2011 nearly half of Fees-Free starters finished in a Level 3 programme. Some of these participants will have started in a lower level and moved to a higher level programme while in Fees-Free places. In 2013 and 2014, less than a third of starters finished in a Level 3 programme.
Programmes in Fees-Free places can be broadly grouped into trades, non-trades and mixed field programmes. Mixed field programmes cover employment skills. In 2010 and 2011, just under half of starters finished in trades programmes. In 2013 and 2014, less than a third of participants finished in trades programmes and around a third finished in mixed field programmes.
Across the period less than half of starters completed most or all of the courses they enrolled in. The proportion who failed most of their courses has decreased from 28% to 17%.
Nearly all Fees-Free participants were enrolled for at least half of an equivalent full-time year of study. Around half were enrolled for more than one full-time year. In 2013 and 2014 there was an increase in the proportion who were enrolled for more than one and a half years of equivalent full-time study, compared with 2010 and 2011.
Figure 6. Distribution of starters by provider and programme characteristics
Classification of 'Subject'
Youth Guarantee Fees-Free participants have been classified into trades and non-trades using the Tertiary Education Commission's rules for funding. Trades participants are identified as having a majority of course enrolments across their entire programme in course classifications related to trades. Most of these finish in a programme in engineering, building, agriculture or hospitability and personal services. Others are classified as non-trades. Most of these students finish in a programme in management and commerce, society and culture or mixed field. Non-trades participants who completed in a mixed field programme have been classified separately. Mixed field programmes are general programmes related to employment and life skills.
5. Education Outcomes
Figure 7. Education outcomes by starting year
Note: Orange lines are the results for the participant group and grey lines are for the comparison group.
Youth Guarantee Fees-Free places were effective in keeping more young people in education during the period of the programme. Nearly all participants remained in education during their starting year, compared with only 70 to 80% of the comparison group. The size of this effect was largest in 2014, when the age range for Fees-Free places was increased. This result indicates that 20 to 30% of participants might not have been in education if Fees-Free places were not available to them.
Young people in Fees-Free places had higher achievement of NCEA Level 2 or equivalent after starting the programme than the comparison group. Across the years, the results indicate that around 15 to 20% of participants might not have gained NCEA Level 2 or equivalent if Fees-Free places were not available to them.
The NCEA Level 2 attainment rate, and differences between participants and the comparison group, dipped in 2012. This was a result of the merger with Youth Training. Both the rate and the difference has increased since then.Young people starting Fees-Free places in 2010 were more likely than the comparison group to enrol in tertiary education at Level 4 or above within two years of starting the programme. For 2011 starters there was no significant difference between participants and the comparison group. The rates for 2012 starters were not statistically significant at two years, and by three years the rates for participants were lower than for the comparison group. For 2013 starters, the rates were lower for participants after two years.
Note: "P" denotes participants; "C" denotes comparison group; "R" denotes the ratio of participants over the comparison group (risk ratio). Green shading indicates participants had a better outcome than the comparison group, orange shading indicates that the difference was not statistically significant, and red indicates that participants had a poorer outcome than the comparison group. | |||||||||
Year | Retention | NCEA Level 2 | Progression to Level 4 or higher | ||||||
---|---|---|---|---|---|---|---|---|---|
P | C | R | P | C | R | P | C | R | |
2010 | 99% | 81% | 1.23 | 68% | 51% | 1.34 | 34% | 29% | 1.18 |
2011 | 100% | 81% | 1.24 | 75% | 60% | 1.24 | 35% | 33% | 1.05 |
2012 | 97% | 75% | 1.29 | 61% | 53% | 1.17 | 25% | 26% | 0.94 |
2013 | 99% | 78% | 1.28 | 68% | 55% | 1.24 | 25% | 28% | 0.90 |
2014 | 99% | 71% | 1.40 | 76% | 60% | 1.26 |
Education activity of the comparison group
The people in the comparison group are drawn from the population of young people who had not started in Youth Guarantee Fees-Free places as of the starting year. A few of them do start in Fees-Free places in subsequent years. They are matched on education and other characteristics in the year before starting the programme. This means the participant and comparison groups have the same pattern of education participation before starting the programme.
For 2010 to 2013 starters, 70% of the comparison group were enrolled in school for at least part of the starting year. This reduced to 60% in 2014, reflecting the wider age range. This compares with 45% of the participants having a school enrolment in the starting year up to 2013 and 34% in 2014.
The proportion of the comparison group in tertiary education only during the starting year has varied. It was around 15% in 2010 and 2011, 10% in 2012 and 2013 and 16% in 2014.
6. Destinations
Figure 8. Destinations by starting year
Note: Orange lines are the results for the participant group and grey lines are for the comparison group.
Young people who started in Youth Guarantee Fees-Free places in 2010 and 2011 were more likely than the comparison group to be in full employment two years after starting the programme. However, this result was not statistically significant for 2010 starters due to the smaller numbers. By four to five years after starting the programme there was no difference in employment rates for these starting cohorts.
Young people who started in Fees-Free places from 2012 to 2014 were no more likely to be in employment than the comparison group. By three years or more after the programme they were slightly less likely to be in employment.
Fees-Free places were successful in engaging a group of young people who might otherwise have been NEET during the period of the programme. During each starting year, around 15 to 20% of the comparison group were NEET. However, by two years after starting the programme participants were more likely to be NEET than the comparison group.
Young people in Fees-Free places were more likely than the comparison group to receive a welfare benefit during the starting year of the programme. The majority of the young people receiving benefits during the starting year, started in the year they turned 18 and therefore were entitled to a standard benefit. Young people who are receiving benefits can be required to attend training. This may explain the higher rate of benefit receipt for participants in the starting year. The increased proportion receiving benefits in 2014 reflects the increased age range of the programme.
In the years following the programme, a higher proportion of participants than the comparison group received a benefit. For the 2010 and 2011 starters there are signs of these rates converging after 3 or 4 years after starting the programme.
Note: "P" denotes participants; "C" denotes comparison group; "R" denotes the ratio of participants over the comparison group (risk ratio). Green shading indicates participants had a better outcome than the comparison group, orange shading indicates that the difference was not statistically significant, and red shading indicates that participants had a poorer outcome than the comparison group | |||||||||
Year | Full employment | Not in education, employment or training | Receiving a benefit | ||||||
---|---|---|---|---|---|---|---|---|---|
P | C | R | P | C | R | P | C | R | |
2010 | 31% | 28% | 1.10 | 29% | 26% | 1.14 | 47% | 40% | 1.17 |
2011 | 36% | 31% | 1.15 | 29% | 24% | 1.25 | 41% | 34% | 1.21 |
2012 | 27% | 29% | 0.93 | 35% | 30% | 1.16 | 49% | 37% | 1.31 |
2013 | 26% | 28% | 0.95 | 36% | 29% | 1.23 | 48% | 35% | 1.38 |
2014 | 29% | 31% | 0.96 |
Figure 9 shows the average annual earnings from wages and salaries for people in Fees-Free places and for those in the comparison group. It is shown just for the years when the person was in employment and not in education, by the number of years in employment since the starting year.Participants in Fees-Free places had very similar starting earnings to the comparison group across the range of employment statuses. Their earnings continued to increase at the same rate as those of the comparison group. This suggests that Fees-Free places do not provide any earnings premiums for participants, compared to other people with similar characteristics.
Figure 9. Average real earnings by type of employment (in 2016 dollars)
Note: Shows earnings for people in employment and not in education. Time period refers to the number of years in employment, while not in education, since the starting year.
7. Who are Youth Guarantee Fees-Free places more effective for?
This section explores which groups of learners Youth Guarantee Fees-Free places are more effective for. The graphs show the proportions enrolled at Level 4 and above within two years of starting the programme, and the proportions in full employment two years later. These measures are not mutually exclusive.All results are based on those who started in 2012 and 2013. The rates have been derived from regression models. The graphs show the differences for each subgroup, when all other factors in the model are held constant. So they show if there is an advantage from the programme for that group having taken into account other differences associated with the group. Differences between groups are statistically significant when the error bars are not overlapping.
Student characteristics
In general, 17 and 18 year olds are more likely to have enrolled at Level 4 and above or be in full employment two years later than 16 year olds. There was no statistically significant difference in the proportion of participants and the comparison group enrolling at Level 4 and above by age group. However, young people starting the programme at age 16 were more likely to be in employment two years later than people of the same age in the comparison group.
Males and female participants had similar rates of enrolment at Level 4 and above. Male participants were slightly less likely to have enrolled at Level 4 and above than males in the comparison group. Male participants had much higher employment rates than female participants. The employment rates of male participants were slightly higher than those of males in the comparison group.
Māori participants were less likely to enrol at Level 4 and above than non-Māori participants, even after controlling for other factors. There was no significant difference between Māori participants and Māori in the comparison group on this measure. Māori participants were also less likely to be in employment two years later than non-Māori participants, even after controlling for other factors. There was no significant difference between Māori participants and Māori in the comparison group on this measure. However, non-Māori participants were more likely to be in employment than non-Māori in the comparison group.
There were no statistically significant differences in enrolment at Level 4 and above between Pasifika and non-Pasifika, or between Pasifika participants and Pasifika in the comparison group. Pasifika participants were less likely to be employed two years later than non-Pasifika participants, even after controlling for other factors. There was no significant difference between Pasifika participants and Pasifika in the comparison group on this measure.
Figure 10. Outcomes by age, gender and ethnic group
Note: Orange bars are the results for the participant group and grey bars for the comparison group. This figure shows the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals.
School achievement
Figure 11 shows outcomes by the level of NCEA achieved prior to starting Fees-Free places. Those with higher levels of prior achievement were more likely to go into employment and/or enrol at Level 4 and above. However, participants with Level 2 prior to starting were less likely to enrol at Level 4 and above than similarly qualified people in the comparison group, and more likely to be in full employment.
Prior education performance shows how well young people did in their NCEA Level 1 assessment standards. Figure 12 shows that higher levels of prior education performance are associated with greater rates of enrolment at Level 4 and above, and lower rates of full employment, once other factors are controlled for. Participants with higher levels of prior education performance were less likely to enrol at Level 4 and above than people in the comparison group with the same level of performance. Participants with higher levels of prior education performance were more likely to be in full employment than people in the comparison group with the same level of performance.
Figure 11. Outcomes by NCEA attainment in year before starting
Figure 12. Outcomes by prior education performance
Note: Orange bars/lines are the results for the participant group and grey bars/lines for the comparison. Both figures show the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals. Prior education performance is shown by deciles of the overall population. The graph shows the range that covers most of the participants in the programme. Those with the lowest performance are indicated by 1.
Risk of poor outcomes
A risk score was calculated for each young person based on their likelihood of having been NEET in at least 2 years and not achieving NCEA Level 2 by age 20. This score was calculated from factors not included in this model. Further information is available in the appendix.
Having a lower risk of poor outcomes is associated with higher rates of enrolment at Level 4 and above. Participants with a lower risk of poor outcomes were slightly less likely to be enrolled at Level 4 and above than those in the comparison group with similar levels of risk.
The relationship between the risk of poor outcomes and full employment is not as strong as for enrolment at Level 4 and above. Participants who had a lower risk of poor outcomes were more likely to be in employment two years later than people in the comparison group with the same risk level.
Figure 13. Outcomes by risk of poor outcomes
Note: Orange bars/lines are the results for the participant group and grey bars/lines for the comparison. This figure shows the results of logistic regressions controlling for age, gender, Māori, Pasifika, prior education performance and risk of poor outcomes. Results are shown where other variables in the model are set their average or modal values. The error bars show the 90% confidence intervals. Risk of poor outcomes is shown by deciles of the overall population. The graph shows the range that covers most of the participants in the programme. 10 indicates those with the highest performance
Provider and programme characteristics
The comparison group method cannot be used to provide valid measures of the different impacts by provider and programme characteristics. This is because it is not possible to associate participants to people in the comparison group who have the same education experience. The option of matching to similar students in other tertiary education was investigated. However, the majority of tertiary students at foundation level at ages 16 to 18 are in Youth Guarantee Fees-Free places and those who are not are doing quite different types of study.
In order to examine the effect of provider and programme characteristics for Fees-Free, regression models were run just for Fees-Free participants. The models compared the outcomes within Fees-Free, having controlled for the differences in background characteristics and the other provider and programme characteristics.
There was no statistically significant difference in enrolment or employment rates by provider type.
Students who finished in a Level 3 programme were more likely to enrol at Level 4 and above, and less likely to be in full employment two years later, than those who finished in a Level 1 programme.
There was no difference in enrolment rates at Level 4 and above across subject areas. Those who finished in a trades programme were more likely to be in full employment than other participants.
Participants who completed most of their courses were much more likely to enrol at Level 4 and above than other participants. There was no statistically significant difference in employment rates by course completion rate.
Employment and enrolment rates were similar for all participants who studied for up to one and a half equivalent full-time years. Those who studied for more than one and a half years had significantly lower employment rates and possibly lower enrolment rates. However, the reason that these students remained longer in Fees-Free may have been their difficulty in finding employment or lack of preparation for higher-level study.
Figure 14. Expected probability of progression to Level 4 and above and full employment for Youth Guarantee Fees-Free participants
Note: These are the results of logistic regression, controlling for age, gender, Māori, Pasifika, prior education performance, risk of poor outcomes and provider and programme characteristics. In each graph, the other variables in the model are set to their average or modal value. The error bars show the 90% confidence intervals.
Appendix: Methodology
Estimated resident population
The data for monitoring the Youth Guarantee programmes is sourced from the Statistics New Zealand's Integrated Data Infrastructure (IDI).
A dataset was constructed of the estimated resident population aged 12 to 24 from 2007 to 2016. This identifies everyone in the IDI who has evidence of presence in New Zealand during these years and ages.
This data set was then turned into a period person dataset with a record for each person from age 12 to 24, irrespective of whether they were in New Zealand in each year. A wide range of indicators were then added to the data set, covering education, travel in and out of NZ, employment, Ministry of Social Development benefit receipt, health and interactions with the justice system.
Matched comparisons
The outcomes are explored for the group of young people who started in each year. There have been various changes to operational policy and scope from year to year. Looking at annual starting groups controls for these differences.
We look at the effect of the programme for those who participated compared to a matched group of young people who did not participate. Each participating young person was matched to another non-participating young person using a propensity score, based on their characteristics in the year prior to starting. The propensity score is the probability of participating based on a range of demographic, education and social background characteristics. People with the same score have a similar likelihood of being in the programme as well as similar likelihood of education and employment outcomes in the absence of the programme. By comparing the outcomes of the participants and the comparison group, we can estimate how much of the outcome is likely to be due to programme participation rather than the characteristics of the young people
The main area that we remain uncertain about is the effect of selection into the programme. For example, those participating may be more interested in gaining employment rather than going on to further study, than those in the matched comparison group. There is a possibility that further unmeasured factors (not available in the IDI) influenced both the choice to participate and the outcomes of participants. These factors could include family and peer influence, career and study preferences, as well as personality and other individual attributes. It is not possible to know if including these unmeasured factors would increase or reduce the apparent differences between the participants and the comparison group, or have no effect.
Over the period of monitoring Youth Guarantee, the number of variables used to create the comparison group has been increased. The first monitoring report only used education variables for matching. As a wider range of variables have been added, the differences between the participants and comparison groups have generally reduced.
Comparing results across the reports shows that adding more variables to the matching reduced the estimated impact on retention, while increasing the estimated impact on progression to Level 4. It made only small differences to the estimates for the other destinations. Across each report, the direction of the programme impact has remained the same.
Given the wider range of variables now used to match the individuals, we can be reasonably confident that the difference in outcomes is to a large degree related to the effect of participation.
Propensity score models
Propensity score models were run for each starting-year cohort. Running models for each starting year takes account of differences in both the programme and the target group across years. These models identify factors associated with the likelihood of participating. The following variables relating to the year before starting were identified for all of the models:
- age
- being in the Māori, Pasifika, Asian or other ethnic groups (tested separately)
- having been truant, suspended or stood down at school (tested separately)
- having received special education funding
- having been involved in notification to Children and Young Persons
- having been the subject of a police investigation and/or been charged in court
- having been sentenced to prison, home detention or community service
- having received treatment or services for mental health issues, including drug dependency
- having changed addresses since the age of 12
- having changed schools since the age of 12
- been the dependent child of a beneficiary
- number of credits achieved on the New Zealand Qualifications Framework
- prior education performance in NCEA Level 1
- highest NCEA qualification achieved
- NZ Deprivation index of address
- whether enrolled in education
- whether in employment
- have at least one child
- whether main activity was NEET
- whether received a welfare benefit.
Forward selection was used to identify the variables that were statistically significant for each starting year. Interactions of each variable with age were tested, with the exception of the ethnic variables.
The models produced a probability of participation for each young person, which is known as the propensity score. These scores are assigned to both programme participants and non-participants. After assigning the scores, a group of programme participant and non-participant records are identified as being within the "zone of common interest". These are records which range from the 5th percentile of the distribution of participants to the 99th percentile of the distribution of non-participants. This removes the outliers from the dataset, i.e. non-participants with a very low likelihood of participation, and participants with a very high likelihood of participation.
Records for participants and non-participants were then put in random order. Participants were matched to the first non-participant of the same age with a close score. A logistic regression was run to see if there were statistically significant differences in the composition of participants and the matched comparison. While a few statistically significant differences were found, the size of the difference was very small. It was decided further controls on the matching would be unlikely to correct these differences.
There are two improvements in this methodology compared to the last monitoring report. One is to run separate models and matching for each starting cohort. The other is to include a wider range of variables in the models. The results presented in this report will differ from those presented in the earlier monitoring reports as a result of the new methodology.
Comparing outcomes
The report compares the outcome measures for participants with their matched individuals. The participants are grouped by the year in which they started the programme. In the graphs, this year is labelled as year zero.
The 90% confidence intervals for each indicator were calculated using the standard formula for the standard error of proportions.
Only participants and comparison group members who are in New Zealand during the year of observation are included.
Comparing outcomes for subgroups
A further analysis was undertaken comparing the outcomes for subgroups. This analysis used a logistic regression model to estimate the impact of the programme for each outcome. The model included the age, gender, ethnicity, school achievement and risk of poor outcomes as covariates. These covariates were each interacted with being in the participant or control group. The value of the interaction terms provides an estimate of the impact for each subgroup. This allows impacts for subgroups to be examined while holding all other variables constant. The regression used PROC GENMOD in SAS, with the matched case numbers identified as repeated measures in order to produce GEE estimates of the standard errors.
A separate model was undertaken to look at the impact of programme and provider characteristics on outcomes. The matched comparison group was not used as people in this group did not share the same programme and provider characteristics. This analysis shows the impact of programme and provider characteristics on outcomes, having controlled for differences in age, gender, sex, school achievement and risk of poor outcomes. This regression used PROC SURVEYLOGISTIC in SAS with provider and qualification set as clusters.
Indicator definitions
The following are the definitions of the indicators used in the reports. These measures were agreed at the start of the monitoring process. These definitions are specific to this monitoring project and may differ from other definitions used by the Ministry of Education and the Tertiary Education Commission.
Retention in education
The retention rate is the number of young people who were retained in school and/or tertiary education, as a proportion of the population of interest.
Being retained in education is defined as being enrolled in one or more education programmes for a total period of at least 75 weekdays during a year. This equates to 15 weeks of education. In school terms, it is an enrolment of one and a half terms. In tertiary terms, it falls just below the minimum time required to complete a 40 credit certificate through full-time study. Students who are enrolled for less than 75 days in a year are unlikely to make substantial learning progress during that time.
Enrolment time is counted from the administrative records. For schools, the first and last date of attendance entered on the ENROL database are used. These dates are then adjusted for secondary school holidays. For tertiary education providers, the start and finish dates supplied with course enrolments are used. No adjustments have been made for breaks within courses, as these vary between courses and providers. For industry training, the start and finish dates of training programmes were used.
NCEA Level 2 or equivalent
The NCEA Level 2 or equivalent attainment rate is the number of young people who have attained NCEA Level 2 or equivalent as a proportion of the population of interest. It includes those who attained it during that year and those who attained it in preceding years.
In this report, NCEA Level 2 or equivalent includes:
- being awarded NCEA Level 2
- completing 80 credits on the New Zealand Qualifications Framework, with at least 60 at Level 2 or higher (the requirement for award of NCEA Level 2)
- being awarded an equivalent level in an international school qualification, such as Cambridge or International Baccalaureate
- being awarded another Level 2 New Zealand Qualification Framework qualification; or
- being awarded a Level 3 or higher New Zealand Qualification Framework qualification, including NCEA Level 3.
The option of restricting the definition to exclude Level 2 qualifications on the New Zealand Qualifications Framework that require less than 80 credits to complete was investigated. This was not adopted as the credit information for qualifications completed through tertiary providers is of variable reliability. Qualifications with the same title can have different credit requirements depending on where and when they were offered. It appears to be only a very small number of young people who have attained a Level 2 qualification on the New Zealand Qualification Framework without also meeting the credit requirements for NCEA Level 2 (as set out in the second bullet above).
Enrolling at Level 4 and higher
The Level 4 and higher progression rate is the number of young people who have had an enrolment in a New Zealand Qualifications Framework qualification at Level 4 or higher after leaving school, as a proportion of the population of interest.
Level 4 on the New Zealand Qualifications Framework represents the lowest end of qualification that leads to skilled employment. A Level 3 certificate provides training for specific roles within an area of work and/or preparation for further study. A Level 4 certificate qualifies individuals to work or study in a broad or specialised area.
Enrolments in bachelors degrees and university qualifications are included in this indicator. All these qualifications are on the New Zealand Qualifications Framework and above Level 4. Enrolments through industry training organisations are included, as well as at tertiary education providers.
The indicator counts whether students have ever enrolled in a Level 4 or higher qualification after leaving school. So if a young person enrolled in a tertiary vocational qualification in the first year after a Youth Guarantee programme and then withdrew, that person will still be counted as having enrolled at Level 4 or higher in subsequent years. The indicator does not count young people who enrolled in a course within a Level 4 or higher tertiary qualification while still enrolled at school.
Employment
The IDI data provides information on employment spells per person and employer and their taxable income from these spells. It does not provide any information on contract type or hours worked. From the length of the spell and the income, we can identify employees who are being paid very low amounts relative to their period of employment. It is likely they are working part-time or irregular hours. For this study, a threshold was set at having an income pro-rated over a 30-hour week that averages to more or less than the minimum wage.
People can have more than one employer during the year, or multiple spells with the same employer under different employment arrangements. This means they can have some spells that fall below the threshold and some above. Furthermore, the length of these spells can vary. For example, they could have a short spell above the threshold and a longer spell below the threshold.
To identify people who are more fully engaged in employment, they were counted as being in full employment in the year if they had spells above the threshold that added up to at least 182 days (six months) in a year. All other people with employment were counted as having part employment for that year.
Earnings
Earnings is the total gross annual earnings from wages and salaries, as recorded by Inland Revenue. It does not include benefits, Accident Compensation Corporation payments, self-employment or other sources of income. Earnings have been adjusted to 2016 dollars using the consumers price index.
Not in employment, education or training (NEET)
The NEET measure used in this report looks at the total number of days each person is in education, employment, NEET or overseas during a year. The activity with the largest number of days is assigned as the main activity.
Benefit receipt
Benefit receipt is calculated from the benefit spell data produced by the Benefit Dynamics Project. This provides exact start and end dates for receiving income from the main benefits. It excludes superannuation and retirement benefits. Only primary beneficiaries have been included in the analysis in this paper. Partners and spouses of beneficiaries have not been included.
In New Zealand or overseas
Young people are counted as being overseas during a specified year if they spend more than 9 months of that year out of the country. Once they are counted as overseas, they are counted as being back in New Zealand if they spend 9 months or more of a subsequent year in the country.
Subgroup definitions
Age
The age at 31 December in the year of starting the programme.
Ethnicity
Ethnicity has been identified from the source ranked ethnicity table in the IDI.
Highest level of NCEA
The highest level of NCEA achieved in the year prior to starting the programme. The level includes equivalent qualifications, on a similar basis to NCEA Level 2 equivalence discussed above.
Prior education performance
Using NCEA Level 1 results, it is possible to calculate a performance score for each student based on the proportion of assessment standards with results of not achieved, achieved, merit or excellence relative to their peers. This provides a performance score for each student for each level of NCEA that they have attempted achievement standards in.
The scores for students who undertook no achievement standards were imputed using background characteristics that are predictive of the score. This included students with no credits at Level 1 or who were only assessed on unit standards.
In the reports on effectiveness by groups of participants, the score is presented as quartiles of the distribution for participants.
Risk of poor outcomes
A model was developed to predict the risk of young people having at least two years NEET and no attaining NCEA Level 2 or equivalent by age 20. The model used the following factors observed at age 15:
- gender
- had been truant * had changed schools since age 12
- had been stood down * had been suspended
- had received special education funding
- had been dependent child of a beneficiary * had a Children and Young Persons notification
- decile of school attended at 15 * whether in NZ at age 15
- NZ Deprivation Index of address at 15 * whether changed address since age 12.
Ethnicity and school achievement were deliberately excluded from this model so that the measure could be used in regressions alongside these factors.
The model was calibrated for the 1991 birth cohort and then same model weightings applied across subsequent birth cohorts. This means that the risk calculations are constant across cohorts and not influenced by prevailing changes in school achievement or labour market conditions.
In the reports on effectiveness by group of participants, the score was converted to deciles for the total population.
Confidence intervals
All standard errors presented in the reports have been calculated using the standard formulas for proportions and risk ratios. These formulas provide a close approximation to the real standard error.
All graphs display the 90% confidence intervals. Where these intervals do not overlap there is approximately a 95% or higher chance that the results are statistically significantly different.
Footnote
- In this report, age refers to age at 31 December. So those who were 20 at the end of the year would have started while they were still 19.
Navigation
Contact Us
For more information about the content on this webpage, please email the: Tertiary Mailbox