I enjoy school now - Outcomes from the Check and Connect trials in New Zealand
This evaluation describes outcomes from the trial of the Check and Connect programme. This programme aims to counter secondary student disengagement with school before it results in dropping out of school, or leaving without qualifications.
Author(s): Cathy Wylie and Rachel Felgate, New Zealand Councils for Educational Research
Date Published: December 2016
This evaluation describes outcomes from the trial of the Check and Connect programme. This programme aims to counter secondary student disengagement with school before it results in dropping out of school, or leaving without qualifications. This trial was funded as part of the Prime Minister's Youth Mental Health strategy. It is part of the Ministry of Education's Positive Behaviour for Learning strategy, with the Ministry of Social Development (MSD) contracting non-government organisations (NGOs) to provide mentor teams, using the Check and Connect programme within the Ministry of Education's national framework.
The Check and Connect programme originated in the United States of America (USA), and had good evidence of success. It provides each student with a mentor who 'checks in' weekly on their attendance, behaviour and achievement, and works with the student to equip them with the skills, confidence, and attitudes that enable them to see the purpose of school and find it worthwhile. The programme emphasises the need for ongoing persistence, with mentors working for two years or more with students in the USA.
This report provides the findings from an evaluation of the trial of the programme in New Zealand, focusing on the changes made by students with at least one year's experience on Check and Connect, at the main three trial sites, by the end of 2015. At this stage in the trial, few of the 138 students covered by the evaluation had the full two years or more participation in Check and Connect recommended in the USA.
This evaluation of the New Zealand trial of the Check and Connect programme in three sites shows that most participating students and their mentors noted positive gains for the students.
Student and mentor reports provide the strongest evidence available for this evaluation. Check and Connect is a customised programme, and consistent information on student needs and school attendance and performance levels before they embarked on Check and Connect to compare with where they were at the end of 2015 was not available. Where we could compare 'start' data from student and teacher surveys with data at the end of 2015, the numbers are small, and we are therefore cautious about patterns deriving from those comparisons, particularly the data from teachers, which covers only 16 students.
We certainly see clear trends where we look at some factors that should influence student gains from the programme, such as the length of time on Check and Connect. Because of the small numbers involved, these trends were not statistically significant. In the few comparisons where we could establish statistical significance, in relation to ethnicity and gender, the size of the difference tended to be very large (eg, 20 percent compared with 0 percent), and may be an artefact of the size of the particular group. We would not pay the statistically significant comparisons more attention than the clear trends we found.
Factors that were linked to student gains on Check and Connect were
- length of participation on the programme
- their relationship with their mentor
- their mentor's use of Check and Connect strategies
- having a mentor who gave high ratings to their training, and support within their team, and
- having a mentor with a good partnership with the school.
We look now at what we found for each of our five evaluation questions.
Student change through their participation in Check and Connect
Evidence from students and mentors shows that the Check and Connect trial has resulted in many positive changes on the outcomes sought, for just over a third of the students, and some or a few changes for almost all the rest of the participants. It was also evident from the interviews and comments on the student survey that some students who would come into our (just) 'a few changes' category had made some fundamental shifts, such as increasing their effort and having better academic results, or gaining better ways to deal with things that upset them, getting more support at school, and putting more effort into their schoolwork.
Just under three-quarters of the students responding to the end of 2015 survey said they put more effort into their schoolwork and had better results. Level 1 NCEA achievement was higher than might have been expected, given that most of those for whom we had the information were performing below the general curriculum level for their year when they were nominated. Around two-thirds reported a better sense of their own strengths, and better ways of dealing with things that used to upset them. Student accounts and comments provide some vivid testimony to the "life-changing" difference it could make to their engagement and achievement in school, and their capacity to live more purposefully, confidently, and contentedly in and out of school.
Length on Check and Connect
Gains for students were higher for those students who had had between 18 months to two years on the programme, supporting the original programme design. Changing adolescent habits, feelings and behaviours, especially around low school engagement and performance, is not quick work. The Check and Connect students often presented quite complex needs and circumstances.
Because the students who had had 18 months to two years Check and Connect participation were in the minority, it is likely that the pattern of changes we found for the students in the trial sites is an underestimate of the gains for students participating for the recommended length of time on the programme.
Female students were more likely than male students to make many changes through their Check and Connect participation. It may be that this is linked to the pull of peers that was evident in student reports of their friendships, with males more likely to have friends who wagged school and wanted them to do stupid things, or who didn't talk about future hopes and plans, and less likely to talk with their friends about what they were learning in school.
Māori students were more likely to be seen by their mentors to make many changes than Pasifika students, but this pattern was not evident in the student reports.
Reason why students were nominated for Check and Connect
The information we have on why individual students were nominated for Check and Connect is limited, and we could find no clear patterns in terms of whether the main reason was truancy, lateness, (non)completion of schoolwork, behaviour, or whether they had experienced a recent major trauma in their life.
The mentor's role in student change
The gains for students that we found have been achieved through the quality of the mentors' work with students. Students who made many changes rated more highly than others their relationship with their mentor and the Check and Connect strategies. These strategies included weekly meetings, setting goals, having feedback from their mentor about their attendance and checking in on attendance.
Mentors found that building the foundations for students to trust them was critical, and that sometimes meant focusing on students' out-of-school relationships and challenges rather than school attendance at the start of their work with them, or during it. Where students trusted their mentors, they felt able to be their full selves with them, often giving the mentor more information to work with than students' teachers or parents might have.
Students generally felt their cultural identity was supported in their work with their mentor.
Additional support was gained by mentors for 34 percent of the students, mostly from their schools, and also through their Check and Connect team and its contacts. As well, mentors had helped 36 percent of the students with direct tutoring or homework.
Mentor support and training
Mentors found the general Check and Connect approach worked with almost all their students. In terms of their training, they were most positive about the use of cognitive behaviour theory, check-in and feedback about school attendance, and motivational interviewing. They were least convinced about the value of the Pathways map. The initial training had come too soon for almost half of the mentors, since they were not then working with students, and just under a third thought it had not been clear about the approach. A third did not think the guidelines they had for the programme were clear. They wanted more ongoing training and support.
Generally the mentors' work has been well supported by having mentors work in teams together, within NGOs that can link to other supports for young people such as mental health, health, and government agencies.
Students were more positive about their work with their mentors when they had mentors who rated their training and support within their team most highly, indicating the value of good training and teams.
It was essential to the students that the mentor was someone they could trust, someone with whom they could speak freely, and someone who behaved and responded differently from the teachers and school counsellors who had 'growled' at them or did not have enough time to put the student at the centre of their attention on a regular basis. Since this evaluation does not include Check and Connect provided by schools, we cannot definitively tell from this evaluation whether it is essential that Check and Connect mentors are not school staff. We think the fact that they were not played a useful part in their ability to win the confidence of students who were not comfortable in their schools, and for the mentors to be able to put individual students first.
On the whole Māori and Pasifika students had similar views of their relationship with their mentor. There were some differences as to how useful some of the Check and Connect strategies were, with more positive views from Māori students about checking in and feedback from their mentor about their attendance, and goal setting.
Matching of student and mentor cultural identity
Trial sites endeavoured to match student and mentor cultural identity, and could do so for around a third of the Māori students, and half of the Pasifika students (within the broad category of Pasifika). Whether student and mentor were matched this way did not show any links with student gains from their Check and Connect work, or with whether they felt their mentor respected their cultural identity.
While female students were more likely to make many changes as a result of their Check and Connect participation, student views of their relationship with their mentor and the work they did with them did not differ for males and females. Mentors of more female than male students identified the relationship they had with the family, data monitoring, persistence plus, capacity building and engagement with the family as elements of most use to the students. Mentors of more male than female students identified goal setting and plans as of most use. Female students had more additional support that their mentor had gained for them, through their school or NGO site, and more had been helped with direct tutoring or homework.
All but three percent of the students had the same gender as their mentor, so we were not able to analyse whether gender-matching was linked to any differences in student experiences and gains.
Prior mentor experience in working with young people
All the mentors brought experience in working with young people to their Check and Connect role, including recreational, community and school-based programmes, with some having worked as social workers, in health, mental health and drug and alcohol addiction programmes, and other kinds of support and coaching. Because all mentors had some previous experience, and often several different kinds of experience, and because there were only 17 mentors in total, it was not possible to analyse whether mentors needed prior experience working with young people to be effective in their Check and Connect role, or whether one kind of experience might be more useful for this role than others.
The school role in student change
Students whose mentors were most positive about their work with the school were more likely to have made many changes than others.
Mentors worked closely with their students' schools. Most were positive about their relationships with these schools, though issues around accessing behaviour, attendance and NCEA data were experienced, and sometimes difficulty in finding a private space to meet with students. Overall, after some teething problems, the partnership with schools was working well for more than half the students, though there was still more variability than desirable.
It was difficult for the Hawke's Bay site to work as well with Wairoa College as they could with their local schools.
School champions and principals were positive about Check and Connect. School champions had some issues with the time it took to get together the information from teachers and others to nominate a student. Most were satisfied with the contact with and information from the mentors working with their school's students.
Students whose mentors were most positive about their work with the school were more likely to have made many changes than others.
The family role in student change
Parent engagement is an essential component of the Check and Connect framework. Overall, mentors had established good communication and relations and support with students' goals with around 50 to 60 percent of the students' families. While the links with gains for students shows that these are important, it is also evident that they are not essential for students to make many changes. Much depends on what changes are needed, and where the family is positioned, as well as what support is also available from school and friends.
Partnerships that enable change
A tripartite partnership between mentor, school and family was not essential for a student to make meaningful gains through their Check and Connect participation, although where they existed, and all could communicate well and support a student's goals, they were effective.
Like any trial of a new programme, the Check and Connect trial has not been without its teething problems. While some of these have been overcome, there are still a number of issues that need Ministry of Education national level attention and coherence across districts to strengthen the programme.
Our recommendations are that the Ministry of Education:
- leads a national approach to ongoing learning from the Check and Connect sites, through providing regular opportunities to share effective mentor-school partnerships (with the input of schools), what has worked to gain the trust of hard to reach students, and work with them to effect change, effective mentor-parent partnerships, and what has worked with students with different issues at nomination
- provides ongoing training on topics identified by mentors, and on key strategies; initially this could include finding out more about the use of the Pathways map and any issues around it
- simplifies site reporting, and uses the site reports for ongoing national learning to support the efficacy of Check and Connect and inform other work on student engagement
- ensures that schools provide mentors with the electronic access to student data from the school that they need – including NCEA credits as well as attendance and behaviour
- reviews the inclusion of Wairoa College in the Hawke's Bay site, since it did seem more difficult for mentors to meet student needs where a school is physically distant from their base
- provides a valid, and simple online monitoring of student progress that is consistent across the sites and that can also be pulled together efficiently to provide periodic national pictures of the effectiveness of Check and Connect, and fed into ongoing learning shared with the sites to keep improving practice –
- this is likely to include some work on how schools record attendance and behaviour data to identify differences, and decide what data or categorisation of data is essential, so that analysis is realistic
- provides more coherence and more efficiency for the sites and for itself by having the national responsibility both for the Check and Connect programme and for contracting the NGOs who provide it
- makes Check and Connect available to Year 7 and 8 students to stay with them into Year 9, to provide a bridge for students showing disengagement before they get to secondary school, and support them to make academic progress so that they are better placed for the demands of the secondary curriculum
- continues to provide Check and Connect through the existing NGO sites, making the most of the expertise now present at those sites
- uses the site expertise and the results of this evaluation to decide on provision in other sites that could benefit from Check and Connect
- investigates its interest in seeing if Check and Connect could be provided as well within school sites by:
- finding out more about the Otaki College use of Check and Connect to see what gains have been made for students, how the team has worked, with a particular focus on the student-mentor relationship and use of strategies, and the particular context of this school's approach to behaviour and student support
- finding out more about the use of Check and Connect in two West Auckland secondary schools, using RTLBs
- discussing with the schools who have worked with Check and Connect mentors in the existing trial sites their views on the viability and sustainability of providing Check and Connect mentoring in-house, and the support that would be needed if they thought this would be desirable
- discussing with the two Check and Connect programme people visiting Wellington on 12 August what lies behind effective school provision of Check and Connect in the USA, and analysing their identification of enabling factors in terms of the New Zealand school context
- discussing with primary schools and intermediates the support they think they would need to provide Check and Connect in-house if it is extended to Years 7 and 8.
For more publication-related information, please email the: Information Officer Mailbox