Executive summary

A typical IYA-P programme involves 14 sessions and an IYA-T programme involves six sessions. Sessions are delivered face-to-face by two trained and accredited facilitators. Caregiver and teacher participants are those who look after or teach a child who is on the autism spectrum or who demonstrates characteristics of autism. The majority of children receiving support through the IYA programme attend an early learning service.

A focus of IYA programmes is equipping participants with the knowledge, skills and attitudes to support children’s social interactions, emotional regulation, communication and school readiness skills. The goals of the programme are achieved through the use of a range of learning-based activities. These include the use of group discussions, reflections on vignettes, role play and homework activities.

The approaches and theoretical underpinnings of the IYA-P/T programmes are sourced from child development, attachment and social learning theories. For example, it is well known that children on the autism spectrum are likely to present with communication and behaviour challenges, and social learning theory suggests that children’s behaviour and development is heavily influenced by the adults they spend time and interact with. Therefore, supporting adults to develop new skills and strategies to interact and communicate can, in turn, support change and development in a child on the autism spectrum. On completion of an IYA-P/T programme, parent and teacher participants are typically better equipped to provide a supportive and enabling environment – at home, early learning services or schools and the wider community. A more comprehensive overview of the IYA programmes is provided in Appendix A.

Evaluation questions

The overarching aim of this evaluation was to assess the effectiveness and impact of the IYA-P/T programmes. This was evaluated by examining the extent to which the IYA programmes contributed to:

1. increased engagement, emotional regulation and communication skills of young children demonstrating behaviours associated with autism (child outcomes);
2. increased wellbeing and coping skills of caregivers enabling them to better support their child (caregiver outcomes);
3. increased teacher capability to help children demonstrating behaviours associated with autism (teacher outcomes); and
4. longer term and unintended benefits for those involved and the wider communities (additional benefits).

Where sufficient data was available, this evaluation also aimed to explore how different participants (e.g., differing by demographic characteristics, region of programme delivery) and their children benefited from the programme.

This evaluation report presents data obtained from Cohort 1 and Cohort 2 evaluations. IYA-T Cohort 2 data, also includes pre- and post-participation data obtained from Cohort 3 participants. An analysis of the Cohort 3 IYA-T data deemed it was appropriate to integrate this data within the Cohort 2 evaluation (henceforth, collectively referred to as Cohort 2).

The evaluation team drew upon qualitative and quantitative data to make an overall judgement about the effectiveness of the IYA programme in achieving expected child, parent, and teacher outcomes. These judgements were made in accordance with the programme specific evaluation rubric provided in the evaluation framework, which describes the criteria for classifying outcomes as ‘excellent’, ‘very good’, ‘adequate’, or ‘poor’. This rubric is presented in Appendix B. It should be noted that according to the theory of change proposed in the Ministry of Education Evaluation Framework, it was first expected that IYA participants would acquire skills, strategies, and behaviours that they would apply around the child. This would then result in secondary improvement in child outcomes. As such, we may expect to observe larger effects for proximal (parent and teacher) outcomes and smaller effects for more distal (child) outcomes.

Data collection and analysis

For the purpose of this evaluation, a combination of quantitative and qualitative methods were used that aligned with the evaluation questions and programme theory. Quantitative data was collected using a combination of caregiver- and teacher-reported assessment measures, administered during the pre-, post-, and/or ex-post training phases. In addition, several interviews were undertaken with parent and teacher participants during the ex-post training phase. Pre- and post-training assessments were administered via the IYA providers using the IYA app. Ex-post assessments were administered through the evaluation team, via telephone or Zoom. Qualitative data was analysed in accordance with inductive qualitative content analysis procedures.

Cohort 1, assessment data was collected from 60, 50, and 20 caregivers during the pre-, post-, and ex-post phases, respectively, and a total of 95, 70, and 47 teachers provided assessment data during the pre-, post-, and ex-post phases, respectively. A total of 14 parent and 27 teacher interviews were undertaken. This represents a survey attrition rate of 67% and an interview attrition rate of 77% from pre- to ex-post phases, for consenting caregiver participants. For consenting teacher participants, this represents survey and interview attrition rates of 51% and 72% respectively, from pre- to ex-post phases.

For Cohort 2, assessment data was collected from 61, 42, and 21 caregivers and 96, 75, and 26 teachers during the pre-, post- and ex-post phases, respectively. A total of 12 caregiver and 21 teacher interviews were undertaken. This represents a survey attrition rate of 66% and an interview attrition rate of 80% from pre- to ex-post phases, for consenting caregiver participants. For consenting teacher participants, this represents survey and interview attrition rates of 73% and 78% respectively, from pre- to ex-post phases.

A summary of the outcomes of this evaluation for Cohort 1 and 2, and a synthesis of these collective findings is provided below, and described in greater detail in the main body of this report. Attrition rates in the sample were high, especially at the ex-post phase, limiting the generality of conclusions drawn. For both quantitative and qualitative measures taken only at the ex-post point, only conclusions about the current state of the participants and their children are possible. It is not possible to draw causal conclusions about the contribution of participation to the state of affairs noted at the ex-post point because of the lack of pre-data.

Child outcomes (evaluation question 1)

Evaluation question 1 examined the extent to which participation in the IYA programme contributed to increased engagement, emotional regulation and communication skills of young children demonstrating behaviours associated with autism. The primary outcome of interest was child engagement, though additional measures were selected by the evaluation team to assess emotional regulation and communication skills, as secondary outcomes. Based on the Young Children’s Participation & Environment Measure (YC-PEM; Khetani, Graham, Davies, Law, & Simeonsson, 2015) there was some improvement reported in the frequency of children’s participation in the home environment for Cohort 1, and a reduction in the percentage of activities where change was desired by caregivers across cohorts. This change was most evident between pre- and ex-post training phases suggesting a delayed effect of training, while also providing evidence of the beneficial long-term effects of training. For the remaining two relevant YC-PEM variables (percentage of activities that the child participates in and the average involvement of the child in home activities), outcomes were trending (to a small extent) in a positive direction; however, the Effect Sizes (ES) were negligible and the practical magnitude of the change was often relatively small, suggesting little change in these outcomes in response to training.

Selected items on the Participant Program Satisfaction Questionnaire: Autism Spectrum and Language Delays Programme (PSQ-P) and Incredible Years Participant Satisfaction Questionnaire – Helping Preschool Children with Autism Program (PSQ-T) were also used by the evaluation team to measure secondary child outcomes (i.e., emotional regulation and communication skills). Both parents and teachers consistently reported that children’s self-regulation and imagination and social and emotional skills had ‘improved’ or ‘greatly improved’ post-participation in the IYA programme.

During interviews, caregivers commonly reported increased engagement with others, increased participation in the learning environment, and improvements in their child’s social and emotional regulation, communication and understanding. Some caregivers also reported that their child was more enthusiastic about attending their early childhood service, and teachers and other children were including the child in a greater number of activities such as structured games and birthday parties. Other caregivers indicated that their child had always been enthusiastic about attending the centre and there was no change in the child’s attendance or inclusion. Several caregivers said that the IYA programme exceeded their expectations and their child has shown significant progress since the IYA strategies have been implemented.

Based on quantitative data, it is the judgement of the IYA evaluation team that the impact of participation in the IYA programme on child outcomes is ‘adequate’. However, based on qualitative data the impact of participation in the IYA programme is considered to be ‘very good’ (while remembering that attrition meant that only a minority of participants supplied interview data).

Caregiver outcomes (evaluation question 2)

The second evaluation question examined the extent to which participation in the IYA programme increased the wellbeing and coping skills of caregivers, enabling them to better support their child. Based on the Autism Parenting Stress Index (APSI; Silva & Schalock, 2012) data, caregivers experienced reduced stress levels post-training, with an effect size indicating a borderline large and medium effect for Cohort 1 and Cohort 2, respectively. This reduction in stress was still evident at follow-up for both cohorts. The effect size was smaller (Cohort 1, -.5; Cohort 2, -.45) at this time point but still indicated a medium effect overall, particularly for those who entered the programme with high stress scores to begin with.

The Depression, Anxiety, and Stress Scale (DASS-21; Lovibond & Lovibond, 1995) was administered for Cohort 1 only. For this cohort, DASS-21 scores were within the normal range during the ex-post training phase. Interestingly, there was a significant correlation between APSI and DASS-21 scores, suggesting that these measures have good convergent validity as measures of stress. Therefore, the DASS-21 was not administered for Cohort 2.

On the PSQ-P, caregivers rated themselves as generally ‘optimistic’ about their progress toward the use of strategies that they were taught during the IYA programme and their goal achievement, suggesting that caregivers felt confident in the use of skills that they had acquired during the programme. Qualitative data analysis also revealed caregiver-reported improvement in their wellbeing and increased feelings of confidence and competence in their use of strategies.

Based on this data, the impact of programme participation on caregiver wellbeing and coping skills is considered to be ‘very good’ (but noting the caveat stated above re attrition).

Teacher outcomes (evaluation question 3)

The third objective of this evaluation was to determine the impact of participation in the IYA-T programme on the wellbeing and coping skills of teachers, enabling them to better support their child. Based on Incredible Years Teacher Strategies Questionnaire for Children with Autism (IYTSQ) pre-, post-, and ex-post data, teachers’ confidence and frequency of use of strategies improved across pre- post- and ex-post training phases. This finding is similarly reflected in PSQ-T (post-) data, which suggests that teachers felt ‘optimistic’ or ‘very optimistic’ about their future use of social and emotional coaching strategies.

Teacher interview data was also overwhelmingly positive, with key themes indicating that teachers felt more confident in their knowledge of Autism and in their ability to apply strategies to support children on the autism spectrum in their educational contexts. Many teachers described the IYA-T programme as the most useful professional development they have attended. A small number of teachers (Cohort 1, n = 4; Cohort 2, n = 1) felt that their participation in the IYA programme had a limited impact, as strategies that they learnt were ineffective when applied to a child on the autism spectrum or they felt they were already using strategies that had been taught.

Based on the available quantitative and qualitative data, the impact of the IYA-T programme on teacher outcomes was determined to be in the range of ‘very good’ to ‘excellent’.

Longer term and unintended benefits (evaluation question 4)

The final evaluation question was designed to assess any long-term and unintended benefits of programme participation. The long-term impact of participation in the IYA programme on children’s health and wellbeing was assessed using the Pediatric Quality of Life Inventory – Generic Core Scales™ (PedsQL; (Varni, 1998) ex-post data. Unfortunately, insufficient ex-post data and limited data variance meant that it was not possible to draw conclusions about the impact of programme attendance rates on wellbeing outcomes, even when data was pooled across cohorts.

The responses of the minority of caregivers who participated in interviews reported several unintended benefits of programme participation. These included increased communication and collaboration between home and the centre. Caregivers felt this was the result of increased confidence to initiate conversations with their child’s teacher. Several caregivers and teachers also reported sharing their learning with their immediate and extended family/whānau and colleagues, thus providing indirect benefit to those around the child. Many caregivers also reported personal benefits, including improvements in their own emotional regulation, the acquisition of new knowledge about autism, the opportunity to share and problem solve collaboratively, a positive effect on their relationship with their child and their partner, and the development of social supports and relationships with other caregivers completing the programme.

Teachers consistently reported an increased ability to support ALL learners, and the ability to share their learning with their colleagues and caregivers. A majority of teachers who were interviewed in Cohort 2 (n = 20) said they were still experiencing the benefits of the IYA-T programme, six months on. Overall, those interviewed described having positive and worthwhile experiences in relation to the children with whom they interacted at home and/or school.

Limitations of this evaluation

While the evaluation outcomes are largely positive, they should be interpreted and generalised cautiously in view of the limitations inherent within the structure of the evaluation design and resulting data. There are noteworthy limitations arising from sample attrition, a small ex-post sample size, and the lack of pre- data for many measures. Data analysis was also limited by the lack of data variance and the relatively small number of participants in the evaluation. This meant it was not possible to analyse the interaction between attendance rates, ethnicity, training region, and child, caregiver, and teacher outcomes, since appropriate analyses require both substantial variance and sample sizes.

This issue is further compounded by the fact that some of the ex-post measures were not administered during the pre-training phase. Additional ex-post measures were added to the data collection process because they were deemed relevant to answering the key evaluation questions. However, this limited the possible approaches to data analysis and prevented any conclusions that participating in training directly resulted in the outcomes measured, since such conclusions can only be based on pre-post, time-series data. It is important to note that, as acknowledged in the evaluation framework, there were limitations to what could feasibly and ethically be administered within the context of the programme.

Finally, a number of measures did not have normative data or criteria for classifications (e.g., the IYPSQ, PSQ, YC-PEM, APSI, and IYTSQ), including sound psychometric evidence for their validity in measuring key constructs, limiting the interpretability of the findings. This is a particular issue for interpretation in Aotearoa New Zealand with its ethnically distinctive and diverse community. These standardised measures were selected as they were designed specifically for the evaluation of IYA programmes and thus, there is a strong rationale for the selection of these tools, however, these limitations do have implication for data analysis, interpretation of findings, and any policy recommendations that may be made based on the findings. These limitations and subsequent recommendations are described more fully in the main body of this report.