Evaluation of Two Autism Spectrum Disorder Parent Education Programmes
Since 2001, the Ministries of Education and Health have co-purchased autism spectrum disorder (ASD) parent education programmes for parents and whānau of young children (from 0 to starting school age) with ASD. Over this time, two programmes have been purchased:
- EarlyBird – a group parent education programme designed and owned by the National Autistic Society in the UK, and delivered by Autism New Zealand in most of the North Island.
- ASD Plus – family and whānau education designed and delivered by Idea Services that can be offered face-to-face or via distance with technology, and is suitable for individual families/whānau or group delivery. Idea Services delivers ASD Plus in the South Island and Northland.
Author(s): Litmus, Prepared for the Ministry of Education
Date Published: September 2011
For the past 10 years the Ministries of Health and Education ('the Ministries') have co-purchased two autism spectrum disorder (ASD) parent education programmes for parents and whānau of young children (from 0 to starting school age) with ASD: the National Autistic Society (NAS) EarlyBird ('EarlyBird')
and ASD Plus: Education for Families ('ASD Plus'). The current contracts for the programmes run for two years, until June 2011.
Litmus undertook concurrent evaluations of the programmes between 6 August 2010 and 10 November 2010. Evaluation data sources included:
- Provider reports to the Ministries, administration and monitoring data, and other relevant documents
- A quantitative self-completion survey of all families who had completed a programme since July 2009
- Qualitative interviews with key stakeholder audiences
- Background 'expert' interviews with two Ministry of Education personnel
Evaluation limitations included:
- A lack of monitoring and evaluation data, from both providers
- Outcomes data was indicative only, however, consistent findings triangulated across all data sources offer some confidence in the findings relating to programme impact
- The evaluation methodology included some risks around coercion, however Litmus took steps to minimise this risk. Evidence was highly consistent, and Litmus detected no evidence that coercion of any kind occurred.
Specific requirements of the EarlyBird contract with the Ministries are as follows:
- Deliver to families situated in the North Island of New Zealand, excluding Northland
- Deliver to at least 120 families in each year of the contract
- Undertake for tripartite discussions with the NAS and Autism NZ within the first six months of the contract
Key evaluation findings
Autism NZ developed and delivered key programme outputs such as the Action Plan and the Cultural Responsiveness Plan, and established a Parent Consumer Advisory Group. Reporting targets were met. 104 families participated in EarlyBird in Year One of the contract. Key points to note for the period July 2009 – September 2010 are:
- 22 EarlyBird programmes were completed, involving a total of 124 participants. On average, 6 participants attended each programme.
- The majority of programmes were run in Auckland (13 of the 22). 4 programmes were delivered in Wellington, 2 in Tauranga, and 1 in each of a number of other provincial centres (Hawkes Bay, New Plymouth and Gisborne).
- All programmes used the 'group' delivery format. However one family in New Plymouth had a 'modified' programme
- Three EarlyBird programmes were run in collaboration with Ohomairangi Trust (2 in South Auckland, 1 in Gisborne).
Key findings in relation to the EarlyBird programme impact are:
- There is consistent evidence across all data sources that EarlyBird is generating favourable outcomes for participants. Particular areas of strength are: improving participants' knowledge of autism; improving communication strategies; using play to interact and teach social interaction skills; and improving behaviour management
- Main changes seen in children, following EarlyBird, are: more interaction, better communication, and less frustration. Main changes in families/whānau include: better relationships, more relaxed households, more unified parenting and better routines/planning
- An unintended impact of the programme is providing some parents with a greater sense of empowerment and confidence outside the family, dealing with 'the system' and advocating on behalf of their child. Another unintended impact of the programme is the emotional benefit many parents derive from the group delivery format
- A small amount of negative feedback was received relating to programme logistics. The other criticism of EarlyBird is a lack of New Zealand-specific content. Follow-up contact, and finding ways to stay in touch with other parents to support each other over the longer term, is suggested as an improvement by some parents.
Key programme strengths
- There is clear evidence from the evaluation, reinforced by previous evaluations undertaken, that EarlyBird is highly valued by families who access the service, and that positive and intended outcomes are occurring for families who take part. For many participants, EarlyBird participation appears to be a turning point in their journey with ASD, providing them with practical tools and strategies to improve communication with their child, and help deal with behaviour. Implications for the wider family are positive, with reduced stress and better relationships occurring as a result of one or more families attending the programme. The deeper understanding of ASD that parents achieve through the programme also empowers them to be more confident advocates for their children in the future.
- Autism NZ has delivered most key outputs and achieved a high volume of families participating in the programme, despite a slow start to the contract and substantial organisational change during Year One of the contract.
- The three collaborative programmes that have been run in partnership with Ohumairangi Trust appear to have increased level of participation for Māori and Pacific peoples families in EarlyBird, compared with previous years. Modifications to the core programme to deliver collaboratively (i.e. providing childcare, transport, co-facilitation with Ohumairangi staff, and so on) appear to have made the programme more accessible to Māori and Pacific peoples' cultures, and are valued by participants.
Key programme weaknesses
- The EarlyBird programme is not reaching those in rural and remote areas. The primary delivery format for EarlyBird of group sessions appears to be the key barrier to achieving higher levels of participation from rural families. While one programme in New Plymouth has been modified to allow some distance learning, further refinements are needed to provide flexibility which will make the programme more accessible.
- Internal monitoring and evaluation systems for EarlyBird by Autism NZ appear to be inadequate. While data is being collected for each programme, it seems to be incomplete, and Autism NZ head office does not appear to have access to all data that has been collected. The current parent evaluation sheet, combined with the 3-month follow up questionnaire, is not generating adequate outcomes data.
- While the collaborative programme has shown real promise, the programme requires more work to tailor it to a New Zealand context and to make it more culturally responsive. The planned DVD may help to provide a New Zealand 'flavour' to the materials. Workforce development strategies to recruit and train Māori and Pacific peoples facilitators need to be implemented, so that EarlyBird becomes more responsive 'from within'.
Priority questions were specified by the Ministries to be investigated for both evaluations. Based on the evidence available to the evaluation, Litmus' response to each question is summarised below.
1. How valuable/high quality is the ASD-specific parent education programme's content/design and delivery?
Overall, there is strong evidence that EarlyBird is a high quality programme, as indicated by the evidence-basis from which it was developed, previous evaluations and outcomes data from the evaluation survey.
Planned work by Autism NZ to check that EarlyBird aligns with the NZ ASD Guideline would support the evidence about EarlyBird programme quality.
The programme could be improved to be more innovative and responsive in its delivery – particularly by offering a wider range of delivery formats. It could also be improved to become more family-centred – encouraging greater participation from the wider family, rather than mainly parents, and providing assistance to enable both parents to attend programmes where possible. Waiting list times may be an inherent result of co-ordinating group delivery, but there is a need for Autism NZ to minimise these to ensure parents receive EarlyBird in a timely manner.
On anecdotal evidence, the collaborative programme appears to have been successful in terms of ensuring cultural and contextual appropriateness for EarlyBird delivery, but more could be done across the programme as a whole to provide a programme that is appropriate for the specific New Zealand context and cultures. There is a need for Autism NZ to undertake the planned evaluation of the collaborative programme to provide further evidence and insights which can be used to improve delivery in future.
2. What changes are evident for the young child with ASD and their family/whānau/carer(s) that can reasonably be attributed to the programme (directly or indirectly)?
There is evidence from the current evaluation that EarlyBird has a positive impact on the child and their family. Areas where families report improvement are around communication and social interaction skills for the child, and managing difficult behaviour. Enhanced confidence and relationships within the family are also evident as a result of EarlyBird participation.
Ongoing evaluation carried out by Autism NZ needs to ensure that it captures outcomes data, and longer-term outcomes for families.
3. How effectively are the participants applying and using their new knowledge?
As noted above there is some evidence that participants are applying skills and knowledge after taking part in EarlyBird. There is a gap in the evidence-basis around longer term impacts for children and families.
4. What unexpected outcomes have resulted?
Two unexpected outcomes are evident: Parents report a greater sense of empowerment and confidence outside the family, dealing with 'the system' and advocating on behalf of their child; and an emotional benefit derived from the group delivery format.
5. How effectively was the trade-off managed between the reach and intensity of the ASD-specific parent education programme, given the relevant constraints?
In many ways Autism NZ appears to be managing the trade off between reach and intensity of the EarlyBird programme effectively – the programme is being delivered to a relatively large number of families in many parts of the North Island. However, EarlyBird has a notable gap in delivery to families in remote and rural areas.
The focus on a group delivery format appears to restrict the ability of EarlyBird to achieve greater geographic coverage. The emphasis on group delivery format also appears to limit participation from a wider range of cultures – while the collaborative programme shows promise as a working model, the waiting times to assemble groups of particular cultural backgrounds may limit accessibility overall.
6. To what extent does this ASD-specific parent education programme represent the best possible use of resources?
The evaluation provides clear evidence that EarlyBird is valued highly by participants, and it continues to contribute to positive outcomes for children with ASD and their families. There do not appear to be any alternative, comparable services for parents, suggesting that the absence of EarlyBird would mean poorer long term consequences for children with ASD and their families.
Taking into account factors that Autism NZ considers to have contributed to lower than expected delivery numbers in Year One of the contract, overall, EarlyBird is reaching a large number of North Island families in cities and provincial towns. The collaborative programme shows signs of being a successful model for improved participation rates in EarlyBird by Māori and Pacific peoples' families.
Programme delivery constraints mean that those in rural and remote areas are not accessing EarlyBird. Autism NZ reports that a 'modified' EarlyBird programme used with one family in New Plymouth will be reviewed as a potential alternative model for rural delivery of the programme – future monitoring and evaluation will be needed to determine whether this is a successful strategy.
Indepth costing analysis of EarlyBird programmes (including the collaborative programme vs the 'standard' EarlyBird programme) and other comparable services would be required to determine whether EarlyBird represents the best possible use of resources. This analysis would also need to take into account the longevity, location and wider organisational context of different programmes – that is, recognising that EarlyBird is a mature programme which has robust evidence of its efficacy, but there may be structural limitations on its ability to deliver to all audiences with funding currently available. Possible cost saving delivery innovations may include online components and use of distance technologies such as video conferencing. However, these innovations require further scoping to determine the extent to which they can successfully address current delivery limitations in reaching rural or isolated families.
Aspects of the contract which are specific to ASD Plus are as follows:
- Design and set-up of the programme. Key outputs required included:
- Designing a Project Development Plan (due 17 July 2009)
- Establishing a Parent Consumer Advisory Group (by 2 November 2009)
- Developing an Action Plan; Cultural Responsiveness Plan and Communications Plan (by 2 November 2009)
- Delivery to families situated in the South Island of New Zealand, and in Northland
- Delivery to 57 families in Year One of the contract, and 75 families in Year Two i.e. a total of 132 families over the period of the contract
Key evaluation findings
IDEA Services designed and developed the ASD Plus programme in Year One of the contract, delivery key outputs including an Action Plan and Cultural Responsiveness Plan. A Parent Consumer Advisory Group was established. All reporting targets were met. Key points to note for the period July 2009 – September 2010 are:
- 36 families participated in ASD Plus programmes, involving a total of 83 participants
- The majority of families were based in the South Island, including those based in cities, provincial towns and rural locations (31 families). 5 families from Northland have participated in ASD Plus since the programme began
- Most families received the 'individual' delivery format for ASD Plus (26 of the 36 families). Two group programmes were delivered, in two locations (Dunedin and Christchurch). 9 families received a 'combined' delivery method involving a mix of group, distance learning and individual delivery
Key findings in relation to the programme impact of ASD Plus are as follows:
- The evidence is consistent across all data sources that participants have been satisfied with the ASD Plus programme delivery. There is indicative evidence that the programme is achieving its objectives and having a positive impact on participants, children, the immediate and wider family. Particular areas of strength indicated by the data are: increasing participants' knowledge and understanding of ASD; increasing use of a range of strategies and tools to encourage improved communication and social interaction of the child with ASD; increasing parents' use of strategies to manage difficult behaviour
- Changes seen in children, following ASD Plus, include: Less frustration and improved communication. Changes for parents and families after ASD Plus participation include: More supportive attitudes, more relaxed households, unified parenting and a sense of being in control/more confident.
- ASD Plus facilitators receive very positive feedback from programme participants.
- An unintended, but positive impact is an ongoing sense of empowerment to be the ASD child's advocate. Other unintended impacts relate to emotional support and networking opportunities afforded by group delivery of the programme
- Negative feedback about ASD Plus is minimal. A few suggested improvements relate to changes in timing and logistics, more opportunity for interaction with other parents via group sessions (from those who did individual programmes), ongoing contact and advice about 'where to next'.
- IDEA Services achieved all programme design and development outputs within Year One of operation, including a New Zealand-made DVD film resource which will be particularly beneficial in enabling distance learning options. Management and monitoring systems appear to be comprehensive, well thought through, and functioning efficiently. The pool of facilitators seems adequate to cope with current and future demand, and supervision and workforce development activities are working well.
- The available evidence for ASD Plus indicates that this new programme is achieving positive, tangible short-term outcomes for participants and families in line with programme objectives. Participant satisfaction is consistently high, and there is evidence that most participants are applying ASD Plus-learnt strategies and tools in their households.
- The tailored, individual approach offered by ASD Plus is a real strength of the programme. Families respond positively to the opportunity to work in partnership with IDEA Services to determine both mode of delivery, and specific goals the family wishes to achieve. The evaluation also captured numerous examples of IDEA Services managers and facilitators 'going the extra mile' to ensure that families can participate and get the most out of ASD Plus.
- While numbers are low, the flexible, family-centred approach appears to be driving participation across a wide range of families, in terms of geographic coverage, cultural backgrounds, and participation from the wider whānau as well as parents. The fact that ASD Plus resources are New Zealand-specific would appear to enhance the relevance of the programme to participants.
- Evaluation findings indicate that IDEA Services is highly responsive to feedback from participants and other sources. There are clear benefits in IDEA Services' ability to readily evolve and adapt the programme, particularly while it is in the inception phase
- The Service Leader for ASD Plus has personal expertise and indepth understanding of ASD, and a long history of working in the field. The fact that he was involved in initial programme design and implementation, and continues to undertake and supervise delivery of the programme has positive quality control implications as he is able to modify delivery and content (as shown in the piloting of the programme) based on results and feedback.
- The monitoring and evaluation system that has been implemented by IDEA Services is thorough and includes the collection of outcomes data which will enable evidence-based evaluation of the programme over time.
- The primary issue for ASD Plus is low referrals to the programme, which mean that the overall participation rate is well below contractual expectations, and the 'per family' cost is much higher. IDEA Services is well aware of this issue and has undertaken a range of communication and networking activities over the course of the contract to try and boost referrals, but the impact of specific activities on referral levels it is not clear.
- To a degree, low referrals may reflect low awareness of a new programme in the inception phase. Other systemic issues may be in play – such as the lack of follow-up to families in the early post-diagnostic phase. The Christchurch earthquake also appears to have reduced expected referrals in the past couple of months.
- It should be noted that Northland referrals, while initially low to non-existent, are now running at expected levels because the District Health Board has overcome capacity issues that meant young children were not being diagnosed.
Priority questions for the evaluation
The Ministries developed a series of priority questions to be investigated for both evaluations. Based on the evidence available to the evaluation, Litmus' response to each question is summarised below.
How valuable/high quality is the ASD-specific parent education programme's content/design and delivery?
Evidence from the evaluation indicates that ASD Plus is a high-quality programme developed from a national and international evidence basis and in line with the NZ ASD Guideline. Early feedback suggests that programme participants find ASD Plus extremely valuable, as they perceive an absence of other support and education services post-diagnosis. The programme is innovative in the New Zealand context, in terms of offering a range of delivery modes including 'hybrid' delivery. The flexibility of delivery options, and the upfront involvement of families in selecting the most appropriate option for their unique needs, makes the programme highly person and family-centred. Participation from across a range of geographic areas and cultural backgrounds suggests that the programme is culturally and contextually appropriate.
What changes are evident for the young child with ASD and their family/whānau/carer(s) that can reasonably be attributed to the programme (directly or indirectly)?
Early evidence suggests that ASD Plus is contributing directly to positive short-term outcomes for participants, their children, and wider family units. Participants report that they are applying a deeper understanding of ASD, and practical tools and strategies in their homes, with positive changes resulting. Changes noted include improved communication and social interaction skills for the child with ASD, a reduction in challenging behaviour (and/or improved coping mechanisms to deal with it), and enhanced relationships within the family as a whole. Longer-term data from IDEA Service's monitoring of families will determine whether these outcomes persist over time.
How effectively are the participants applying and using their new knowledge?
As noted above, there is evidence that ASD Plus participants are applying their new knowledge and skills successfully in their homes. Further work is needed to determine how extensively and effectively this is occurring, and whether the programme has lasting impact.
What unexpected outcomes have resulted?
There is a small amount of evidence that ASD Plus contributes to three unexpected positive outcomes: Some parents becoming more confident and empowered to communicate with others about their child's needs, and push to have these needs met; some deriving emotional benefit from participating in group sessions with other families; ongoing family support networks arising out of connections made through ASD Plus group sessions.
How effectively was the trade-off managed between the reach and intensity of the ASD-specific parent education programme, given the relevant constraints?
On a per programme basis, the duration and intensity of ASD Plus appears to be about right to achieve the desired outcomes for participants. IDEA Services have already responded to feedback to change the length of some modules to ensure adequate coverage of content.
As a whole, ASD Plus is not achieving the expected reach to South Island and Northland families – the total number of families participating is considerably lower than expected, reflecting low referrals to the programme. IDEA Services has not found it feasible to deliver as many group programmes as expected, given the low referral numbers
To what extent does this ASD-specific parent education programme represent the best possible use of resources?
Early evidence for ASD Plus suggests that parents and whānau of young children with ASD value the programme extremely highly, and that it directly contributes to positive outcomes in line with programme objectives. Parents' perception of an absence of other alternative services, particularly for those in rural areas, means that the programme appears to be hugely beneficial to those who take part.
The programme is currently running at below capacity levels. In this context, IDEA Services has been able to operate without a waiting list, delivering highly tailored programmes and catering well to those in rural areas. If referrals increase and IDEA Services can maintain the high quality of service provision seen to date across a larger number of families, then we could conclude that ASD Plus is maximising the use of Ministry resources available to the programme.
Indepth costing analysis of comparable programmes would be required to determine whether ASD Plus represents the best possible use of resources. This analysis would also need to take into account the longevity, location and wider health and education context of different programmes – that is, recognising that ASD Plus is in the inception stage, and that there may be systemic or sectoral issues beyond its control that impact on referrals.
Education Data Requests
If you have any questions about education data then please contact us at:
Email: Requests EDK
Phone: +64 4 463 8065