Executive Summary

Preamble

Applied Behaviour Analysis (ABA) is defined as a scientific approach for discovering environmental variables that reliably influence socially significant behavior and for developing a technology of behavior change that takes practical advantage of those discoveries (Cooper et al., 2007, p. 3). As a means to produce meaningful changes to behaviour ABA has demonstrated its effectiveness at the level of the individual, group or community. This effectiveness has been highlighted by the large number of publications, spanning back five decades, illustrating significant behavioural change that can be maintained after the intervention has ceased across a range of settings and behaviours (including educational, health, animal welfare, sporting performance, and psychological well being).

As outlined in the ‘New Zealand Context’ section, New Zealand has access to high quality training programmes that carry international recognition in the areas of ABA and the Experimental Analysis of Behaviour. As such, we are fortunate in this country to have people with the sufficient depth of skill required to successfully develop ABA interventions as well as conduct the comprehensive and internationally-credible review reported in this document.

Purpose and Scope of the Review

The attached report consists of a comprehensive review of the evidence concerning the effectiveness of applied behaviour analytic intervention methods for people with Autism Spectrum Disorders (ASD). The review was in response to a call by the New Zealand Ministries of Health and Education for an assessment of the effectiveness of behaviourally-based interventions as determined by studies published between 1998 and 2007. The data used for this review was gained from two main sources:

• from existing analyses of peer-reviewed publications previously collected by the National Autism Center National Standards Project (NSP) based in the USA; and
• from additional publications that met the appropriate criteria laid out by the Ministries of Health and Education that had not been included in the NSP.

Method

Studies included for analysis (irrespective of whether they were sourced from the NSP or by NZ-based reviewers) had to meet a range of criteria (as outlined in the ‘Review Method’ section). In addition to restricting articles to those published between 1998 and 2007, studies were only included if they met, or exceeded a score of 2.0 on the Scientific Merit Rating Scale (SMRS). In total 463 items¹ were retained from the NSP database along with 45 New Zealand unique items (that met inclusion criteria but had not formed part of the NSP database).

Articles were also analysed to determine the specific components of the behavioural intervention package (see Table 2), the type of behaviours assessed (as per Ministry of Education supplied classifications identified on Table 5), as well as the impact of the interventions utilised (using the Strength of Evidence Classification System – SECS). Note that the 13 New  Zealand based reviewers underwent extensive training in the coding schemes (as did their NSP counterparts) and national and international checks of inter-rater reliability were determined to be generally high.

Main Findings

Overall, there is strong evidence that behavioural interventions result in beneficial outcomes for individuals with ASD. Figure 1 shows the percentage of outcome type (‘beneficial’, ‘ineffective’, harmful’, or ‘unknown’) averaged across the range of specific behaviours assessed (ranging across a variety of social and cognitive domains). Although the efficacy of approximately one quarter of the items examined was unable to be clearly determined, the vast majority of outcomes were beneficial in the remaining cases, and thus a meaningful and desirable change in behaviour occurred as a specific result of that intervention. Only 2% of 508 items that contributed to our results were rated to show that a behavioural intervention was ineffective in a particular case. However, no behavioural interventions were rated overall as ineffective. In no case was harm reported as a result of behavioural intervention.²

Figure 1: Overall efficacy of behavioural treatment across all behavioural categories assessed.

In addition to the overall findings it should be noted that this same overall pattern of beneficial effects was also observed across specific individual behavioural categories. Figures 2 to 9 illustrate this for each of the relevant behavioural categories identified in Table 5 (refer to Results sections 1 to 9 for details). Specifically, the overall evidence for beneficial outcomes was rated as ‘strong’ in the areas of social development, cognitive development, communication, play / vocational engagement, development of organisational skills, and prevention and replacement of challenging behaviours. There was strong evidence for the benefits of comprehensive early intervention programmes for some areas of development.

Efficacy of ABA interventions by specific behavioural categories

Comments on generalisation and maintenance

Greater utility of an intervention (irrespective of how effective it is in changing a specific behaviour in a specific setting) often depends on whether desirable changes go beyond initial settings and can be maintained beyond the initial intervention period. ‘Generalisation’ is concerned with the spreading of an intervention's effects beyond initial settings to novel naturalistic ones; whereas ‘maintenance’ is concerned with ongoing intervention effectiveness after formal intervention has ceased to be applied.

Because the relevant data was not able to be obtained from the NSP study, generalisation and maintenance were only assessed for 169 research items reviewed by New Zealand reviewers. This analysis (outlined in Results 9) revealed that close to two thirds of ABA interventions displayed strong evidence that main intervention effects generalised beyond their initial training environments. There did not appear to be any particular relationship between type of intervention, level of intervention efficacy, and the behaviours targeted and whether generalisation was displayed or not. Similarly, maintenance of main effects was demonstrated in 75% of studies that reported the appropriate data. Maintenance, also, was observed irrespective of intervention type and behavioural category. Therefore, there is convincing evidence that ABA treatments generalised beyond their initial settings and their effectiveness could be maintained for the majority of ASD participants beyond the initial intervention period.

Scope of Current Research and Future Opportunities

One of the strengths of the current review was the utilisation of a large number of independent reviewers and the incorporation of a large international data base. Despite the number of individuals involved there was generally high inter-observer agreement and the pattern of findings was remarkably consistent across reviewers charged with analysing separate behavioural categories (compare Figs 2 to 9). However, lower levels of inter-observer reliability were observed for the analyses of generalisation and maintenance effect sections; so these results may need to be treated with caution until a larger database is obtained or further work can be conducted.

Several other observations to emerge during the review also indicate the need for further research in the area of ABA treatments of ASD, specifically:

• A number of items had to be excluded from further analysis because they failed to meet an adequate level of scientific rigour (defined as an SMRS score of less than 2.0). A number of reasons for this are identified in Results 12. One important factor is the common use of single-participant research designs. However, the implication is that researchers in this area need to take in to account various design features if their studies are to be included in reviews such as the NSP and the current one.
• The extent of evidence concerning beneficial outcomes is lacking (that is, either emerging or unknown) for persons with an Asperger Syndrome diagnosis. Strong evidence exists for benefits for children up to the age of 15 years. However, across the ASD spectrum, there is, to date, insufficient evidence to provide strong support for ABA interventions for participants aged 15-21 years, and in the target categories of academic skills, learning readiness, and problem behaviours of all types. The New Zealand review data (see Appendix C) may be seen as providing emerging evidence for benefits for adults > 21 years in some target areas, but clearly there is more scope for research in these specific areas.

Final Comment

The current review demonstrated that ABA interventions can produce meaningful and desirable behavioural change in individuals with ASD, and taken as a whole, there is strong support for the effectiveness of ABA interventions in the treatment of ASD. Significant benefit was also consistently illustrated at the level of specific behavioural categories; with evidence at this level either meeting the criterion of ‘emerging evidence’ or ‘strong support’ for the conclusion of intervention effectiveness. It is important that agencies charged with providing resources for the treatment and education of people with ASD take notice of these findings. Fortunately, New Zealand has two internationally-recognised ABA training programs that lead to qualifications recognised by the Behavior Analyst Certification Board that can potentially produce skilled practitioners able to implement the types of skilled interventions outlined in the current review.

Footnotes

1. Many articles reviewed reported more than one independent variable and/or dependent variable and/or two or more studies within the article. Every variation can be viewed as a discrete study, or research “item”, where the simplest case (or item) is one study with one dependent variable and one independent variable. Each article was scored as many times as there were variations within it. Hence, in reporting findings, numbers of items rather than numbers of articles are counted.
2. Although the vast majority of articles reviewed reported on the results of ‘intervention packages’ (i.e. where more than one specific behaviourally-based intervention was applied within the study) a range of specific features can be identified in those packages. Those features are illustrated in Tables 2 and 6.