Technical review of published research on applied behaviour analysis interventions for people with Autism Spectrum Disorder
New Zealand Ministries of Education and of Health requested a technical review of the evidence base on the effectiveness of Applied Behaviour Analysis (ABA) for people with Autism Spectrum Disorders (ASD).
Released on Education Counts: April 2010
Author(s): Oliver Mudford, Neville Blampied, Katrina Phillips, Dave Harper, Mary Foster, John Church, Maree Hunt, Jane Prochnow, Dennis Rose, Angela Arnold-Saritepe, Heather Peters, Celia Lie, Katrina Jeffrey, Eric Messick, Catherine Sumpter, James McEwan and Susan Wilczynski (2009), Auckland UniServices Limited.
Date Published: 15 January 2009 - Revised 16 January 2009
This report is available as a download (please refer to the 'Downloads/Links' inset box, top right). This inset box also has links to related publications and information that may be of interest. Please consider the environment before printing.
New Zealand Ministries of Education and of Health requested a technical review of the evidence base on the effectiveness of Applied Behaviour Analysis (ABA) for people with Autism Spectrum Disorders (ASD). All data presented in this report were from review of original research articles using the methods developed by the National Autism Center National Standards Project (NSP; see Wilczynski & Christian, 20083). The New Zealand ABA Review Group operated within an affiliation agreement between NSP and Auckland UniServices Ltd enabling cooperation between the review team and NSP. NSP recruited approximately 80 reviewers across the US and internationally to conduct their reviews. The New Zealand review team included 13 reviewers, of whom approximately half had been trained and provided reviews for NSP before the New Zealand review project commenced.
The NSP review covers peer-reviewed scientific publications investigating the effects of interventions for children and young people (ages 0-21 years). The interventions include all those that could realistically be replicated in educational and/or clinical settings. The New Zealand Ministries of Health and of Education called for behaviour analytic intervention studies, published from 1998-2007, to be reviewed without any participant age limits. The New Zealand review scope is, for the most part, a subset of the NSP review’s scope. The NSP did not include studies in which all the participants were >21 years old, nor participants’ data if they had medical complications or psychotic disorders.
The aim of the NSP was to determine the interventions that are best supported by the scientific evidence concerning their benefits. The basic process adopted by NSP was to identify likely relevant articles from the scientific literature, measure the scientific merit and treatment benefits from each article, group research articles by categories, and calculate the overall strength of evidence from consideration of all articles in each category. Strength of evidence for benefits was rated by NSP as (A) strongest, (B) strong, (E) emerging, (U) unestablished, (I) ineffective, and (H) harmful. Two points relevant to the present report arise. First, the New Zealand reviewers are reviewing only behavioural interventions and therefore no direct comparison is possible with the strength of evidence concerning other approaches (e.g., “developmental-pragmatic”). Readers will need to consult the NSP’s report to enable that. Wilczynski and Christian (2008) predicted that the NSP report would be published in the (northern) Spring of 2008. The most recent estimate is “by June 2009” (Susan Wilczynski, personal communication, 10th February, 2009). Second, it is not obvious how the NSP’s A, B, E, etc. ratings map onto those employed by the Ministries in the New Zealand Autism Spectrum Disorder Guideline (2008, p. 14). Wilczynski and Christian (2008) concluded that “Those interventions that hold strong or the strongest levels of research support after the completion of the National Standards Project should be given greatest consideration when selecting treatments”. As of January 2009, NSP have now eliminated the distinction between Strong and Strongest levels of support to avoid confusion for their readers. We acknowledge that this may create confusion for readers of both our reports. NSP findings contained in our report that are rated as providing Strong evidence will be categorised by NSP as indicating “Established Treatments” in line with the wider evidence-based practice movement in human services. Further discussion in New Zealand concerning these issues in the light of the reviews’ findings is to be encouraged.
NSP provided the New Zealand team with coding manuals; coding forms; algorithms for determining the scientific merit of research articles, treatment effects, and strength of evidence; and evidence tables containing their data from behavioural intervention studies. In the NSP, scientific merit was assessed by the Scientific Merit Rating Scale (SMRS), a copy of which is appended (Table 1, Appendix A). The factors rated 0-5 for scientific merit were: Research design (.30), measurement of the dependent variable (.25), measurement of the independent variable (.15), quality of diagnosis (.20), and the extent to which generalization and maintenance effects could be determined (.10). The scores were multiplied by the number in brackets and added to obtain a “composite SMRS” score from 0 – 5.
The NSP’s method for rating of “treatment effects”, i.e., whether a particular study demonstrated beneficial, unknown, ineffective, or adverse effects for the participants with ASD, is copied as Table 2, Appendix A. The NSP method for summarising the strength of evidence across groups of studies that reported similar independent variables (e.g., ABA methods) and similar dependent variables (e.g., communication skills, problem behaviours), the Strength of Evidence Classification System (SECS) is copied in Table 3, Appendix A.
- References to sources other than articles included or excluded in the Review are given in a Reference list below.
Where to find out more
For more information on this publication or Special Education services, please email: Special Education Mailbox