Technical review of published research on applied behaviour analysis interventions for people with Autism Spectrum Disorder
‘Technical Review of Published Research on Applied Behaviour Analysis Interventions for People with Autism Spectrum Disorder’ New Zealand Ministries of Education and of Health requested a technical review of the evidence base on the effectiveness of Applied Behaviour Analysis (ABA) for people with Autism Spectrum Disorders (ASD).
Released on Education Counts: April 2010
Author(s): Oliver Mudford, Neville Blampied, Katrina Phillips, Dave Harper, Mary Foster, John Church, Maree Hunt, Jane Prochnow, Dennis Rose, Angela Arnold-Saritepe, Heather Peters, Celia Lie, Katrina Jeffrey, Eric Messick, Catherine Sumpter, James McEwan and Susan Wilczynski (2009), Auckland UniServices Limited.
Date Published: 15 January 2009 - Revised 16 January 2009
This report is available as a download (please refer to the 'Downloads' inset box). For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
Overall, there was strong evidence that behavioural interventions result in beneficial outcomes for individuals with ASD. Although the efficacy of approximately one quarter of the items examined was unable to be clearly determined, the vast majority of outcomes were beneficial in the remaining cases, and thus a meaningful and desirable change in behaviour occurred as a specific result of that intervention. Only 2% of 508 items that contributed to our results were rated to show that a behavioural intervention was ineffective in a particular case. However, no behavioural interventions were rated overall as ineffective. In no case was there harm reported as a result of behavioural intervention
Despite there being strong evidence overall for many of the categories of behavioural interventions reviewed, the strength of evidence is necessarily diluted as the unit of analysis of the evidence is reduced. To illustrate with one detailed example, in excess of 350 items were rated by National Standards Project (NSP) for the “behavioural package” category of interventions. Only 140 contributed to the results as others did not obtain a Scientific Merit Rating Scale (SMRS) composite score ≥ 2. Possible reasons for this are discussed below. Although the overall evidence level (Strength of Evidence Classification System: SECS rating) for behavioural package category was strong, this was true only for (a) the autistic disorder diagnostic classification; (b) children between the ages of 3 and 6 years; and (c) for interpersonal, learning readiness, play, and self-regulation target behaviour categories. Effects for other diagnostic classes, age groups, and target behaviours were either emerging (12 sub-samples) or unknown (two sub-samples).
The following two paragraphs summarise evidence of the type presented in the illustration with “behavioural packages” across all categories of intervention. It is important to note at the outset that there were no conclusions that suggested the interventions included in this package were ineffective or harmful, and that applies to all categories of intervention (Table 2).
Strengths in the evidence
There is strong evidence that behavioural interventions can have beneficial effects for individuals with diagnoses of autistic disorder and pervasive developmental disorder. Strong evidence exists for benefits for children up to the age of 15 years. There is strong evidence for benefits in the areas of communication, higher cognitive functions, interpersonal skills, personal responsibility, play, and self-regulation (NSP categories), i.e., social development, cognitive development, communication, play / vocational engagement, and development of organisational skills (Ministry of Education categories). This is not to say that the evidence is strong for all age groups up to 15 in all areas targeted for change for both autism and PDD, but that there is at least one area of development for which there is strong evidence for the benefits of ABA.
Weaknesses in the evidence
Evidence is lacking (that is, either emerging or unknown) for persons with an Asperger Syndrome diagnosis. Across the ASD spectrum, there is, to date, insufficient evidence to provide strong support for ABA interventions for participants aged 15-21 years, and in the target categories of academic skills, learning readiness, and problem behaviours of all types. The New Zealand review data (see Appendix C) may be seen as providing emerging evidence for benefits for adults > 21 years in some target areas.
Comments on the quality of ABA research for the purposes of determining evidence-based practices that can be strongly recommended for widespread adoption.
We applied fairly stringent exclusion criteria to the literature before we started to examine research articles to rate with the SMRS (Appendix B). Nevertheless, we had to exclude 78/169 items (46%) following SMRS rating because their composite SMRS score was <2.0. The mean composite SMRS score was 2.17 out of a maximum of 5 across all the items we reviewed. Does this suggest that ABA research is generally of low quality?
Further examination of the SMRS scores that made up the composite score showed that the mean scores for design, measurement of the dependent variables, measurement of the independent variable, diagnosis, and generalisation/maintenance (all with a maximum of 5) were: 2.00, 3.81, 1.85, 0.69, and 1.77 respectively. Clearly, the quality of diagnosis was poorest. The extent to which studies were designed to assess generalisation and maintenance and treatment fidelity was, on average, low. Design appears barely acceptable, and only measurement of the behaviours of interest (target behaviours) was more than adequate. However, the studies were rated by NSP criteria, which are not the general scientific criteria employed by researchers, authors, editors and journal reviewers.
Reports of participants’ diagnosis and classification according to DSM or similar criteria are vanishingly rare in the ABA literature. The likely reason for that is that behaviour analysts work with behaviours, not hypothesised underlying disorders (e.g., Autism), and apply functional analyses to these behaviours, rather than prescriptions based on diagnoses. Diagnosis is usually seen as irrelevant, however, it has more importance when a research study is being assessed for its potential for generalisation to other people with the same diagnosis, as is the case in the NSP and New Zealand reviews.
The extra resources required for direct observational measurement of the implementation of the intervention, and of behaviour change in other settings (generalization) and beyond the conclusion of the treatment (maintenance) may be a factor that prohibits the routine assessment of these variables. If behaviour analysts aim to have their research considered for its generality across individuals with, say ASD, we will need to find the resources to do that.
Regarding experimental design, by NSP criteria, no research study with one participant could score more than 1 out of 5 on the SMRS design factor. These true N=1 studies comprised 56% of the 169 items we reviewed.
The conclusion must be that ABA researchers who would prefer to design their research so that it has influence in quantitative reviews using methods like NSP should aim to score 5 in all factors in the SMRS. An implication of this is that ABA ASD research may become more expensive if research teams need to buy independent diagnostic services and extra observation time, and include larger numbers of participants. We can only hope that resources will be made available to cover the additional costs of producing high quality, generalisable research.
Limitations of review
Literature search methods
The initial literature searches were conducted using two databases (PsycINFO and Web of Science) using search terms that should have detected all research articles that included ASD or its diagnostic sub-groups provided that one of the search terms appeared in the title, abstract, or keyword fields for the article (see Appendix B). Despite considering more than 16000 titles located during these initial searches, it is likely that reliance on these search methods will have missed relevant articles for two reasons. First, not all potentially relevant research journals are indexed on either PI or WoS. Second, although one or more participants in a study may have been reported to have an ASD in the Method section of an article, that may not have been mentioned in any of the fields searched. Hence, we believe that we will have underestimated the amount of ASD-relevant ABA research that has been conducted. A hand-search of ten volumes of all potentially relevant journals could have been conducted. We ruled this out for the present review when we stopped counting potentially relevant journals when the tally reached 70.
As a result of two factors, the limited literature search and the generally lax approach to reporting diagnostic methods by behaviour analytic researchers (mentioned above), the articles reviewed by NSP and our New Zealand reviewers can be seen as a sample of the relevant literature. Nevertheless, the sample was large enough to draw conclusions about the benefits and deficiencies shown by research on ABA for ASD.
Definition of ABA
Following identification of research articles on interventions for ASD and its subgroups, Appendix B described our methods for identifying research articles as being ABA-relevant. Scrutiny of titles, abstracts and method sections of publications sought to identify types of articles as specified in the Ministries’ Request for Tender (RfT): “The relevant interventions are any interventions or combined approaches using the principles of applied behaviour analysis for the purpose of treating individuals with ASD” (RfT, p.21, in Appendix 2). The principles of ABA were defined by Baer, Wolf, and Risley (1968) and are still generally accepted by behaviour analysts. Briefly, Baer et al. (1968) listed the dimensions of ABA as: Applied, Behavioural, Analytic, Technological, Conceptually Systematic, Effective, and demonstrating Generality. One of the defining features of ABA is that procedures are described in a manner conceptually systematic with the Experimental Analysis of Behaviour (EAB), the basic science of behaviour. Hence, we searched for conceptually systematic terms and, if they appeared in the Method sections, we reviewed those articles as ABA-relevant, provided that no exclusionary criteria were met.
This approach may be questioned by those who are misinformed concerning the definition of Baer et al. (1968). For example, there is a common and incorrect belief in the ASD-interested community of families, carers, and non-behaviourally oriented professionals and paraprofessionals that ABA for ASD equates to discrete trial training (e.g., Lovaas, 1987). There is an erroneous implication in other descriptions of ABA that intervention procedures that can be studied in ABA, and therefore become part of the body of scientific evidence from applications of ABA, can include only discoveries from EAB. An interesting clarifying example was provided by Baer et al. (1968) when they considered “play therapy”. Play therapy may have been justified by play therapists by resorting to the principles of psychodynamic theory, not EAB. However, if the behaviours of the play therapist under study are clearly described so they can be replicated by others and related conceptually to EAB principles, then the therapeutic behaviours studied can conform to the technological dimension of ABA and the study of the play therapist’s behaviour can become part of the knowledge-base for ABA, provided all other dimensions of ABA are satisfied. This does not, however, mean that all activities of any “play therapist” can be justified as being part of the empirically-derived knowledge base of ABA. Another example that arose in our review concerned research publications on training joint attention. Some may be confused because joint attention is conceptually related to cognitive, social and developmental psychology, not EAB. However joint attention (like play) is widely acknowledged to be a socially significant behaviour to improve for children with ASD, i.e., joint attention meets the criteria for “applied” in the sense of Baer et al.. Hence, if joint attention was trained using technologically described procedures conceptually related to EAB principles, we viewed the study as a relevant intervention.
Despite our review team sharing commitment to the Baer et al. (1968) definition of ABA, this did not always allow for complete agreement on whether a study could be included as ABA. For example, the NSP did not provide data from their review of some studies we had located that they considered non-behavioural. Social stories interventions and the use of alternative and augmentative communication devices were two categories that we had to exclude for that reason even if the research had been conducted by behaviour analysts according to the Baer et al. (1968) criteria.
Answering all RfT questions
Table 5 showed how we attempted to provide a best fit between the Ministry of Education’s RfT classifications and the NSP classifications of intervention targets. Anonymous reviewers agreed that we had addressed the RfT questions, although one requested clarification concerning the most obvious mismatch which was between the Ministry’s division of problem behaviours into two categories: (a) “prevention of challenging behaviours and substitution with more appropriate and conventional behaviours”; and (b) “improvement in behaviours considered non-core ASD behaviours, such as sleep disturbance, self mutilation, aggression, attention and concentration problems” (RfT, p. 4). We did not wish to question the Ministry about how to divide particular research articles among these two categories as this would have been laborious for all concerned: for one example, how to decide whether a report on repetitive self-injurious behaviour counted as (a) “challenging behaviour”, or (b) “self mutilation”; and for another, what if aggression were treated functionally by (a) “substitution with more appropriate . . .”, or (b) “improved” by substitution. Hence we proposed the division between categories as shown in Table 5 between (a) “prevention and substitution”, and (b) “reducing challenging behaviours” by other methods. This alteration of the RfT questions was accepted by the Ministry and the subsequent match with NSP categories can be considered as “close”.
Assessing generalisation and maintenance
Results sections 9 and 10 reviewed our findings concerning generalisation and maintenance of the main effects of behaviour change interventions. Although we had concluded that the majority of studies reviewed for these outcome variables showed that generalisation and maintenance had been successfully demonstrated, the findings need to be viewed with some caution. First, agreement between reviewers on the presence and quality of generalisation and maintenance data was relatively low at 68% and 64% for generalisation and maintenance, respectively. From post hoc discussion with New Zealand reviewers and NSP it became clear that there were differences in interpretation of the NSP coding manual concerning generalisation and maintenance. The view of NSP and some of the New Zealand reviewers was that generalisation and/or maintenance could only be coded as being demonstrated if the specific words “generalisation” and/or “maintenance” were used as descriptors for observational sessions that reported on generalisation and maintenance. Other New Zealand reviewers did not apply that rule and noted generalisation as having been assessed in multiple baseline across settings experimental designs where generalisation effects could be detected from close examination of the data-paths for untreated settings. Similar confusion arose for maintenance in studies which faded out treatment or reported “follow-up” data without specifying the term “maintenance”. Incomplete reversion of levels of behaviour to baseline in ABAB withdrawal designs also was viewed as maintenance by some reviewers, but not others. All these views are legitimate, but prior failure to agree on criteria for assessing generalisation and maintenance by reviewers inevitably led to lack of agreement between them.
A second reason for treating generalisation and maintenance data with caution was that NSP did not provide data on generalisation and maintenance effects because, as stated earlier, they did not consider that warranted unless multiple studies were rated with SMRS scores of ≥4. Consequently we were able to report on generalisation and maintenance for 169 of 911 1998-2007 items (18.6%). However, we can report that 45.4% of 616 NSP-reviewed ABA studies from which NSP provided any data on generalisation or maintenance assessed generalisation and/or maintenance.
The New Zealand Context
The term behaviour analysis refers to a scientific discipline conventionally regarded as falling within the behavioural and psychological sciences, but also found in education and some other social sciences (Morris, 1992). Behaviour analysis is both a basic, experimental science (the experimental analysis of behaviour; EAB), and an applied science and technology (applied behaviour analysis; ABA). For behaviour analysis, Science is a systematic approach to the understanding of natural phenomena … that relies on determinism as its fundamental assumption, empiricism as its prime directive, experimentation as its basic strategy, replication as its necessary requirement for believability, parsimony as its conservative value, and philosophic doubt as its guiding conscience. (Cooper, Heron, & Heward, 2007, p 7).
Consistent with this conception of science, EAB studies the behaviour of individuals (both human and non-human) using experimental functional analysis to understand the relationships between environment and behaviour (Delprato & Midgley, 1992; Skinner, 1966). ABA shares this commitment, but specifically is devoted to the understanding and improvement of human behaviour. ABA is defined as a scientific approach for discovering environmental variables that reliably influence socially significant behavior and for developing a technology of behavior change that takes practical advantage of those discoveries. (Cooper, et al., 2007, p 3.).
Behaviour Analysis in New Zealand
EAB began to be established in New Zealand universities in the late 1960’s, with the setting up of laboratories at the University of Auckland and the University of Canterbury. By the late 1970’s EAB research was being actively pursued at the Universities of Auckland, Waikato, Victoria, Canterbury, and Otago. New Zealand researchers have published extensively in the EAB field. For instance, in the period 1968 – 2008, authors with an affiliation in New Zealand have contributed 183 articles (an average rate of six per year) to the Journal of the Experimental Analysis of Behavior, the leading international journal [from a PsycINFO database search, 28 October, 2008, using descriptors “Journal of the Experimental Analysis of Behavior” (journal title) and “New Zealand” (affiliation)].
Given that there have been numbers of university academic staff with research expertise in the field for the past 30 years or more in all NZ universities (except AUT and Lincoln), there have been opportunities throughout that time for university students to take courses in behaviour analysis at both undergraduate and postgraduate levels. This instruction has typically been incorporated as part of wider courses in learning and behaviour change, particularly at the undergraduate level.
From the 1970’s onward there was also growing interest in ABA in New Zealand. Significant catalysts for this were the return of several New Zealanders who had studied ABA in the USA to academic positions in New Zealand (Blampied, 1978). These individuals were successful both in recruiting postgraduate students to do ABA research, many of whom later went on to academic appointments in New Zealand, and also in inspiring colleagues in EAB to collaborate in ABA research and teaching. By the 1980’s there were academic staff in a majority of New Zealand universities teaching and researching in ABA (Blampied, 1999a; Singh & Blampied, 1983). These were found in both Departments/Schools of Psychology, and in Education. Two universities in New Zealand now have postgraduate courses in applied behaviour analysis approved by the Behavior Analysis Certification Board. Graduates of these programmes become Registered Psychologists in New Zealand and may become certified as behaviour analysts, an internationally recognised qualification, but as yet there are only a few such graduates. Many either work or would like to work with ASD populations.
Informal conferences on behaviour analysis had been held in New Zealand from the 1970’s onwards, but in 2004 the New Zealand Chapter of the Association for Behavior Analysis International (NZABA) was established. Current membership is approximately 70 (Leland, L, personal communication, 28 October, 2008), including a substantial number of students as well as individuals who have completed postgraduate training in behaviour analysis, a majority of whom are likely to be interested in EAB rather than ABA. In short, estimating the number of behaviour analysts, or individuals with sufficient knowledge of behaviour analysis, available in New Zealand to provide evidence-based support for services for persons with ASD is hard to do. As a guess, in addition to university staff, the number of behaviour analysis practitioners in New Zealand is probably less than 30, including a small number of New Zealand graduates working in various non-academic settings who have received additional training in New Zealand and overseas specifically in the treatment of ASD primarily using early intensive behavioural interventions (see Keenan, 2006, for more information about ABA and ASD in New Zealand).
There are, however, a much larger number of graduates in psychology and education who are not experts in ABA but have some familiarity with and understanding of applied behaviour analysis, at least at a basic level. Without postgraduate training in ABA, however, they would be unable to design, supervise, and evaluate intervention programmes such as those listed in Results section 11. In fact, it could be considered irresponsible and unethical to encourage people with some basic knowledge of ABA to develop and conduct behaviour reduction programmes based on functional assessment unless they first received additional ABA training. Given opportunities for such training, however, these individuals are a resource that might be deployed to meet the need for ABA treatment in New Zealand.
Research on Applied Behaviour Analysis in New Zealand
Reviews of ABA and related research in New Zealand from the 1970’s onward may be found in Blampied (1978; 1999a) and Singh and Blampied (1983). There appears to have been a small, but reasonably steady rate of publication of ABA research by New-Zealand resident researchers over the past four decades, but much less than for EAB (see above). A feature of much of the published research has been its focus on infants and children. There has been relatively little research on adolescents and adults. How much of this research has been about ASD is somewhat hard to say, both because in many cases reliable diagnoses of participants have not been available (sometimes because privacy law has restricted researchers’ access to diagnostic information) and sometimes because participants with ASD have been included in groups with other diagnostic conditions (e.g., intellectual disabilities).
Interrogation of the PsycINFO database using the descriptors “autism” and “autism spectrum disorder” (as keywords) and “New Zealand” or “NZ” (in author affiliation), and limited to the period 1997 – 2008 yielded 128 journal articles. Inspection of the titles and abstracts (with no attempt made to systematically check the New Zealand association of any authors) permitted 41 (22%) of the articles to be rejected as irrelevant to the search (e.g., they were articles considering general questions of developmental psychology, cognitive psychology, or psychopathology). Of the remainder (where the diagnosis or psychopathology was clearly stated), an almost even number (21 and 18 respectively) were concerned with ASD and Intellectual and/or Learning Disabilities (ID/LD). These were overshadowed by the 42 articles on attention deficit hyperactivity disorder (ADHD) and related conditions. Six articles were in the area of ABA (but not ASD, ADHD, or ID/LD), several featuring methodological issues such as functional analysis.
Behaviour Analysis and the New Zealand Autism Spectrum Disorder Guideline
The Zealand Autism Spectrum Disorder Guideline (NZASDG; Ministries of Health & Education, 2008) was produced for the purpose of providing evidence-based information for health, disability, and education professionals and social service agencies for the provision of services for people with ASD, their families and whanau. (NZASDG, p 12). The Guideline is divided into eight parts: Diagnosis and initial assessment of ASD; Support for individuals, families and carers; Education for learners with ASD; Treatment and management of ASD; Living in the community; Professional learning and development; Maori perspectives; and Pacific people’s perspectives.
With respect to Part 1 of the Guideline, experts in ABA may be, but are unlikely, to be found in any of the settings where children, adolescents, or adults will receive diagnostic services leading to a diagnosis of ASD (or some alternative diagnosis). Experts in ABA, if they are found in such settings at all, are likely to be employed in tertiary health services serving children and families, or in specialist education services (e.g., Group Special Education). There are some private practitioners with ABA expertise who may provide diagnostic services, prior to providing behavioural therapies of one kind or another, generally supplied directly to the family/whanau.
ABA is, in principle, devoted to solving applied problems, and might well, therefore, be applied to problems experienced by any person (e.g., parents) affected by ASD (NZASDG, Part 2). Furthermore, since early intensive behavioural intervention for ASD was developed initially to be delivered in family settings (Lovaas, 1987) extensive practical experience has been developed within ABA for dealing with the manifold problems associated with working in family and community settings (e.g., Jacobson, 2000; Keenan, 2006; Maurice, Green, & Luce, 1996). Nevertheless, the focus of ABA research has been on the individual with the diagnosis of ASD, rather than on caregivers, teachers, or family members.
The science and practice of ABA has, therefore, both internationally and in New Zealand been focussed on those areas covered in Parts 3 – 6 of the NZASDG, especially on education, treatment, and management of ASD. While the majority of research in New Zealand (and probably elsewhere) has been focussed on children, ABA is in principle applicable throughout the lifespan, and might be applied to solve problems for any age group, with any class of behaviour, and in any context or setting. It is potentially applicable to all the areas covered by the Guideline.
In several places the NZASDG identifies the need for research, and makes recommendations that research on various matters be carried out. A highly relevant contribution that ABA can make to help fulfil these recommendations is through the provision of single-case research designs that were developed within behaviour analysis (Church, 1996; Hersen & Barlow, 1976). ABA practitioners are uniquely qualified to design and conduct such research. Note that these research designs are not limited to evaluation of behavioural interventions. Any kind of intervention (including those without any pretensions of being based on scientific principles) may be evaluated, with the advantage that the evaluation is scientifically rigorous, and permits the drawing of causal inferences about the outcome of the interventions, but without requiring the recruitment of large numbers of participants, something that is often impractical or very difficult (Blampied, 1999b). Experts in ABA are a resource for professional learning and development in understanding and using single case designs. Further, it is in the context of providing research at the level of individuals and their whanau that ABA can probably make the greatest contribution to Maori and Pacific Peoples in the context of ASD.
Where to find out more
For more information on this publication or Special Education services, please email: Special Education Mailbox