Enhancing and Igniting Talent Development Initiatives: Research to determine effectiveness

Publication Details

This report presents the findings from research involving five Talent Development Initiatives for improving outcomes for gifted and talented students or their teachers. The purpose of the research was to consider how well the objectives of each participating initiative had been achieved, how the initiative contributed to improved outcomes for gifted and talented learners or their teachers, and how planning to continue to meet the learners’ needs after 2008 had been considered.

Author(s): Tracy Riley and Roger Moltzen, Massey University. Report prepared for the Ministry of Education.

Date Published: March 2010

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  To view the individual chapters please refer to the 'Sections' inset box.  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Section 2: The Evaluation of Gifted and Talented Programmes

This section of the report provides an overview of the theory and research related to evaluating programmes for gifted and talented students, demonstrating the relevance of this evaluation of TDIs. It begins by outlining the results of recent research in New Zealand, and then contextualises this within the broader international field of gifted and talented education. Principles and practices relevant to evaluating programmes for gifted and talented students are explored, highlighting some problematic issues.

Evaluating Gifted and Talented Programmes in New Zealand

Research commissioned by the Ministry of Education concluded, "There is a paucity of reported national or international research which evaluated the effectiveness of provisions for gifted and talented students in relation to social, cultural, emotional, creative, and intellectual outcomes" (Riley, Bevan-Brown, Bicknell, Carroll-Lind, & Kearney, 2004, p. 3). The researchers reported that formal evaluation and dissemination of identification and provisions for gifted and talented students is seldom undertaken in New Zealand, leaving educators to adapt or adopt many of the models, methods, and programmes reported in the international field of gifted and talented education, without a critical, research-driven analysis of their appropriateness or effectiveness within the cultural, social, and educational climate of this country. While there is a growing body of literature and resources in New Zealand, including those produced by the Ministry of Education, to date most of this has been descriptive reports of practices, as opposed to empirical research of programme effectiveness. The researchers predicted that the evaluation of programmes "would enable better decision making" (Riley et al., 2004, p. 270) at a local and national level.

In 2008, the Education Review Office (ERO) released a report evaluating the provisions for gifted and talented students in 315 schools reviewed in 2007. Their report mirrored the research findings of Riley and her colleagues (2004), specifically in regards to evaluation. ERO probed how well schools reviewed the effectiveness of their provisions for gifted and talented students, outlining the following indicators of effective practice:

  • Systematic and ongoing processes for evaluating the outcomes for students;
  • Sharing and consultation about evaluation findings with staff, parents/whanau, students, and the community;
  • Actions based on the recommendations of the evaluation; and
  • Evaluating the impact of programmes and provisions, both internal and external to the school (ERO, 2008).

Their report shows that only 23 percent of schools had highly developed or developed evaluation processes; nearly half the schools (46%) had not developed any such processes. While some schools were evaluating both in-school and out-of-school programmes, these were often limited to classroom programmes in reading, writing, and mathematics, sometimes supported by anecdotal information about cultural or sporting gifts and talents. Some schools had collected information about school-based programmes, but this, too was limited. These figures are similar to those reported in the analysis of questionnaire responses provided by Riley et al. (2004) whereby only a quarter of schools (314) reported evaluation and monitoring as part of their school-based documentation to support gifted students. However, the survey did not extend to an investigation of the quality or nature of these policy documents or the evaluation measures being used. The case study schools in this study showed some evidence of the evaluation of withdrawal programmes and specialised provisions, but unlike ERO's sample, limited review of classroom-based programmes (Riley et al., 2004).

Riley et al. (2004) stated in their conclusion that evaluation enhanced informed decision-making about student programmes; ERO (2008) reported a strong correlation (of statistical significance) between school-based evaluations and the responsiveness and appropriateness of provisions for gifted and talented students. Interestingly, regardless of ERO's evaluation, three-quarters of schools reported (in their own self-review processes for ERO), the majority of their provisions for gifted and talented students as making significant contributions, or at least some contribution (with some recognised need for improvement) to meeting the needs of gifted and talented students. ERO (2008) concluded, "The lack of school self-review culture hindered schools' ability to ascertain how well they were providing for gifted and talented students" (p. 52).

These two studies demonstrate a lack of planned evaluations to inform provisions for gifted and talented students in New Zealand, but this is equally the case world wide. Evaluation of programmes is one of the most neglected areas of gifted and talented education (Callahan, 2001; Tomlinson & Callahan, 1994; Reid, 2004; Riley et al., 2004). There are many reasons reported for the paucity of effective evaluation processes, including weakness in evaluation skills of teachers, lack of time and funding, complex problems posed in appropriately evaluating the multi-pronged outcomes of gifted programmes, and a fear of public discussion of programming for gifted students, especially where funding is tenuous (Tomlinson, Bland, Moon, & Callahan, 1993).

As VanTassel-Baska (2004a) explains, the lack of programme impact data, collected through both summative and formative evaluations, has, in part, led to "gifted programs in a stalemate based on lack of sufficient infrastructure" (p. 228). She reports a lack of "deep program development and implementation" (p. 228) which can potentially lead to superficial programmes. This is why, within New Zealand, the Ministry of Education (2000) considers programme evaluation as a "necessary aspect of gifted education" (p. 58), but not an element unrelated to a comprehensive programme comprised of definitions, identification, and differentiated programme goals, provisions, and learning opportunities. Programme evaluations should, therefore, be designed to determine the effectiveness of all of these elements.

Principles and Practices

The indicators of quality in provisions for gifted and talented students have been outlined by Maker (1993) as follows:

  • Appropriate: differentiated provisions match to individual differences in gifted and talented students.
  • Articulated: long-term, monitored, and comprehensive planning.
  • Clear: clarity of all interrelated elements of the programme which is regularly and openly communicated and shared with stakeholders.
  • Consistent: reflects philosophy of programme context (e.g., school) and demonstrates interrelationships amongst programme components.
  • Comprehensive: utilises a continuum of approaches which will meet cognitive, affective, physical, social, emotional, and cultural needs.
  • Responsive: flexibility of programme based upon ongoing evaluation.
  • Unique: driven by uniqueness of individual gifted and talented students and fitted to his or her needs.
  • Valid: based upon theory and research-driven models, strategies, and so on, and continually evaluated for effectiveness.

Riley et al. (2004) also highlighted an important quality indicator as the cultural appropriateness and relevance of all aspects of gifted and talented programmes. Other important indicators, particularly for determining the quality of the TDIs, are outlined in the core principles advocated by the Ministry of Education (2002). These principles align well with Maker's (1993) quality indicators, but also highlight other indicators to consider, namely the need for inclusive and bicultural provisions for New Zealand's gifted and talented students. What New Zealand's principles do not address, as explicitly as the quality indicators outlined by Maker (1993), is the importance of documentation and evaluation for continual programme improvement. These are, however, outlined in the Ministry of Education's (2000) handbook for gifted and talented education. Pulling together the quality indicators alongside the core principles results in the following set of benchmarks for evaluating the effectiveness of gifted and talented programmes in New Zealand:

  • Appropriate, Unique, and Consistent: Schools should aim to provide all learners with an education matched to their individual learning needs.
  • Inclusive: Gifted and talented learners are found in every group within society.
  • Bicultural: Māori perspectives and values must be embodied in all aspects of the education of gifted learners.
  • Articulated, Comprehensive, and Responsive: The school environment is a powerful catalyst for the demonstration and development of talent.
  • Inclusive, Clear, and Unique: Parents, caregivers, and whanau should be given opportunities to be involved in decision-making regarding their children's education
  • Valid: Programmes for gifted and talented students should be based upon sound practice, taking into account research and literature in the field.
  • Appropriate and Comprehensive: Gifted and talented students should be offered a curriculum rich in-depth and breadth, and at a pace commensurate with their abilities
  • Unique, Comprehensive, and Appropriate: Schools should aim to meet the specific social and emotional needs of gifted and talented learners.
  • Appropriate and Valid: Provision for gifted and talented students should be supported by ongoing high-quality teacher education.

Analysis of these indicators shows that the evaluation of gifted and talented programmes measures two elements in conjunction with one another: the outcomes for students and the programme's effectiveness (Taylor, 2000). These two elements are linked because the outcomes for students contribute to the overall effectiveness of programmes. Gallagher (1998) refers to these as management objectives, the pragmatic steps in each programme's implementation, and programme objectives, the specific outcomes for students. In New Zealand, Taylor (2000) recommends programme evaluation that considers input (i.e., resources), process (i.e., identification and differentiated programmes), and output (i.e., student and school achievement of goals and objectives). By analysing each of these elements, evaluators should be able to provide a strong basis for decision-making and future directions. Therefore, evaluation of gifted and talented programmes should be ongoing, utilising both formative and summative approaches.

There are different types of programme evaluations recommended: incoming evaluations, or needs analyses; transition evaluations (when students move from one programme to another); year-end evaluations; and on-going evaluations (National Association for Gifted Children, 2003). As VanTassel-Baska (2004b) points out, it is important that evaluation work be timely – too-early can inhibit the testing of innovation, whereas, if too-late, formative data can be lost. Ongoing evaluations are most appropriate for programmes like the TDIs. Of the different models for programme evaluation described by VanTassel-Baska (2004b), case study; ultilisation-focused; client-centred; and context, input, process, product approaches – or an eclectic mix of these – best encapsulate an ongoing evaluation of innovative programmes for gifted and talented students.

VanTassel-Baska (2004b) explains these different approaches as follows. Case study approaches allow evaluators to understand the programme deeply and holistically, providing both descriptive data, as well as its intended and actual outcomes. Case study research provides rich, thick descriptive answers, from multiple perspectives, to two important questions: how and why? Utilisation-focused approaches are aimed at creating positive programme changes, meaning evaluation is seen as part of the programme's development and implementation, rather than acting in isolation. Client-centred approaches to evaluation are responsive to the needs of those who support gifted and talented programmes (e.g., teachers, administrators, programme coordinators). In a sense, when client-centred approaches are used, those who coordinate, develop, and implement programmes become part of the 'research team' with opportunities to shape evaluation questions and methods, for direct involvement and input in the evaluation, and to provide feedback and input into final reporting. Finally, a context, input, process, product approach aims to determine the extent to which programme goals have been met, by examining the four elements as evidenced in programme planning, documentation, and procedures. The evaluation of the TDIs would ideally incorporate all these elements.

Employing such an eclectic approach means that the evaluation of gifted and talented programmes involves multiple data sources analysed both qualitatively and quantitatively. The data collection methods could include interviews (both individual and focus group), document analysis, observations, surveys, and programme data related to student assessment. Using multiple data collection methods and analyses provides evaluators with opportunities to triangulate data sources, substantiating findings through the identification of themes, patterns, and issues. Gathering data from multiple sources also requires collaboration with and input from key stakeholders.

Thus, the employment of these approaches should be undertaken using a team approach, inclusive of the many stakeholders in gifted education (Ministry of Education, 2000; Reid, 2004; Tomlinson & Callahan, 1994; VanTassel-Baska, 2004b). As the Ministry of Education (2000) points out, "The use of a cooperative team approach is helpful because it allows an evaluation to be worked out together and evaluation tasks to be shared" (p. 54). The evaluation of gifted and talented programmes needs to allow opportunities for involvement of people within and outside of an organisation: parents and whanau; community members; teachers; administrators; and, of course, gifted and talented students. Tomlinson and Callahan (1994) also recommend the inclusion of 'qualified evaluators' – individuals with experience and knowledge in both the evaluation process and gifted and talented education. An inclusive approach better ensures support for the evaluation, better shared understandings of the findings, and implementation of the recommendations (Tomlinson & Callahan, 1994). As VanTassel-Baska (2004b) states, "Involvement increases relevance, understanding, and ownership of the evaluation, all of which facilitate informed and appropriate use" (p. 20). An inclusive, team approach to evaluating innovative programmes, like the TDIs, could potentially generate findings relevant and applicable to the TDI itself, but also understandings transferable to other gifted and talented education initiatives.

The evaluation of programmes for gifted and talented students will potentially generate a multitude of purposes and questions, and these will be of greater or lesser importance to different stakeholders (Reid, 2004). For example, teachers may be most interested in the effectiveness of instructional approaches, parents may want to know if the programme is positively impacting their son or daughter's growth, and administrators might query the programme's financial viability. It is important at the outset that the evaluation purposes be determined and shared. In a best case scenario, the evaluation would be planned at the outset of programme development, when outcomes are being established (Tomlinson & Callahan, 1994). In the case of the TDIs, the major stakeholder in the establishment of the research purposes was the Ministry of Education, but as the research questions were probed, it was equally important to include the TDI programme directors' and external evaluators' perspectives.

Merging all of these considerations means that evaluations of gifted and talented programmes are premised on three core beliefs, outlined by VanTassel-Baska (2004c):

  • The fundamental role of evaluation is to provide information that can be used to improve and advance the state of the art of gifted programmes;
  • Evaluation research is a collaborative process among stakeholders; and
  • The use of multiple data sources helps to illuminate the complexity and salience of issues needing consideration (pp. 23-24).

These beliefs underpin evaluations that are part of an overall programme development cycle. Given the innovative, experimental nature of the TDIs, these should be important considerations in their evaluation. The stages of such an evaluation are outlined by Tomlinson and Callahan (1994):

  1. Planning the evaluation should begin during programme development.
  2. Designing data collection methods and analyses, ensuring a match with evaluation goals and questions.
  3. Conducting the evaluation.
  4. Reporting findings to appropriate audiences and following up on recommendations.

VanTassel-Baska (2004c) believes that until there is interplay between planning, implementing, and evaluating programmes, there will be no programme improvement for gifted and talented students. She has created a dynamic model for gifted programme improvement with four cyclical stages: planning; doing; studying; and acting. In other words, the first stages are planning the programme or curriculum, followed by implementing it, evaluating the results, and creating a plan of action or improvement which feeds back into the next cycle of planning, doing, studying, and acting.

Issues in the Evaluation of Gifted and Talented Programmes

As the above sections highlight, evaluating gifted and talented programmes is an essential, but complex, aspect of their development, but it is not without problems. The limited published empirical research demonstrating the effectiveness of gifted and talented programmes may stem from these difficulties, but is exacerbated by deeper problems within the field.

Firstly, there are differences in the ways in which giftedness and talent are defined, and, subsequently, the identification tools utilised for determining one's abilities. This is made more complex by recognition of the broadening, multi-dimensional, and dynamic nature of giftedness and talent, determined by multiple measures. What this means, in regard to evaluating programmes, is that difficulties arise in making generalisations or comparisons across evaluations, creating what Ziegler and Raul (2000) refer to as a "more or less fragmented" research field. Obviously, there are implications for the evaluation of TDIs, which are guided by New Zealand's philosophy that each school or organisation create, adapt, or adopt their own definition of giftedness (Ministry of Education, 2000). Caution must be taken in the interpretation, integration, and application of evaluation findings across the five TDIs evaluated in this report. Equally important, the findings of this report cannot be generalised to all TDIs or all programmes for gifted and talented students.

Secondly, the outcomes of gifted and talented programmes are multi-faceted, reflecting not only cognitive, intellectual, or academic development, but also social, emotional, affective, and cultural growth. Some intended outcomes of gifted and talented programmes are difficult or impossible to measure by using standard assessment practices – particularly those related to social, emotional, affective, and cultural development. Equally, outcomes are often multi-dimensional and short- or long-term. For example, measuring academic outcomes is not as simplistic as administering a pre-post achievement test. The test may not measure the specific academic outcomes, particularly when these are advanced, complex, or integrated. Gifted students may experience the ceiling effect, whereby they easily reach the upper-limits of the test, with their score potentially masking their actual degree of ability.

Therefore, in evaluating outcomes for students, it is recommended that multiple and varied measures of assessment be utilised (VanTassel-Baska, 2002) and decided upon at the time of designing the outcomes (Winebrenner, 2000; VanTassel-Baska, 2002). Furthermore, if outcomes are in-depth, complex, integrated, process-oriented, and oriented to real-world problems and solutions, alternative assessment, including portfolios, performances, and self or peer evaluation, needs to be considered (Callahan, 2001; McAlpine, 2000; Callahan & Moon, 2003; VanTassel-Baska, 2002; Winebrenner, 2000). Feng and VanTassel-Baska (2004) also recommend the use of off-level assessment and product assessment tools. In addition to these assessment measures, evaluations of gifted and talented programmes may also include observations, behavioural and affective scales (e.g., self-concept, social skills), critical thinking tests, and longitudinal data of ability development (e.g., course selection, competition results) to determine the impacts (Feng & VanTassel-Baska, 2004).

Given the complexity of measuring student outcomes, it is not surprising that in New Zealand, recent evidence shows that teachers struggle to demonstrate gifted and talented students' progress from a range of assessment information (ERO, 2008). This is yet another limitation in evaluating TDIs: as innovative programmes, often setting off on a 'new' path in their provisions for gifted and talented students, measures may not have been determined at the time outcomes were decided. There is also the potential for outcomes to be overly ambitious, to develop over time as the programme is implemented, and to alter or change as a result. Since these were Ministry of Education funded programmes, student outcomes were also influenced by the Ministry and its objectives. Further to this, the researchers were not involved in the development or negotiation of programme outcomes nor their indicators of effectiveness.

A third issue in the field is the diversity of approaches to teaching and learning along a continuum of provisions. What this means, in regards to evaluating programmes for gifted and talented students, is that any growth or development of outcomes cannot be directly related to only one provision, opportunity, or intervention. An evaluation of a solitary provision, in isolation of other opportunities, will not include the many other variables which may be impacting upon student outcomes. This is particularly magnified in the evaluation of gifted and talented programmes, which are often characterised as piecemeal, part-time programmes that may or may not complement other learning opportunities. Also, the different approaches to provisions make it difficult to compare the effects upon students (Slavin, 1988) – a student in a part-time enrichment programme experiences something completely different to a student in a mentoring programme. The goals, objectives, delivery, and so on of different approaches make comparisons of outcomes across programmes untenable. It would be unwise to view the impact of the TDI outside the context of other learning opportunities for gifted and talented students, or to generalise the impact of five programmes to all TDIs. However, some generalisations may be made regarding the development, implementation, and evaluation of gifted and talented programmes, despite the different approaches to teaching and learning. This is an important element of evaluating the TDIs.

Finally, it cannot be assumed that there is a direct 'cause-effect' relationship between an educational programme and its impact upon student outcomes. Attempts to pinpoint a direct cause-effect relationship would be somewhat futile. Each individual student's motivation, personality, family, relationships, teachers, opportunities, attitudes, resources, support, experiences, and so on will play a part in his or her development. Taking all these individual differences into account, alongside the necessary processes of evaluation, would be unwieldy and problematic. Winner (1996) explains that the only way one could demonstrate that gains experienced by students were due to the programme, and not down to individual differences, would be by random assignment – identifying students as gifted and talented, creating a control and treatment group, and evaluating the impact of the intervention. As Reid (2004) points out, research of this nature creates ethical dilemmas. This would also conflict with the principles of gifted and talented education in New Zealand, and so, the purposes in the TDIs. Therefore, in evaluating the TDIs, it is important to take individual student differences into consideration, employ appropriate means of assessment and evaluation, avoid broad sweeping generalisations or cause-effect conclusions, and acknowledge the limitations of each individual programme evaluation.

Summary

This section has highlighted key factors in evaluating gifted and talented programmes. The lack of planned evaluations to inform provisions in New Zealand will continue to hinder our progress of the field. There are a number of guiding principles for the evaluation of gifted and talented programmes: the measurement of both student outcomes and programme effectiveness; the utilisation of a variety of qualitative and quantitative data collection methods matched to programme goals; and the involvement of key stakeholders in both planning and undertaking of the evaluation. Of critical importance is the dynamic relationship between programme development and evaluation, with each informing and shaping the other. Potential problems surrounding the evaluation of gifted and talented programmes stem from broader issues in the field, and any evaluation needs to be cognisant in addressing these through both its methodology and analysis, framed against acknowledged limitations. For these reasons, the proposed methodology for the evaluation of the TDIs was participatory action research using multiple case studies, as the next section of the report explains.