Enhancing and Igniting Talent Development Initiatives: Research to determine effectiveness

Publication Details

This report presents the findings from research involving five Talent Development Initiatives for improving outcomes for gifted and talented students or their teachers. The purpose of the research was to consider how well the objectives of each participating initiative had been achieved, how the initiative contributed to improved outcomes for gifted and talented learners or their teachers, and how planning to continue to meet the learners’ needs after 2008 had been considered.

Author(s): Tracy Riley and Roger Moltzen, Massey University. Report prepared for the Ministry of Education.

Date Published: March 2010

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  To view the individual chapters please refer to the 'Sections' inset box.  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Section 3: Enhancing and Igniting Talent Development Initiatives

This section of the report provides a detailed account of the research processes, including a justification for selecting participatory action research as a method of evaluation. It also details the process of selecting TDIs for the evaluation, and a divergence from the planned approach to include case study research for the Enhance programmes. The limitations of using different methodologies are also discussed.

Participatory Action Research of Multiple Case Studies

The proposed methodology for the evaluation of the TDIs was participatory action research using multiple case studies and employing both qualitative and quantitative data collection methods. Because each TDI is unique, with different programme outcomes and processes, it was important that while there was some consistency in the overall research framework, there was also a tailored evaluation plan for each individual programme's evaluation. The key elements of the research design were:

  • Inclusion of stakeholders in the evaluation process.
  • Evaluative analysis of each individual programme, but with generalisations made, where applicable, across programmes.
  • Employment of multiple methods and measures of data collection and analyses.
  • Facilitation of formative and summative accounts of effectiveness from different stakeholders' perspectives.

With these design elements in mind, and cognisant of the principles, practices, and issues in the evaluation of gifted and talented programmes, the researchers decided to employ participatory action research to evaluate the TDIs. Basically, research of this nature can be defined as learning by doing. It is "a systematic inquiry by collaborative, self-critical communities … out of the need to improve educational knowledge and practices" (Watts & Watts, 1993, p. 36). Some of the principles underlying participatory action research are collaboration and participation, empowerment, knowledge, and social change (Grundy, 1982). These principles align well with the purposes and nature of the TDI programmes, especially as new initiatives in New Zealand.

The implementation of a participatory action research model is contingent upon authentic participation by "communities of people committed to enlightening themselves about the relationship between circumstance, action, and consequence" (McTaggart, as cited by Wadsworth, 1997, p. 70). Collaboration amongst researchers and programme stakeholders was well-suited to the process for the selection of programmes, as was outlined in the Ministry's Request for Proposals (2005), but more importantly allowed programme stakeholders greater input into the direction and content of the evaluation. The researchers also felt that using this model allowed for the development of research teams of academics and practitioners from throughout New Zealand, bringing together appropriate knowledge and skills in specialised areas of gifted and talented education.

Additionally, participatory action research yields not only a set of final results and conclusions (summative), but also opportunities for ongoing feedback to participants (formative). "Action research is the way groups of people can organise the conditions under which they can learn from their own experiences and make this experience accessible to others" (McTaggart, cited in Wadsworth, 1998). This is because action research proceeds in a spiral of steps composed of planning, action, and an evaluation of the result of the action (Kemmis & McTaggart, 1998). The action research spiral requires research teams (comprised of researchers and programme stakeholders) to:

  1. Develop a plan of action (evaluation of outcomes and processes).
  2. Act to implement the plan (carry out the evaluation).
  3. Observe the effects of the action (systematically observe and document the evaluation).
  4. Reflect on the action and plan further action (group reflection for further planning of the evaluation).

Using this approach to evaluation is both responsive and illuminative, and it aligned well to the purposes of the TDI evaluation: "to inform the ongoing work of the groups involved" (Request for Proposals, 2005, p. 2). As a formative evaluation, it would allow for building upon emergent themes, reporting back the process to participants, problem-solving, and new action planning processes, as is advocated in the gifted and talented education literature previously discussed. The benefit of this approach is that the research findings are constantly being reported back to those involved, allowing for judgement of the worth of the programme while it is forming or happening. It also allows for the programme to be shaped based on an ongoing evaluation, rather than a 'too-late' evaluation.

This type of evaluation takes into account the ongoing, interactive process of programme development and implementation (including internal evaluation processes), as well as the perspectives of many different stakeholders. This was of importance in evaluating the TDIs, especially in light of their innovation and 'infancy' – it was critical the evaluation provided a holistic account of each programme, based upon a variety of viewpoints. This would allow the researchers to discover and document how a programme works from the experiences and perspectives of its stakeholders, by using a fluid methodology to unearth important factors and issues. Additionally, an evaluation of this sort requires participants to build records of their improvements. Thus, the researchers believed this design also had the potential to enhance the documentation and dissemination explaining the evolution of each TDI by including elements related to effective programme evaluation.

The unique nature of each TDI could be evaluated using a single case study approach. Yin (1984) described case study research as "an empirical inquiry that investigates a contemporary phenomenon within its real-life context; when the boundaries between the phenomenon and context are not clearly evident; and in which multiple sources of evidence are used" (p. 23). Case study research provides rich, thick descriptive answers to two questions: how and why? These were important questions in regards to the research purposes, and their answers would actually be more robust if based upon multiple case study design. Multiple case study design treats each case as a single case, but each case's conclusions are used as information contributing to the whole study. Given the overall evaluation purposes for the TDIs were similar, some generalisations regarding effective programmes for gifted and talented students were likely to arise. The researchers proposed that given the paucity of New Zealand-based programme evaluations, a multiple case design could also allow for participatory action research to be tested as a model of programme evaluation.

From the outset, the researchers were aware that the methods of data collection would vary across programmes, but were likely to include individual and focus group interviews, document analysis, observations, and programme-based student assessment data. The involvement of stakeholders would also vary dependent upon each programme, but could include programme coordinators, specialist teachers, community members, school administrators, parents, and gifted and talented students. The proposal for this research stated, "… the overall research procedures must remain flexible … The final decisions regarding the measures of effectiveness will be reliant on each programme's determined goals and objectives and agreed methods of measurement" (2005, p. 6). Furthermore, as the research cycles were enacted, it was likely that the research procedures would be adapted or changed to enhance the overall evaluation. This need for flexibility and responsiveness characterised the three years of the TDI evaluations, beginning with the selection of TDI programmes.

Research and Selection Processes

The research directors were invited to attend an all-day meeting of TDI programmes selected for the 2006-2008 funding round hosted by the Ministry of Education in August 2005. The purpose in this meeting was two-fold: to allow the researchers an opportunity to present an overview of the evaluation; and to assist all the TDI coordinators in clarifying their programme objectives, outcomes, and methods of monitoring and evaluation (through whole group sessions and limited group activities). This meeting also gave TDI providers an opportunity to share their TDI aims and aspirations and a chance for meeting one another, the Ministry representatives, and the researchers. The research directors were able to gain some further insights into each programme's aims and purposes, adding to the written documentation summarised by the Ministry from each programme's proposal.

Following this introductory session, the research directors met with the Ministry of Education's Senior Adviser in gifted and talented and a member of the Research Division to determine the criteria for the selection of four TDIs to participate in the evaluation. The over-arching principles for selection were:

  1. The evaluation should be focused on a range of different outcomes for students. This would mean that the evaluation may only focus on certain aspects of some programmes. Furthermore, the evaluation should be focused on the programme in action, and its related outcomes for gifted and talented students, as opposed to professional development elements.
  2. Proposals that demonstrated a sound understanding of theory and research, were based on an established commitment to gifted and talented education, showed innovation, and reflected the core principles (Ministry of Education, 2002) would be considered.

The priorities identified in the call for TDI proposals were also considered and these included programmes addressing diverse cultures, multi-categorical definitions; potentially under-represented groups; demographic factors; decile; sectors/levels; cross-sector partnerships; continuum of approaches; including regular classroom provisions; continuum of outcomes, including social, emotional, academic, and cultural; and home-school partnerships. Pragmatics such as geographical issues, the strengths and weaknesses within the research team, costs, and conflicts of interest were also discussed. From these discussions four Ignite TDIs were selected which the researchers felt met the criteria and overall allowed for a range of priorities to be addressed.

The Ministry of Education, however, requested that two Enhance programmes be included in the four to be evaluated. The rationale for this was explained as, "TDI Enhance groups are those who have had prior experience in the TDI network and offer opportunity for researchers to explore innovation in progressive stages and the link between longer-term change and student outcomes" (personal communication, Senior Adviser in gifted and talented, 2 September 2005). After ongoing discussions with the Ministry, the contract for the evaluation was renegotiated to reflect changes in the research sample, methodology, and purposes:

  • Three Ignite programmes evaluated using a participatory action research approach, as originally proposed: Rutherford College; New Zealand Marine Studies Centre; and Te Manu Aute. The purpose in the Ignite evaluations was to determine the effectiveness of programmes on outcomes for students (as opposed to other stakeholders). A team comprised of one research director and two researchers would conduct each action research-based case study, with the TDI coordinators, based on two three-day visits per annum.
  • Two Enhance programmes evaluated using a case study approach: the professional development programmes of the Gifted Kids Programme (GiftNet) and the Gifted Education Centre (formerly the George Parkyn Centre). The purpose in the Enhance evaluations was to determine the effectiveness of professional development on outcomes for teachers (as opposed to other stakeholders). One of the research directors would conduct each case study with one two-day visit per annum.

For both the Ignite and Enhance evaluations, a multiple case study approach was used, but the analysis was evaluative in relation to each individual programme's intended goals, with no cross-analysis or comparative analysis, particularly in relation to outcomes. There are some generalisations that can be made in relation to programme development, implementation, and evaluation.

With the evaluation of Ignite and Enhance TDIs, the research questions needed to be refined and tailored, as many related to programmes targeting gifted and talented students. The researchers achieved this by focusing the evaluation on three overarching purposes, and it was these purposes that were shared with the participants in the research.

  • To determine how providers design, implement, maintain, and evaluate programmes for gifted and talented students, or their teachers.
  • To determine how providers structure relevant and engaging learning and growth opportunities for gifted and talented students, or their teachers, as evidenced in the achievement of programme objectives; improved outcomes for students, or their teachers; impact upon key stakeholders; and planning for sustainability.

And for the Ignite programmes:

  • To determine how, and what, formative feedback effects the development, implementation, maintenance, and evaluation of programmes for gifted and talented students, or their teachers, by using an action research approach to evaluation.

The specific research questions, outlined by the Ministry of Education are shown in Table 1 on the following page. As the table shows, there were additional research questions for the Ignite programmes, which were evaluated using an action research approach. The specific questions for the Ignite and Enhance programmes are also listed in the introductory sections of each set of evaluations and answered in each section summary.

Table 1: Research Purposes and Questions
Note:
  1. Question 1 only.
Research Purposes and Questions Ignite TDIs Enhance TDIs

To  determine how providers design, implement, maintain, and evaluate  programmes for gifted and talented students, or their teachers.

How  were decisions around the programme design arrived at? Who was involved  in the decision-making process? How has the process impacted the  sustainability of the programme?
What  changes in climate and philosophy have been required for the successful  implementation of this programme? How have professional leaders  approached the task of climate change, how were these changes managed,  and how were changes in practice achieved? 1
How  comprehensive are provider initiated programme monitoring and  evaluations? How do the findings of the monitoring or evaluation inform  the programme?
To  determine how providers structure relevant and engaging learning and  growth opportunities for gifted and talented students, or their  teachers, as evidenced in the achievement of programme objectives;  improved outcomes for students, or their teachers; impact upon key  stakeholders; and planning for sustainability.
How appropriate were the identification procedures, curriculum adaptations,  and forms of assessment in relation to the goals of the programme? How  has this contributed to student outcomes?
What  aspects of curriculum differentiation have been designed specifically  to meet the major objectives of the programme? To what extent has this  specific programme design contributed to improved student outcomes?

What  is the evidence for improved student learning and social, emotional or  cultural outcomes as a result of participation in the programme?


How have resources and personnel impacted on the success or otherwise of the programme?
What role has staff professional development played in achieving the programme goals?
How  well has the programme planning occurred in regard to sustainability of  the programme after the three-year period of funding ceases?
What has been the impact of the programme for its stakeholders?
To  determine how, and what, formative feedback effects the development,  implementation, maintenance, and evaluation of programmes for gifted  and talented students, or their teachers, by using an action research  approach to evaluation.

Limitations

In essence, the changes from the initial proposal to the renegotiated contract means that this report is comprised of two different evaluation projects: the Enhance evaluations and the Ignite evaluations. The different approaches used in the evaluation mean that generalisations across all five programmes should be cautiously made. The Ignite focus on outcomes for students and the Enhance focus on outcomes for teachers led to different research questions and data collection methods. Generalisations, let alone, conclusions, related to outcomes across the Ignite and Enhance programmes cannot be made; however, some learnings are generalisable. The development, implementation, and evaluation of programmes for students and professional development programmes for teachers have different aims, processes, purposes, and outcomes.

It is also very important to remain cognisant of the fact that this research only evaluated five of some 38 TDIs funded by the Ministry of Education since 2003. Of these five, only three of the 16 Ignite programmes for the 2006-2008 funding period were evaluated; two of the 5 Enhance programmes were evaluated. Therefore, the results reported here cannot be generalised to all TDI programmes for gifted and talented students, or their teachers. Furthermore, given the unique nature of gifted and talented education programmes and professional development in New Zealand, the findings cannot be generalised to all opportunities. This evaluation did not examine the processes and structures of the Ministry of Education in relation to the TDI funding pool, but shares the perspectives of five TDI programme's stakeholders. It was beyond the scope of this research to review the TDI funding pool, and any common themes arising from this evaluation should not be generalised beyond its scope and purposes.

Readers of this report need to keep these limitations in mind, though every effort has been made by the researchers to ensure quality. The use of multiple sources of data, collected, analysed, and member checked by researchers and TDI coordinators, allowed for a chain of evidence, and ensures construct validity. Internal validity could be checked through analysis of qualitative and quantitative data both within, and, to a lesser degree, across cases. Within each case, there was consistency in the data collection techniques. Other factors that aim towards ensuring quality are: the triangulation of data collection tools and multiple data sources; the involvement of research team and research advisory group members with different areas of expertise and skill in gifted and talented education; adherence to ethical codes of conduct; and ongoing consultation and discussions with the programme providers and the Ministry of Education. The TDI providers were also given the opportunity to view the findings reported, correcting any factual information or commented where they considered something to be misconstrued or misinterpreted. Finally, the researchers aimed for integrity in the reporting of the research findings, providing rich, descriptive accounts of each TDI's evolution.

Summary

This section has shown that participatory action research using multiple case studies is potentially a powerful tool for the evaluation of innovative programmes such as the Ignite TDIs. Three Ignite TDIs, Rutherford College, Te Manu Aute, and the New Zealand Marine Studies Centre, were evaluated over three years using this action-oriented focus. This enables their stories to be told, as the programmes developed and evolved. The Enhance programmes were already-established, previously funded TDIs, and therefore, the Ministry of Education felt it important to determine their effectiveness. The professional development programmes of the Gifted Education Centre and Gifted Kids Programmes were evaluated using a case study approach. There are limitations to the evaluation of these two different samples of TDIs, using different methodologies and approaches – namely, caution should be taken in making generalisations within or across the cases reported here, to other TDI programmes, the Ministry of Education funding of TDIs, or programmes in schools and communities. The following sections of the report describe each individual case, beginning with the Enhance evaluations, and followed by the Ignite evaluations. For each set of cases, conclusions are drawn and recommendations made, when appropriate.

Footnotes

  1. Each TDI was engaged in its own internal evaluation process, as well as the external evaluation described in this report.