Enhancing and Igniting Talent Development Initiatives: Research to determine effectiveness

Publication Details

This report presents the findings from research involving five Talent Development Initiatives for improving outcomes for gifted and talented students or their teachers. The purpose of the research was to consider how well the objectives of each participating initiative had been achieved, how the initiative contributed to improved outcomes for gifted and talented learners or their teachers, and how planning to continue to meet the learners’ needs after 2008 had been considered.

Author(s): Tracy Riley and Roger Moltzen, Massey University. Report prepared for the Ministry of Education.

Date Published: March 2010

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  To view the individual chapters please refer to the 'Sections' inset box.  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Section 6: Overall Conclusions, Limitations, and Recommendations

This report contains case studies of five Talent Development Initiatives, three of which were Ignite programmes and two were Enhance programmes. The Ignite programmes were designed for gifted and talented students and participatory action research was the approach to their evaluation. The Enhance programmes were professional development programmes for teachers of gifted and talented students and a case study approach was used for the evaluation. As has been reported, each programme was unique in its intended outcomes and indicators of effectiveness, which resulted in different methods of data collection, analysis, and reporting of results. Furthermore, the purposes in the Enhance and Ignite programmes were different. The Enhance TDIs were previously funded programmes which were entering a new, innovative phase in their development, whereas, the Ignite programmes were for new initiatives. Because of the different TDI purposes, the decision was made to use different approaches in the evaluation, creating, in a sense, two different research studies. This created some limitations to this study which are described in the next section.

Limitations

Although it was never the intention of the evaluation of the TDIs to cross-analyse or compare and contrast programmes, it was hoped that some common findings and understandings would arise, based on recurring themes. These have been reported separately in the syntheses of the Ignite and Enhance evaluations. However, given the different purposes in the Enhance and Ignite TDIs, coupled with differences between programmes for students and programmes for their teachers, it is a difficult exercise to bring these results together for analysis and synthesis of broad generalisations. This is a limitation of this evaluation; however, in providing the individual case studies for the five programmes, along with a synthesis of the two different types of TDIs, it is hoped that a rich, descriptive story has been told. There are also some important common findings and themes that arise. Further to this, some conclusions have been made in regards to the research purposes.

A further limitation of this evaluation was the size and make-up of the sample. Only three of the eleven Ignite TDIs were evaluated, yet two of the five Enhance TDIs were evaluated. It is also important to point out that four of the five programmes evaluated were provided by organisations external to schools: two private providers of one-day-a-week programmes and two university-based programmes. Care needs to be taken in considering the results of these programmes as they relate to school-based initiatives. To address these potential differences, the background to each programme was provided, as well as information related to how the TDI funding was used.

While attempts were made in the selection of the Ignite evaluations to ensure a range of different innovative programmes, encompassing a variety of different outcomes for students, having only three Ignite programmes limited the scope of diversity. Other pragmatic issues also hindered diversity: for example, the financial and human resources of the evaluation meant that consideration had to be given to researcher expertise and availability; geographical location; and potential conflicts of interest within the small community of gifted and talented education. These restrictions also meant that not all aspects of all programmes were evaluated. In the reporting of each case study, care was given in explaining each individual programme's evaluation methods and processes as these related to the overall research purposes, as a means of clarifying these limitations.

The Enhance evaluations comprised a disproportionate number of case studies in this research and both digressed from the original intentions of the funding pool as a resource for innovative programmes for gifted and talented students: two of the five Enhance programmes were evaluated and both of these were for professional development. Only four of the sixteen TDIs funded between 2005 and 2008 were solely for professional development, although many of the programmes for students included teacher outcomes. While the use of TDI funding to support gifted and talented students by way of enhancing teacher capability does not run counter to the aims of the funding pool, the outcomes and indicators for determining effectiveness vary widely from those of other TDIs (both Enhance and Ignite).

What this resulted in was two different approaches to the evaluation: participatory action research using a team approach and case study research conducted by an individual researcher. The number of onsite visits also differed, with the Enhance evaluations only being allocated two daily visits annually. This greatly limited the scope of the data collection and analysis, hindering the triangulation of data from multiple sources, opportunities for evaluative reflection and formative feedback, and a team of researchers able to check each others analyses and understandings. To address this, verification was sought from the programme coordinators of the Enhance TDIs, and when possible and appropriate multiple data collection methods were used. The researchers aimed to address these issues through flexibility, availability, and ongoing communication with these providers before and after visits, and to provide information and guidance when sought.

This evaluation was limited to participation of the key stakeholders in each TDI programme. It did not include participation by the TDI coordinator or representatives of the Ministry of Education; therefore, it only provides the voices and perspectives of those directly involved in the funded programme. There was ongoing communication and consultation with the Ministry, which provided some opportunity for sharing observations and concerns, but the roles of the evaluators versus those of the Ministry monitoring processes were sometimes blurred. Similarly, the level of support and input into the TDIs ongoing development sometimes created tensions for the researchers, who at times acted in a role that was also carried out by the Ministry. There was potential here for conflicting views, advice, and feedback between the researchers and those of the Ministry of Education. To address this issue, the researchers analysed milestone reports and Ministry feedback, attended the annual hui for TDIs, and had an ongoing relationship and communication with the TDI coordinator.

Finally, this evaluation did not seek to investigate the overall effectiveness of the TDI funding pool as a means of supporting gifted and talented students in New Zealand. It does provide insights into how five programmes were developed, implemented, and evaluated, but to extend these five cases to the model of funding would be inappropriate. What this evaluation does provide is the opportunity to highlight potential problems and solutions within the current funding pool model – but again, this is only based on five programmes and their stakeholders.

The limitations of this evaluation must be kept at the forefront. It is against these that any conclusions and recommendations must be considered and framed. Every effort was made by the researchers to address these issues, but some were beyond the scope and control of the researchers. Nevertheless, the conclusions and recommendations add to understandings of gifted and talented education in New Zealand.

Conclusions

This evaluation was guided by three overarching research purposes, each of which will be addressed in the conclusions:

  1. To determine how providers design, implement, maintain, and evaluate programmes for gifted and talented students, and their teachers.
  2. To determine how providers structure relevant and engaging learning and growth opportunities for gifted and talented students, or their teachers, as evidenced in the achievement of programme objectives; improved outcomes for students, or their teachers; impact upon key stakeholders; and planning for sustainability.
  3. To determine how, and what, formative feedback effects the development, implementation, maintenance, and evaluation of programmes for gifted and talented students by using an action research approach to evaluation.

Gifted and Talented Education Programmes for Students and Teachers

These five cases show that there is no set formula to gifted and talented education programmes for students or professional development to support their teachers. It does show, however, that there are guiding principles that are transferable and applicable across a continuum of approaches. Many of these principles have been outlined by the Ministry of Education in its policies and publications in gifted and talented education (e.g., the 2000 handbook, Gifted and Talented Students: Meeting Their Needs in New Zealand Schools; 2002 Initiatives for Gifted and Talented Learners).

There was variability in how these principles were transformed to practice. A wide range of practices in working with gifted and talented students and their teachers was being used. Practice is influenced by many factors (e.g., different stakeholder needs lead to different intended outcomes and means of delivery). Furthermore, New Zealand's history in formally acknowledging and supporting gifted and talented students, at a Ministry of Education level, is relatively short: the handbook for schools was published less than a decade ago; the initiatives were not released until 2002; and in the case of TDIs, they have only been operating for the past six years. The variability in practice is an important part of the developmental process as the field grows: it allows for experimentation, or trial and error, as New Zealand determines the most effective approaches to meeting student and teacher needs.

Variability also demonstrates that one-size-doesn't-fit-all in the development, implementation, maintenance, and evaluation of gifted and talented programmes. A needs-based approach to transforming the principles underpinning gifted and talented education into effective practice for students and their teachers was evidenced by the TDIs in this study. By determining the needs of each programme's key stakeholders, the TDIs were able to shape programmes better matched to meeting those. Each TDI programme was based on a recognised need and attempts were made to address those needs. This required flexibility, adaptability, creativity, and innovation, which all of these TDIs demonstrated to a lesser or greater extent.

Another common element shared by all of the TDIs evaluated in this study was the dynamic nature of their programme development, implementation, and evaluation. The study shows that conducting programme evaluations and being responsive to the findings can lead to programme improvement. The programmes' internal evaluation processes, and to varying degrees those of the external evaluation, were formative, leading to a dynamic model of programme development, implementation, and evaluation. Given the paucity of evaluation research conducted and disseminated in New Zealand, it is important that these programmes share their experiences – some already have and will no doubt continue.

Relevant, Engaging Learning and Growth Opportunities for Students and Teachers

The TDIs in this evaluation had varying levels of success in achieving their programme objectives, as the case studies have shown. Not surprisingly, there was variability in the achievement of outcomes for students and teachers. As the rich stories of each programme's journey in development explain, there are both enablers and barriers to meeting programme objectives, and, as a result, evidencing growth in outcomes. Some of the enablers included having a passionate and committed programme director with knowledge and skills in gifted and talented education; documenting and planning the programme in writing; ongoing evaluation and reflective practice; physical, human, and financial resources; using a team approach; open communication with stakeholders; and making connections between teacher and learner outcomes (e.g., programmes and professional development). In some cases, these same enablers actually acted as barriers: without these enablers, achievement of objectives was lost. Other barriers to be worked through included changes in personnel; changes between the proposal aims (provider-driven) and the negotiated contract aims (Ministry-influenced); and lack of expertise and support in determining appropriate outcomes and indicators during programme development. What this evaluation shows is that the pathway to the achievement of programme objectives will have twists and turns, hurdles, humps and bumps, and that ongoing support is needed to overcome the potential barriers and maximise the potential enablers.

Determining the impact of each programme upon its key stakeholders was at times difficult for both the providers and the researchers. Each programme was able to show some impact to some degree for some stakeholders. This was influenced by the programme's objectives, intended outcomes, and indicators of effectiveness. All of these programmes were also only part of a broader continuum of provisions for students and teachers; to view them in isolation would be inappropriate. In addition to this, some impacts are not easily or appropriately measurable, others may be short-term or long-term, and evidence of others may be unattainable, especially if the programme is one-step removed from its stakeholders, as is the case with out-of-school providers. It is important that when programmes are devised, providers are very clear at the outset of what impacts can be realistically expected, measured, and evidenced. This evidence can then, in turn, inform the programme's ongoing development.

All programmes showed evidence of planning for the sustainability of the programme, but not necessarily financial viability beyond the Ministry's funding. The programmes' documentation, including milestone reports, written policies and procedures, and websites; professional development and support for staff; and ongoing evaluation enable sustainability, and potentially transferability, with some degree of certainty. The financial sustainability of the programmes, however, is untenable in most cases. Four of the five cases evaluated were provided by private providers, and in all these cases, financial sustainability is at risk. These providers are not funded by the Ministry of Education, as is the case with schools, and so are reliant upon external contracts, donations, user-pay fees, and so on in order to operate. Their programmes were developed with little or no cost to stakeholders, in essence creating an expectation of 'free' services, reliant upon Ministry funding. This shows that the Ministry basically supported unsustainable initiatives from the outset. Some costs borne by schools, teachers, parents, and students may have offset this mismatch between the Ministry's expectation of sustainability and granting of full-funding for private provider programmes.

Participatory Action Research as a Model for Programme Evaluation

The Ignite TDIs shared a common approach to research – participatory action research. While this approach was implemented in different ways using different data collection methods and different teams of researchers to implement and analyse those, the aim was to evaluate each programme in response to its development. This enabled an evaluation of the Ignite TDIs that could be defined as 'learning by doing'. The researchers and programme coordinators were able to plan, do, study, and act, as advocated by VanTassel-Baska (2004c) in the evaluation of gifted and talented programmes. Working together through the cycles of research, enabled planning, implementation, evaluation, and the creation of a plan of action or improvement which then fed into the next cycle of planning, doing, studying, and acting. Thus, conducting a programme evaluation based on this approach to research enabled collaboration and participation, empowerment, and change, as described by Grundy (1982). The use of a participatory action research approach to the evaluation of the Ignite programmes was useful and instrumental in their ongoing development and evolution, and this has been shown in the rich case studies.

These evaluations also added to knowledge about the associated advantages and disadvantages of using a participatory action research model, guided by experts and practitioners working collaboratively, in the evaluation of gifted and talented programmes. While some of these advantages can be obtained through other evaluative research approaches, it was the collaboration between 'expert' researchers and 'expert' practitioners in reflective, evaluative work that made this particular project unique. A primary advantage of this approach was that it enabled fluidity and responsiveness in its implementation. It enabled the researchers to be responsive to change: nothing was set in concrete. As the three case studies show, just as the programmes evolved, so too, did each evaluation. This resulted in three very unique programme evaluations. However, some common advantages for the TDI programmes, their coordinators, and the researchers were shared. These include:

  • Programme coordinators had access to expert knowledge in a field relevant to the TDI, and this was achieved through careful matching of researchers to programmes which, in turn, enabled a collaborative approach to action research.
  • The TDI had access to materials, resources, and expertise offered by university-based research teams.
  • Each TDI coordinator was provided with support, encouragement, insights, and validation of their practices.
  • TDI coordinators had assistance and support in the ongoing dissemination of work being undertaken (e.g., conference presentations).
  • The research team offered an external perspective to programme development, implementation, and evaluation.
  • Being part of an action research team contributed positively to the professional development of TDI coordinators and the researchers.
  • Key stakeholders in gifted and talented programmes, including students, were provided with a 'voice'.
  • It provided validation and triangulation of results and emerging results.
  • For the gifted and talented students who were involved in the research, it provided opportunities for sharing their work and ideas in a broader context. It also provided an opportunity for their perspectives to influence the design and implementation of the programmes as they developed and evolved.
  • The researchers were provided with opportunities to be engaged in practice. This enabled practice-informed research, an often overlooked link between theory and practice whereby research usually drives practice.
  • It was rewarding for the researchers to have input into shaping and developing a programme for gifted and talented students.
  • Researchers became a natural part of the programme, increasing the authenticity of data collection.
  • Researchers were able to take on the role of critical friend.

While there were clearly some positive advantages to using an action-oriented approach to this evaluation, it was not unproblematic. For example, at the outset, it was important that trust be established between the researchers and the TDI teams, who were initially threatened, wary, and, possibly, even in fear of criticism. The need to develop collaborative, respectful and trusting relationships was driven by the research model, but heightened by two distinct elements of this research project:

  • Both the TDIs and the evaluation approach were new, innovative initiatives still under development; and
  • Both the TDIs and the evaluation were funded by the Ministry of Education, with each having different outcomes and reporting.

Therefore, both the research and the programmes being researched were in processes of evolution; yet, both had very clear and different expected outcomes. This created a tension for the researchers, in particular, between process and product: finding the balance between developing, shaping and influencing the TDI itself and answering the research questions. Another tension was one of bottom-up/top-down: the difficulty of respecting the TDI circumstances and needs while simultaneously operating within set parameters.

The roles of the researchers and participants were not always clearly defined, and ranged from novice to expert, researcher to practitioner, and insider to outsider. The researchers were regarded as academics with expertise in theory and research, while the TDI team members were expert practitioners. Adding to this complexity, the research teams were comprised of members with not only expertise in gifted and talented education, but specific related fields (e.g., science education, secondary schooling). Therefore, at the commencement of the project, the researchers could be defined by their academic expertise, but for this project the role shifted to that of a researcher expected to become involved in practice. Similarly, the expert practitioners were expected to become active researchers.

Initially, there was an underlying perception that the researchers were the 'experts.' Over time, as the roles shifted and the lines between researcher and practitioner blurred, the expertise became more balanced. As questions and problems were raised, it became a collaborative effort to create possible solutions. There was a transferral of ownership from the researchers to the practitioners, particularly in relation to each research cycle. In other words, the 'problems' identified by the research teams needed to be equally owned by the TDI teams who had the roles of identifying and implementing potential solutions. Also, as the TDI programmes and research evolved, the TDI directors developed confidence in their own expertise as a result of their programmes' successes, the feedback from both internal and external evaluations, and the dissemination and sharing of their work.

Role fluidity, whereby research team roles and TDI team roles became transferable and interchangeable, both across and within the collaborative teams, became more and more evident as the project progressed. It was important that each individual team members' strengths be recognised and utilised in balancing the collaborative roles. This was achieved in part through the careful matching of research teams to TDI programmes, but more importantly as the project developed there was a fluidity between the roles of problem-finders and problem-solvers, with these tasks often shared.

For the researchers, who had to take both an insider and outsider role, the tension played out as working in the capacity of collaborator versus external evaluator. For example, the development of relationships between the TDI coordinators and research team members meant that some objectivity could be lost. Also, given the innovative nature of the TDIs, alongside their grounding in theory and research which had been heavily influenced by some of the researchers themselves, it sometimes became difficult for the researchers to be critical of aspects of programme development. Another tension that arose was in relation to the extent of being hands-on/hands-off. What this meant was that the researchers often felt a conflict of roles: researcher versus expert, researcher versus monitor, and supporter versus advocate.

Finally, the researchers were, naturally, not part of the planning of outcomes and indicators when the programmes were conceptualised and eventually negotiated with the Ministry of Education. This meant that while all attempts were made to evaluate the outcomes for each programme, some of these may have been overly ambitious, many were multi-faceted and difficult to measure, and others were simply immeasurable as a causal relationship between programmes and outcomes for students. While this is not a reflection of the use of participatory action research to evaluate the programmes, it did present issues in determining appropriate methods of data collection and analysis in implementing such an approach.

Summary of Conclusions

The five programmes evaluated in this study demonstrated responsiveness and growth in their development, to varying degrees and in different ways. Each programme's journey over the three years of TDI funding was distinct, and this is reflected in their unique stories. The Ignite and Enhance TDIs described in this study add to our shared understandings of effective practice in gifted and talented education in New Zealand. They show promise in our country's aim of fostering all students' potential, making real the promise of gifted and talented students.

Recommendations

Future Practice

These recommendations for practice are applicable to programmes for gifted and talented students, as well as their teachers. Whether designing a programme for students, professional development for their teachers or other professionals, or a combination of the two, it is important that these recommendations be considered by programme developers. The recommendations arise from the five case studies in this report, but are also informed by wider theory and research in the field.

  • Programmes need to have clear and measurable outcomes, with shared understandings by all stakeholders. These outcomes need to be developed alongside programme evaluation plans.
  • Coordinators need to have background and experience in gifted and talented education, as well as ongoing professional development and support related to their practice. Financial resources should be made available for the employment of specialist staff in gifted and talented education, and their ongoing professional development and support.
  • An advisory group or committee, representative of stakeholders, should be developed to ensure viability, effectiveness, and sustainability. Programmes should not be reliant upon a single staff member, but rather supported by a group.
  • Ongoing engagement with stakeholders, including involvement in programme evaluation and responsiveness to findings, must be undertaken to ensure effectiveness and sustainability.
  • Programmes must have a strong philosophical grounding in contemporary theory and research, and developers must remain cognisant that shifting from philosophy to practice takes time and needs to be shaped by the relationship between implementation and evaluation.
  • Professional development and support is necessary in the implementation of comprehensive approaches to gifted and talented education. Professional development must be planned, responsive, needs-based, and flexible.
  • Curricular differentiation should take different forms, matched to student needs and supported by professional development.
  • Formative and summative evaluations of gifted and talented education initiatives should be both internally and externally conducted, using a variety of data collection methods and inclusive of all stakeholders. Programme evaluations should be designed at the outset of the programme.
  • Documentation of systematic, transparent models, based on sound principles and best practice, must be undertaken for sustainability of practice.
  • The impact of provisions should be measured on the understanding that this will vary in extent and nature based on the programme goals and implementation of those.

Future Research

  • Participatory action research as a model for evaluation needs to be further investigated and developed in gifted and talented education.
  • The roles of participants and researchers, and especially the fluidity and transferability of these roles as action research progresses, should be further considered.
  • Research investigating effective models of professional development in gifted and talented education needs to be undertaken.
  • Evaluation of the effectiveness of a wide range of other programmes for gifted and talented learners, including those from diverse backgrounds and cultures and employing different programme models should be conducted.
  • Models of funding for gifted and talented education programmes at a national level need to be investigated and explored.