Enhancing and Igniting Talent Development Initiatives: Research to determine effectiveness

Publication Details

This report presents the findings from research involving five Talent Development Initiatives for improving outcomes for gifted and talented students or their teachers. The purpose of the research was to consider how well the objectives of each participating initiative had been achieved, how the initiative contributed to improved outcomes for gifted and talented learners or their teachers, and how planning to continue to meet the learners’ needs after 2008 had been considered.

Author(s): Tracy Riley and Roger Moltzen, Massey University. Report prepared for the Ministry of Education.

Date Published: March 2010

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  To view the individual chapters please refer to the 'Sections' inset box.  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Section 4: Ignite Evaluations

This section of the report consists of the case studies for the New Zealand Marine Studies Centre, Rutherford College, and Te Manu Aute. Each of these Ignite TDIs was evaluated using participatory action research. The figure below illustrates the research cycles for the evaluation of the Ignite TDIs. These 'big questions' guided the work in each cycle and each one has a subset of more specific questions (adapted from the Ministry of Education Request for Proposals, 2005). The underlying research questions are presented below the figure. It should be noted that throughout each cycle of the research, and for each individual TDI, the answers to these questions are complex and interrelated. The ways in which each programme was evaluated, in an attempt to seek these answers, was different, as will be explained for each case in the following sections.

Figure 1. Action Research Cycles

Figure 1. Action Research Cycles

Research Questions

What is going on?

  • How were decisions around the programme design arrived at? Who was involved in the decision-making process? How has the process impacted upon the sustainability of the programme?
  • What changes in climate and philosophy have been required for the successful implementation of this programme? How have professional leaders approached the task of climate change, how were these changes managed, and how were changes in practice achieved?
  • How appropriate were the identification procedures, curriculum adaptations, and forms of assessment in relation to the goals of the programme?
  • What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme?

Is it working?

  • How appropriate were the identification procedures, curriculum adaptations, and forms of assessment in relation to the goals of the programme? How has this contributed to student outcomes?
  • What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme? To what extend has this specific programme design contributed to improved student outcomes?

Is it working? How do we know?

  • How comprehensive are provider initiated student and programme monitoring and evaluations? How do the findings of the monitoring or evaluation inform the programme?
  • What is the evidence for improved student learning and social, emotional, or cultural outcomes as a result of participation in the programme?
  • How have resources and personnel impacted on the success or otherwise of the programme?
  • What role has staff professional development played in achieving programme goals?
  • How well has the programme planning occurred in regard to sustainability?
  • What has the impact of the programme been on the whole organisation?

Research Methodology

This research was qualitative and interpretive in nature and drew on a variety of data collection methods. Within the broad paradigm of qualitative research, the approach taken in this instance is best described as participatory action research using multiple case studies, which was described in more detail in the previous section of this report. Essentially, this approach is about learning with others to affect change. It is not a step-by-step process but typically a spiral cycle of planning, implementation, observation, reflection, and replanning. Each case study explains the cycles of research. All participant groups in this study were fully informed of the ideas behind, and practices employed in, participatory action research.

The data collection methods used in each of these TDI evaluations can be organised under several broad methods: document analysis; observation; surveys; and interviews. As the evaluation progressed and different questions arose, the focus for all of these evaluations remained fixed on student outcomes. The perspectives of the programme's stakeholders, including the students themselves, the programme directors, teachers, and parents, were sought to determine the overall impact for students. For each case study, different data collection methods were used, with different stakeholders, and these are explained in detail in the following sections. It is also important to note that all participants in the research were provided with information sheets describing the study (as shown in the example in Appendix A) and asked to give informed consent before participation. Following Massey University's code of ethical conduct, students required parental consent to participate in the study.

The bulk of the data collected is qualitative, and the analysis of this data, derived from documents, interviews, observations, and surveys, began with the task of sorting and storing records based upon the sample, method, and year of collection into manageable chunks. From there, it was possible to begin an analysis of these according to their speaker and the context, guided by a framework of broad questions. This enables the report to include the voices of students, parents, teachers, mentors, and so on, to address the overarching research purposes:

  • To determine how providers design, implement, maintain, and evaluate programmes for gifted and talented students, or their teachers.
  • To determine how providers structure relevant and engaging learning and growth opportunities for gifted and talented students, or their teachers, as evidenced in the achievement of programme objectives; improved outcomes for students, or their teachers; impact upon key stakeholders; and planning for sustainability.

These questions guided a number of pre-ordinate themes, and these varied across cases. The content was analysed and coded based on these themes, in the first instance.

From these pre-ordinate themes, emerging sub-themes or conceptual categories were identified. The conceptual themes which derive from the data were then compared and combined in new ways to put create a 'big picture' that includes causal events, descriptive details, and ramifications. The pulling together of all the analysed data purposefully sought to provide a rich, detailed account that approximates the reality of the Ignite programmes. Although this analysis is presented in a linear fashion, in practice, it required analysis that was simultaneous and repetitive. For example, given the nature of this evaluation, informal analysis (as the data was being collected) guided subsequent data collection. This reflective thinking is an important element of action research.

The qualitative data is complemented, in some cases, by some quantitative survey and observational responses. Since these were limited in scope and sample sizes, descriptive statistics were used to determine the frequencies of responses. The quantitative responses were then considered in relation to qualitative data for explanation, verification, and data triangulation.

Thus, the analysis of data was aiming for coherence, consensus, and instrumental utility. The aim of the analysis was to present a rich, tightly woven account that made sense, reflected a reality with which others could concur, and was useful. The credibility of the findings is assured through the triangulation of methods and data used to unearth the multiple realities of those involved in the programmes. The generalisability of this research is limited in the sense that its findings cannot necessarily be applied to all programmes for all gifted and talented children in all contexts. However, an attempt has been made to provide sufficient information so that others can determine which findings may be applicable to other situations, thus allowing for some transferability. Finally, the analysis aimed to unveil not just what outcomes occurred as a result of the programme, but, equally important, why and how those were achieved.

Summary: Ignite Evaluations

The next section of this report describes the development, implementation, and evolution of the three Ignite programmes. For each evaluation, a description of the programme, including background and purposes, research cycles, data collection methods, and results, is provided. Given the unique nature of each case study, common themes explored under these major headings vary. A summary of each programme's evaluation results is framed against the research questions. A synthesis across results for the three Ignite programme evaluations shows some common themes related to how the programmes were designed, implemented, and evaluated.

New Zealand Marine Studies Centre

Background

The New Zealand Marine Studies Centre (NZMSC) is an educational facility located on the shores of the Otago Peninsula. The Centre operates in association with the adjacent research facility, the Portobello Marine Laboratory, as part of the University of Otago. The facilities of the NZMSC include lecture rooms, a teaching laboratory with running seawater and associated microscopes and scientific equipment, and an Aquarium. Combined with the Portobello Marine Laboratory, the research vessel Polaris , and the local marine environment, it creates a unique responsive learning environment for students of all ages and abilities. More importantly, the NZMSC provides access to expertise through the scientists, postgraduate students, and technical staff of the Department of Marine Science.

The NZMSC has long used these resources to provide learning experiences for mainstream primary and secondary school students through Learning Experiences Outside The Classroom (LEOTC) contracts with the Ministry of Education, catering for schools from throughout Otago, Southland, South Canterbury, and South Westland. An outreach programme in Nelson caters for schools in the Tasman, Nelson, and Marlborough regions. These programmes offer 'hands-on' science opportunities with live organisms in natural habitats through investigations and observations.

In 2005, the NZMSC was approached by a cluster of primary schools on the Otago Peninsula requesting a series of enrichment programmes for gifted and talented students, based on a withdrawal model similar to the style of one-day-a-week programmes. It is from that request that the NZMSC developed an application for funding as a Talent Development Initiative (TDI) in the 2006-2008 round. In addition to providing a programme for primary school students, the NZMSC also designed a programme for gifted and talented secondary students in local (Dunedin) and rural schools. The TDI was thus comprised of three linked programmes:

  1. The Primary Schools Cluster Programme targets Years 4-8 students from local Dunedin area schools as a day programme (5 days) for young potential scientists to work with a mentor on a research investigation.
  2. The Local Secondary Schools Programme targets students in Year 10 from local Dunedin area schools as a day programme (6 days) mirroring the approach of the primary programme. Teams of potential scientists are mentored through a marine inquiry.
  3. The Rural Secondary Schools Programme targets students in Year 10 from rural schools as a residential programme of three 3-day blocks. Like the other programmes, students work in teams to conduct scientific investigations led by a mentor; however, these students also stay overnight on Quarantine Island providing opportunities for socialisation with like-minded peers.

The funding provided by the Ministry of Education was primarily used for professional fees for an educational director, operational director, and mentors; operational costs (e.g., facility costs, equipment, transport); and teacher release days for contributing schools to allow teachers to attend the programme. There were no costs borne by contributing schools, their teachers, the students, or their parents.

For the purposes of this research, only the secondary school programmes (rural and local) were evaluated. This decision was based on several factors: the opportunity to compare and contrast the outcomes for rural and urban gifted students; the unique nature of a university-based residential programme; the focus on provisions for secondary-aged students; and the limited resources allocated to the research. The next section of the report provides details of the purposes and structure of the secondary schools programmes at the start of the three-year period of funding.

The Secondary Schools Programmes

The delivery of the two secondary programmes was originally based on a model whereby Year 10 students, selected by their schools, participated in a programme at the NZMSC at Portobello. Local school students attended six full days in two blocks of one week between each session (e.g., in 2006 dates were 23 May, 31 May, 6 June, 31 July, 8 August, and 17 August); whereas, the rural students attended for nine days in three blocks of 3 days and 2 overnight stays at Quarantine Island. Detailed planning for each session included a balance of whole group activities and team investigations. The whole group sessions focused on the conceptual theme (making sense of the marine world aids survival and enriches lives),and introduced students to the marine lab facilities, teaching scientific skills and sharing scientific knowledge and expertise. Research teams of four or fivestudents engaged in a scientific inquiry that was facilitated and mentored by a postgraduate student in marine science. As the investigations progressed, students shared their team's progress during these whole group sessions for constructive, yet critical, peer evaluation. At the end of the programme, each team presented their research findings in a variety of products (e.g., visual, oral, kinaesthetic) to an audience of their peers, parents, teachers, mentors, and members of the Department of Marine Studies.

Students in the rural programme had an additional feature: overnight stays on Quarantine Island (a boat ride away from the NZMSC). During these overnights, the students had responsibilities for cooking, cleaning, and general care of themselves and one another, but also participated in a range of recreational and social activities. Each night involved students purposefully placed in different teams (from their scientific investigation) and presented with exciting challenges – always marine-related.

Over the three year period, 25 schools (11 local and 14 rural) in the Otago and surrounding regions (Southland and Canterbury) had students participating in the secondary programme. This equates to 125 gifted and talented students, participating in approximately 30 team investigations with postgraduate student mentors. Their investigations ranged from studying meiofauna of intertidal sand ecosystems to exploring the feed relationships of southern arrow squid.

The intended outcomes of the NZMSC programmes for secondary schools were designed with both students and their teachers in mind. For the purposes of this evaluation, the primary focus was on student outcomes because of the lack of research related to the effectiveness of programmes for gifted and talented learners. Therefore, the evaluation aimed to uncover and explore evidence of the following outcomes for students:

  • Increased understanding of the relationship between perception (sense) and response (making sense) and the significance of this in terms of survival (for both marine organisms and humans).
  • Challenged to think more about their own perception, thinking, and problem-solving processes.
  • Critical evaluation of ideas and processes.
  • Use of application, analysis, and synthesis in the development of an inquiry through to a product, an action, and/or a presentation.
  • Increased understanding of self, true peers, and community through working in teams of like minds.

As the outcomes show, the major thrust of the programmes was to enhance students' abilities to know, think, act, and feel like a scientist working with a team as they developed: science knowledge and skills; metacognitive, thinking, and problem-solving skills; team work skills; communication skills; and self-awareness and understanding of their own abilities and talents. It was also hoped that by providing opportunities in science, students' course and career decision-making would be positively influenced.

The secondary school programmes also addressed outcomes for teachers from participating schools through the provision of professional development opportunities. The professional development involved an annual teacher-only day onsite at the Marine Studies Centre, featuring guest speakers (including student participants and mentors), and teacher release days to observe the programme in action. The teacher release days were designed to allow teachers to share in the experiences of their gifted and talented students and provide opportunities for teacher involvement in the programme. The intended outcomes for supporting schools and teachers included:

  • Increased complementary nature of the NZMSC withdrawal programme in relation to school-based programmes.
  • Increased understanding and strategies for working with gifted and talented students.
  • Enhanced solutions to the issues of isolation faced by rural gifted and talented students.

As stated previously, these outcomes were not the primary focus of this evaluation; however, elements of the programme, such as identification of students, cannot be examined without consideration of school and teacher involvement. Therefore, in the reporting of results, these outcomes will be appropriately addressed.

This is how the story began, but as this report will show, over the three years of the contract the model of delivery evolved in response to feedback from the many programme stakeholders and as a result of the action-oriented evaluation.

Research Methodology

This section describes the research team, cycles of research, and data collection methods used in this case study.

Research Team

Over the three year period of data collection, a team of researchers worked together onsite at the Marine Studies Centre using a participatory action research approach to evaluation. The research team was comprised of three University-based academic staff working in education: a specialist in gifted and talented education, and a science educator and a mathematics educator, both with knowledge and experience in gifted education. In the first year of the research, a social studies educator with an interest in gifted education was involved; however, he was unable to continue due to other professional commitments. Given the nature of action research, and in particular the participatory inquiry of this evaluation, the team included the programme directors. This interdisciplinary team of academics and practitioners enabled an ongoing examination well grounded in both theory and practice.

The onsite evaluation was initially comprised of two three-day visits annually and during the actual sessions with students. Additionally, one researcher was invited to present at the NZMSC professional development day for teachers, as well as the Ministry of Education sponsored Talent Development Initiative hui, on an annual basis, and these allowed further opportunities for members of the team to work together. The ongoing visits enabled the team to design, implement, and refine a range of data collection tools over the three years with which to evaluate the programme's intended outcomes.

Research Cycles

The evaluation of the NZMSC was an action research-oriented case study, thus, just as the programme developed and changed, so too did the research focus. From each attempt at answering a research question, a new set of challenges arose – in this sense, the evaluation was cyclical. The research began in 2006 with one simple question: What is going on here? The overall purpose in the first year of the evaluation was to familiarise the researchers with the programme 'in action' while at the same time piloting data collection tools. Two three-day visits were made to the NZMSC during 2006, allowing the researchers an opportunity to observe both the local and rural school groups, and to build relationships with the NZMSC staff. It was, in many ways, 'a fishing trip'!

It is not surprising, therefore, that in the second year, a new question presented itself: Is it working? By this stage in the research, the programme was well underway, the pilot tools had proven useful, and the researchers had a strong grasp of the programme. It was decided that rather than making one case study visit to each group, there was a need for an intense focus on only the rural schools programme, and for continuity, a researcher needed to attend each of the three 3-day sessions. In 2007, all students, all sessions, and all data collection tools were implemented with the rural schools programme. This allowed for continuity in the data collection, as well as a closer examination of the delivery of the programme as it was evolving.

The decision was made to follow a similar approach in 2008, but with the local schools group. The question – is it working? – still remained, but the next question arose: how do we know? The programme was entering its final year of funding and, yet, the evaluation methods being used by the NZMSC were not proving useful in providing evidence that all learning outcomes were being met. There was a need to reconsider the development, use, and analysis of the programme's internal evaluation methods. The findings in relation to the research questions focusing on provider initiated student and programme monitoring, improved student outcomes, and impact of the programme for stakeholders showed that while there was some evidence of the programme 'working' the internal evaluation methods were not as effective as intended. Therefore, the focus in 2008 involved all local secondary students, in all sessions, and using all data collection tools by the researchers. In addition, the evaluation of the programme became a major focus of the research team's work.

In order to answer the research questions, a range of qualitative and quantitative data collection methods was employed, as the next section describes.

Data Collection Methods

The data collection methods used in this case study are shown in the following table. A detailed explanation of each method and the tools utilised follows in the next sections.

Table 2: Data Collection Methods
Document Analysis
  • Programme Proposal and Contract
  • Milestone Reports
  • Programme Communications
  • Student Applications
Observation
  • Classroom Observation Form and field
  • Student Product
  • Assessment Form
Survey
  • PMI - Plus, Minus, Interesting: students
  • Mentor Reflection Form
  • Metacognitive Awareness Questionnaire: students
  • Online Questionnaire: students, teachers, parents
Interview
  • Focus Groups: students, mentors, teachers, parents
  • Individual: programme directors
Document Analysis

The original programme proposal, Ministry of Education contract, and milestone reports provide descriptive information about the NZMSC secondary school programmes. The programme communications with stakeholders (e.g., information packs for schools, www sites, agendas for professional development) provide further descriptive evidence. The milestone reports, in particular, verify the evolving nature of the programme over the three-year period. These documents also provide insight into how the programme was developed, implemented, and evaluated.

The other documents analysed in this evaluation, providing rich detail of the individual students who participated in the programme, were the student applications. Students participating in the programme complete a simulated 'job application' to be a research assistant. The application queries their relevant experiences and interests; their understanding of the conceptual theme, the relationship between senses, responses, and survival; a practical scientific investigation they have undertaken; and personal strengths and qualities. Since its inception, the application has been fine-tuned and re-developed to include surveys of learning preferences regarding group/team composition and differentiated processes; roles in a team (e.g., delegating tasks, communicating); leadership styles; and perceptions of self as gifted and talented (e.g., enjoyment of learning with like-minded peers). A Metacognitive Awareness Survey (adapted from Schraw & Dennison, 1994) is included in the student application. Students are also asked to provide a covering letter and curriculum vitae.

The information on these applications provides baseline data for measuring the impact of the programme upon their own perceptions, thinking, and problem-solving processes, as well as their understanding of self as gifted and talented. They also provide insight into the implementation of the programme, as the student information provided is used for grouping students and shared with their mentors for better understanding students. Applications for the majority of student participants were made available to the researchers by the NZMSC (issues related to incomplete or missing applications are raised later in the report).

Observation

Observation of the programme in action is a seminal part of this programme evaluation, as it provided an authentic opportunity for determining the quality of the learning experiences for gifted and talented students as implemented by the NZMSC. To guide the observations of the programme in action, the Classroom Observation Form (COF), developed by Van Tassel-Baska and colleagues at the College of William and Mary (Van Tassel-Baska & Feng, 2004), was employed. This form enables observers to evaluate the programme by sampling a range of teacher behaviours, as observed or not observed, and in relation to:

  1. Curriculum Planning
  2. Expectations for Learners
  3. Accommodation to Individual Differences
  4. Curriculum Delivery Features
  5. General Teaching Strategies
  6. Critical Thinking Strategies
  7. Problem-solving Strategies
  8. Metacognition
  9. Classroom Extensions

The form was developed based on best practices in gifted and talented education and for the purposes of programme evaluation (as opposed to teacher appraisal). The developers encourage an aggregation of the data across teachers or classrooms, to provide "a snap-shot of instructional practice" (Van Tassel-Baska & Feng, 2004). In its implementation at the NZMSC, the COF was utilised for observations of each team of students, as they worked with their mentor in their investigations. Field notes of observations were added to the form, so that observers could note details of events observed. Over the three year period, 30 observations were made by the research team with each observation being at least 30 minutes in duration. Analysis of data across observations was used to determine the effectiveness of the programme in meeting its goals, but also as a formative tool for enhancing the programme (and in particular professional development and support for mentors). The data contribute to understanding the development and implementation of differentiated educational opportunities, evaluating effectiveness, and professional development and support, and to a lesser degree, outcomes for students and the impact of the programme.

A second tool for observation was the Student Product Assessment Form (SPAF) developed by Reis and Renzulli (1991) for the evaluation of student products. The form enables an assessment of a range of student products and the guided observations of the final team presentations in the NZMSC programme. The SPAF focuses the assessment on:

  • Early Statement of Purpose
  • Problem Focusing
  • Level, Diversity and Appropriateness of Resources
  • Logic, Sequence, and Transition
  • Action Orientation
  • Audience
  • Overall Assessment (e.g., originality, quality)

These elements are rated using a 5-point likert scale ranging from 'a great extent' to 'a limited extent'. Account is also taken of items which are not applicable to the project. The SPAF was utilised in all three years of the programme to assess the projects of the secondary programmes (2006 and 2007, rural; 2008, urban). Analysis of data across observations of student products was used to determine the effectiveness of the programme, but also as a formative tool for enhancing the programme. The data contribute to the determination of outcomes for students and the impact of the programme, and, to a lesser degree, understanding the development, implementation, and evaluation of differentiated educational opportunities.

Survey

Several forms of surveys were implemented in the evaluation. The first of these was developed by the research team to gather data from students in the programme and used in conjunction with focus group interviews. The PMI - Plus, Minus, Interesting (see Appendix ) probed what students felt was positive, negative, and interesting about their experiences in the programme and in relation to the content, processes, products, learning environment, and the programme staff (mentors and directors). During 2006, these forms were trialled on one team of students from each programme (four rural students and four urban students). In 2007 and 2008, it was administered twice to all students participating, in the rural and urban groups, respectively. These surveys were administered prior to focus group interviews with each team, as a way of guiding the discussions, and as a tool for verifying interview data. The data gathered aid in determining the outcomes for students and impact of the programme, particularly in regards to being part of a differentiated educational experience.

These experiences are facilitated by mentors, University postgraduate students in marine science, and their role is an important aspect of the programme's implementation, as well as its effectiveness. To gauge their perceptions, a Mentor Reflection Form (MRF) was developed. This form is an adaptation of the COF, but rather than being used as a tool for an outside observer to assess teaching, it was adapted as a means of self-reflection. Mentors were given the MRF prior to focus group interviews, to guide their thinking, and as a measure of triangulation with the observation and interview data. Over the three year period, 15 Mentor Reflection Forms were collected and analysed. The MRF also included a section for mentors to report their perceptions on how well the overall programme was meeting its outcomes for students, providing further evidence of its effectiveness. As a formative tool, and coupled with the focus group discussions, the MRF gave insight into areas of professional development and support for the mentors. The data are critical in understanding the role of personnel and professional development and support in the implementation of differentiated programmes. The results of the MRF also contribute to determining the impact of the programme in relation to outcomes for students.

An outcome of the programme was the enhancement of students' thinking and problem-solving, in particular, their metacognitive awareness and development. As the programme developed, an issue that arose was a lack of data to support or refute changes in metacognitive awareness. Therefore, in 2008 (the final year of the study), the Metacognitive Awareness Inventory was included in the students' application as a pre-assessment, and in the NZMSC evaluation as a post-assessment (completed and analysed for the 22 local students in the 2008 programme for the purposes of this evaluation). The questionnaire probed students' awareness of metacognitive strategies such as setting and monitoring goals, the adoption of situation-specific learning strategies, and task analysis, using a yes/no format. Analysis of the pre/post-responses provides an indication of the effectiveness and impact of the programme upon student outcomes. Its implementation in the final year of the programme also adds to the rich detail of how evaluation is developed and implemented.

Another tool for evaluating the overall effectiveness and impact of the secondary school programmes was an online questionnaire for all students and their parents and teachers, administered in 2008. The questionnaires were developed by the research team using an online survey tool called Survey Monkey (www.surveymonkey.com). The questionnaires were developed in conjunction with one another to simultaneously collect multiple perspectives from key stakeholders in the programme. The information sought in each of the three surveys is shown in Table 3 below. As the table shows, common questions were asked regarding participation, school-based learning, and impact of the programme. Parents and teachers were queried about funding (as a means of addressing sustainability) and future directions. Teachers were also probed regarding the school-based facilitation of student involvement (to clarify alignment between the withdrawal programme and school-based initiatives).

Table 3: 2008 Final Survey Themes
Student Survey
  • Participation: year, mentor, sources of support (e.g., parental, school)
  • School-based learning: science-related study, gifted and talented programmes, future plans
  • Impact: science knowledge and skills, process skills, personal development, overall impact
Parent Survey
  • Participation: year, other NZMSC programmes, parental involvement, school support
  • School-based learning: gifted and talented programmes, future plans
  • Impact: science knowledge and skills, process skills, personal development, overall impact
  • Future directions: funding, parent support
Teacher Survey
  • Participation: year(s), involvement (e.g., identification, professional development)
  • Facilitation: identification, ongoing support for students
  • School-based learning: gifted and talented programmes, alignment with school-based programmes
  • Impact: science knowledge and skills, process skills, personal development, overall impact
  • Future directions: funding, further programme development


All former students, their parents and their school's head of science and gifted and talented education coordinator were invited to participate in the study via written information distributed to schools in October 2008, followed by a reminder email to gifted and talented education coordinators. The reliance upon schools to distribute the information to all 125 former participants, as well as their parents and teachers, was problematic, but unavoidable, as the NZMSC did not have complete home addresses for students. Additionally, over the three years, some teachers and students had relocated. In total, 32 students representing 15 schools (seven local and eight rural) completed the online survey. Of these students, half were participants in the 2008 programme and a quarter each were from 2006 and 2007. A similar low response was received from parents (16 in total) and teachers (13 in total).

Despite these low response rates, the data from the surveys help clarify the overall impact of the programme, verifying other data sources, from the perspectives of students and their teachers and parents. It also delves into issues of funding, and, hence, sustainability of the programme beyond Ministry funding.

Interviews

Individual and focus group interviews were also conducted with a range of different stakeholders over the three year period. The primary sources of interview material were the programme directors, student participants, and mentors; however, when opportunities arose to interview visiting parents and teachers, they were also invited to participate. Individual interviews were conducted with each of the two programme directors; focus group interviews were conducted with all other groups. For the planned interviews, schedules were developed and consistently employed throughout the evaluation. Informal interviews probed parental and teacher perceptions of the programme. All interviews were recorded and transcribed, as well as supported by the researchers' written notes, and, in some cases, survey data from participants.

Each year of the programme, the two directors were interviewed individually, to gain their unique perspectives of the programme to date. For example, the education director was asked questions about the preparation, planning, and professional development and support of mentors; the identification of students by their schools; the overall planning and delivery framework; and his communications with schools. The other director, who managed the operational side of the Ministry of Education contract, was queried about budgetary issues, personnel, and sustainability. Both also provided data regarding their perceptions of the programme, Ministry of Education feedback and support, and the evaluation itself. This data was useful in understanding programme development, implementation, and evaluation, as well as the issue of programme sustainability. In the final 2008 visit, the two directors were interviewed together, mainly to determine future directions of the programme based on their three-year experience and its evolution.

Each year of the programme, all mentors (for both rural and local groups) were invited to participate in a focus group interview. These interviews probed the mentors' responses and reflections as recorded on the Mentor Reflection Form; thus, the themes explored planning, expectations and accommodation of learners, and teaching strategies. Mentors also provided their perspectives of the impact of the programme upon students. The data clarify issues regarding the delivery of differentiated programmes, professional development and support, outcomes for students, and the impact for all stakeholders (including the mentors themselves).

In 2006, the focus group interview for students was trialled with one local group of students at the end of the programme. In 2007 and 2008, all teams of students from the rural and urban groups (respectively) participated: this comprised nine focus group interviews. The focus group interviews provided students an opportunity to share their perceptions of the 'plus, minus, and interesting' aspects of the programme, with the interviews being guided by the PMI surveys. This opportunity to elaborate verbally provided rich evidence of students' views of the programme and its impact upon them as gifted and talented learners. The student focus groups also provide data that describes the elements of differentiated programmes most valued by gifted and talented students themselves.

In 2006 and 2008, two different groups of visiting teachers were interviewed. This was very much a spontaneous, opportunity-taking exercise on the part of the researchers. In 2007, one interview was conducted with two parents of rural students who were chaperoning their children. These interviews were open-ended, basically seeking parental and teacher perceptions of the programme. It is also worth noting that throughout the three years, many conversations took place amongst the researchers and all stakeholders, as the programme was being delivered.

Employing a wide range of both qualitative and quantitative data collection methods, inclusive of the perspectives of the many key stakeholders in the programme, provided rich insights – and a tremendous amount of 'raw data'. The next challenge was to make sense of it all through careful analysis and recording of this TDI's story. The table below summarises the pre-ordinate themes for this case study.

Table 4: Guiding Questions and Themes
Framework: Guiding Questions
How are programmes developed and implemented for gifted and talented students?
Themes
  • Programme design
  • Definition of giftedness and talent
  • Identification methods
  • Differentiated programmes
  • Evaluation
  • Resources
  • Professional support and development
What are the outcomes for students in relation to the programme goals?
Themes
  • Science knowledge and skills
  • Metacognitive, thinking, and problem solving skills
  • Team work skills (including communication and presentation skills)
  • Self-awareness and understanding
What is the impact of the programme for all stakeholders?
Themes
  • Evidence of growth, development, and change Science knowledge and skills
Is the programme sustainable?
Themes
  • Resourcing
  • Future plans and directions


The results are presented in relation to programme development and implementation; outcome for students; the impact of the programme for key stakeholders; and its sustainability.

Programme Development and Implementation

This section describes the many interrelating aspects of programme development and implementation as they evolved in the three year period of the NZMSC programme for secondary schools. It begins by describing the design of the programme, followed by detailed elements underlying its implementation.

Designing the Programme: Key Players

As was stated in the background information, the secondary school programme arose as part of a response to a request for specialised opportunities for gifted and talented students. As the initial programme proposal was being developed, the coordinators realised that their resources, expertise, and systems were well suited for meeting the unique needs of gifted and talented students (NZMSC Proposal, 2005). The NZMSC was able to provide students an authentic learning environment, a working marine laboratory, with specialised equipment, access to highly skilled and knowledgeable marine scientists, and educational expertise in developing programmes for learning outside the classroom.

The programme was coordinated by an educational director (.5 FTE) and a programme director (.2 FTE). The design of the programme began with a clear set of roles and responsibilities for the two programme co-directors, with one overseeing teaching and the other administration. This partnership was important in designing the structure of the programme, ensuring it was educationally sound, as well as feasible and accountable. An advisory board was established in the first year of the programme, with representatives from the Department of Marine Studies, a board member of the Otago Gifted and Talented Association, an adviser to schools, and teachers from local schools (Milestone Report 1, 2006). The advisory board provides guidance, advice, and suggestions, as well as monitoring progress. A team approach to coordination enabled checks and balances in the design, delivery, and evaluation of the secondary programmes. However, there was one key player in this management team: the educational director.

The educational director's passion and interest in gifted and talented students was a driving force behind the development of the programme. As the programme director stated:

We wouldn't be doing this is we didn't have [name]. He is the one with the vision and knowledge to actually make it happen. (Programme Director, interview, 2006)

The educational director had experience as a secondary school teacher delivering and developing a residential camp for gifted students in rural secondary schools and was very much of the belief that these students needed opportunities to work with like-minded peers. The design of the programme exploited his experience, professional skills, and interests in a facilitative role, what he described as

… the conductor as it were of the programme, I set the tone and the framework of the programme. (Educational Director, interview, 2006)

The educational director basically designed all aspects of the programme and facilitated its implementation. As he explained in interviews, he was the first point of contact. His responsibilities included all planning, communication with stakeholders, and overseeing implementation and evaluation of the programme. He worked closely alongside another group of key players in the programme – the mentors, postgraduate students in marine science who facilitated the research teams as they designed and undertook their scientific investigations. As he explained,

I group the students, I go to the mentors and ask them what they are planning, how they think they are doing, and then I set up meetings with the head of the lab about what has to be done in regards to negotiating boats and equipment and all sorts of gear and make the booking for the plans. (Educational Director, interview, 2006)

The programme director, advisory group, and mentors were dependent upon the educational director, and in the first year of the TDI this became somewhat problematic. Two issues arose, the first being that at the start of the contract, he was engaged in two jobs. The Ministry funding did not kick in early enough in the planning stages to allow him release from his full-time employment as a science educator at NZMSC. It is important to understand that in order to make use of the NZMSC facilities and staff, although the TDI did not 'officially' begin until 2006, the planning and scheduling of the programme had to be undertaken six months to a year in advance of its actual delivery. The second issue grew out of this attempt to do two jobs simultaneously: much of the planning and design was "in his head" (Programme Director, interview, 2006). As he explained, upon the eventual employment of another science educator, "I expect to find that the workload becomes more manageable and I can focus a little bit more on the programme" (Educational Director, interview, 2006).

The programme director was aware of these issues and the impact upon programme design. While the educational director was reliant upon her for managing the contract, as a scientist, she was heavily reliant upon him to design and deliver the educational programme. Her role was one of managing the budget, the milestone reports, the advisory group – or as she described, "the administration side of it" (interview, 2006). She was also able to facilitate and provide administrative support for the educational director, though he was at times reluctant to accept it:

I mean he's very much do it himself, it's easier to do it himself and he likes to do it himself … I don't think he realises that he hasn't let go. (Programme Director, interview, 2007).

An important design issue that arose from this evaluation is in regards to resources. The forward planning needed for the design of the secondary schools programmes was not well resourced and heavily reliant upon one staff member. Thus, in the first year, it was difficult for the research team to ascertain some elements of the design. However, over time, as systems of support and resources were put into place, and the programme was implemented, these issues were resolved. At the end of the second year, the programme director explained, "… we were flying by the seat of our pants last year really and I think there has been lots more planning … this year" (interview, 2006). The next section explains the structure and delivery of the programme design, as it evolved over three years of careful planning, implementation, and evaluation.

Designing the Programme: Structure and Delivery

The design of the programme was explained in the initial proposal for TDI funding (2005). The proposal included many aspects of a policy for gifted and talented students:

  1. A rationale which described the underlying theoretical principles and research-driven basis of the programme design: integrated curriculum based upon a conceptual theme facilitated through small group investigations guided by a mentor in a university-based withdrawal programme (Ministry of Education, 2000; Riley et al., 2004).
  2. Inclusive definition and identification : a multi-categorical definition of giftedness and talent, inclusive of both potential and performance, with ability determined by school-based multiple methods of identification, as advocated by the Ministry of Education (2000).
  3. A structural framework based on curricular models in gifted education: Renzulli's Enrichment Triad (1977) and Betts' Autonomous Learner Model (1985) supported the content, process, and product differentiation, as well as the development of self-understandings of giftedness.
  4. Detailed plans for differentiated educational opportunities: who, what, why, when, where, and how of delivery.
  5. Resources : human, physical, and financial.
  6. Evaluation methods for determining effectiveness of aims, outcomes, and objectives of the programme: multiple qualitative and quantitative, formative and summative approaches inclusive of all stakeholders.

These elements of design are further described in later sections of the report. The structure and delivery of the programme was also addressed in the original proposal (NZMSC Proposal, 2005). When the TDI began in 2006, the rural and local programmes differed in delivery, but not structure, as Table 5 shows. The major difference was that the rural students engaged in a nine-day programme, including overnight stays; whereas, the local students were involved in a six-day programme. The only other difference was a half-day full group session for the rural students during their second block that focused on team investigations, roles, and presentations.

Table 5: NZMSC Secondary Programme Structure (2006)
Programme Structure Rural Delivery Local Delivery
  • Introduction to programme
  • Hands-on activities to introduce facilities
  • Selecting and 'bidding' on a topic of study 
  • Meeting with mentor to explore research questions
Term 2:
Three Day Block,
Two Overnights
One Full Day
(Term 2)
  • Seminar/guest speaker on marine science topic
  • Teams planning and initiating investigations (goal setting, designing, and recording experiments) 
  • Report back by each team on progress, feedback from peers and mentors

One full day
(One Week Later)
  • Team investigations continue 
  • Report back by each team on progress, feedback from peers and mentors
Term 3:
Three Day Block,
Two Overnights
One full day
(One Week Later)
  • Teams are given challenge to present a final presentation, including visual and sound challenges, as part of an oral, audio-visual, and poster/display presentation 
  • Teams plan challenges using brainstorming

One Full Day
(Term 3)
  • Teams work on presentation challenge 
  • Report back by each team on progress, feedback from peers and mentors
Term 4:
Three Day Block,
two overnights
Term 3:
One Full Day
(One Week Later)
  • Completion and presentation of presentations to an audience of peers, parents, teachers, and marine scientists

Term 3:
One Full Day
(One Week Later)


The rationale for the rural students having overnight stays was both pragmatic, given travel costs, time, and logistics, and specially designed to address their different needs. As the proposal (2005) explained,

… this programme is designed to address the particular needs of the gifted and talented in small rural schools. Though they may have the support of knowledgeable teachers and negotiated individual challenges, they often miss the stimulus and reassurance of working in teams of like-minds and the challenging enrichment and extension a university-based learning centre can provide.

Thus, their programme was more extensive, providing different opportunities for engagement at the NZMSC, as well as the inclusion of social activities and challenges. The overnight stays on 'the island' were as integral a part of the programme as the time spent in the marine laboratory.

The overnight stays were also well-planned and organised in advance of the programme implementation. Students were placed in island teams, made up of different combinations from their research teams or school groups. This intentional mixing and matching of students provided opportunities for rural students to engage with many like-minded peers in different learning and social experiences. Teams not only worked together sharing responsibilities for cleaning and cooking duties, but also in a wide range of recreational and social activities on the island. These ranged from quiz challenges to Sea Animal Olympics, and all were created and facilitated by the educational director.

After the first year of the programme, the delivery for local students changed, to include an overnight stay. This was based upon several factors. Student feedback indicated that local students wanted an overnight stay, and rural students were positive about the social and recreational aspects of the programme. Observations by the research team, mentors, and programme directors, showed a greater sense of cohesion amongst the rural group than the urban group – both socially and in relation to their scientific research teams. Students' commitment to the programme also varied, with some local students not attending some sessions. Thus, in 2007, local students were provided with an overnight stay of one night during the term 2 block. In 2008, the local students had an overnight stay of one night in term 2 and two nights in term 3. These changes also meant the local students were engaged in an 8-day programme.

As the programme progressed through its three-year funding cycle, the delivery was adapted in responsiveness to student needs and feedback. The structure of the programme remained intact as originally proposed. The implementation was driven by the interrelated elements of definition, identification, a differentiated programme, and evaluation (Ministry of Education, 2000). The following sections explain these elements.

Defining Gifted and Talented Students

The NZMSC did not provide a definition of giftedness and talent for its programme, recognising there is not a universally accepted definition (NZMSC Proposal, 2005). It did, however, outline several principles that underlie contemporary definitions of giftedness: exceptionality; performance and potential; possession of one or more of a wide range of abilities; and recognition of dual exceptionality. These principles are in keeping with Ministry of Education (2000) guidelines for defining giftedness. Also, the lack of a definition for this programme is appropriate, as it adheres to the Ministry of Education's recommendation (2000) that each school in New Zealand develop a school-based definition of giftedness and talent. For the NZMSC to develop a definition and impose it upon contributing schools would be inapt.

It is also important to note, that the programme was not based on a definition limited to the scientifically gifted, but took a much broader approach. Schools were asked to nominate gifted and talented students who showed a "strong interest in science" (Information Packet for Schools, 2006-2008), a characteristic of students gifted and talented in science. Over the three year period, other characteristics emerged that typified the students in the programme (as detailed in the next paragraph). These are derived from analyses of the student applications and school-based teacher nomination forms.

Students were asked to describe their personal strengths and qualities that would enable them to work effectively in a scientific research team. Their perceptions very clearly showed strong motivation and commitment, good people skills, a range of communication skills, and a willingness to work with and contribute to a team. This was confirmed by the teacher nomination forms, which also highlighted a strong thirst for learning. As one student commented, "I have an excellent attitude and a desire to learn more" (Student Application, 2007). The teacher nomination forms also highlighted "well-balanced" students who in many cases were very good "all-rounders".

This notion of having multiple and wide-ranging strengths and interests was confirmed in the student applications. Academic strengths were multiple, with science, mathematics, and English being the most commonly reported areas of recognised abilities. Many of the students were top performers in local and national competitions, including the Australasian exams and science fairs. Overwhelmingly, these gifted and talented students all participated in an array of sporting activities – from Greco-Roman wrestling to trampoline – with some performing at top national levels. Many of the students reported an interest in water sports, fishing, and hunting. Other major interests included playing instruments, singing, or just listening to music; reading a range of fiction and non-fiction books and magazines; and to a lesser degree, computers and technology. Arts and crafts, dance, drama, and writing were also cited by some students, with one reporting "I am working on a novel at the moment" (Student Application, 2007). The following student responses show their diverse interests and hobbies:

Snorkeling, reading, violin, Greek mythology, and thinking! (Student Application, 2007)

Rugby, cricket, basketball, Algebra, and Māori. (Student Application, 2007)

If you asked anyone else what my interests and hobbies were they'd just say "reading". I do read a lot but I also do a whole lot more other things. I play the oboe … I sing in two different choirs … I am in the school orchestra, the creative writing group, and I am a sea-scout…I play netball and hockey. I am starting music theory lessons … I am doing The Duke of Edinburgh Award (Bronze). There is so much more to say about me but this room is used up. (Student Application, 2008)

In 2008, the student application was amended to include detailed information about students' experiences "as a gifted learner working in the area of science". Students were asked about their learning preferences regarding grouping and differentiation. The table below summarises the responses for the 2008 local school students. As Table 6 shows, the students demonstrated an interest in working in a variety of grouping arrangements, but strong preferences for flexible groups, depending on the task, and with others of similar ability. They clearly did not show much preference for set tasks, preferring intellectual challenge, choice, access to resources and expertise, depth and detail, using different modes of communication, and hands-on investigations. Their responses also show a preference for working with others in a scientific investigation, with the majority of students indicating small groups of three or four peers (in response to another question).

Table 6: 2008 Local School Students' Scientific Learning Preferences (Pre-Programme Results)
Students' Scientific Learning Preferences Strongly Disagree
/Disagree
Agree/
Strongly Agree
In a team or group 17% 83%
With a partner/in pairs 8% 92%
Independently 32% 68%
In flexible groups depending on the task
100%
With others of similar ability levels
100%
With others of different ability levels 18% 82%
Sharing and discussion 10% 90%
Questioning 8% 92%
Reading 22% 78%
Criticism and critique 17% 83%
Having access to expertise 4% 96%
Having access to resources 4% 96%
Set tasks 43% 57%
Intellectual challenge
100%
Choice
100%
Pace 17% 83%
Depth and detail 4% 96%
Over long, dedicated periods of time 39% 61%
Using different modes of communication and presentation 4% 96%
Hands-on investigations 4% 96%


In summary, the lack of a definition of giftedness and talent specific to this programme is appropriate. The students participating had some shared characteristics, including a set of interpersonal skills that enabled them to enjoy working as a member of a team on a scientific inquiry. Over the three years of the programme, only two students requested withdrawal from the programme: a male student whose teachers explained that he did not want to be recognised as able, and a female student whose parents withdrew her as a sort of punishment. Did the teachers nominating students just 'get it right' or was something else happening? The next section begins to answer this question by describing the identification procedures.

Identifying Gifted and Talented Students

As has been previously explained, local and rural secondary schools were invited to nominate Year 10 students based upon their school-based identification methods, but with consideration of the principles of identification outlined by the Ministry of Education (2000). The NZMSC expected all participating schools to nominate up to two students who were identified as gifted and talented based upon no less than three different methods of identification and who showed a strong interest in science. The students needed to be available for full participation in the programme, including overnight stays. All students also completed an application form to work as a marine assistant, submitted with a covering letter and curriculum vitae. The anticipated number of participating students was 20-25 in each programme, and this remained the case throughout the three years. The number of nominated students from each school, however, varied from year to year, as have the participating schools.

The NZMSC hosted professional development annually, during which identification tools and methods were discussed and shared by a school adviser, or other expert in the field, alongside the education director who explained the programme expectations. Also, teachers were expected to observe the programme in action, giving them insight into the nature of the programme and which students might benefit from it. In student interviews (2007 and 2008), some students indicated involvement in the identification process, as a sort of student recruitment whereby past participants shared information about the programme with prospective students. A multi-method approach was expected and supported by the NZMSC.

An analysis of student applications shows that most schools adhered to the request of using at least three methods of identification, but some used less and others more. The most common form of identification was teacher observation and nomination, used by all schools, and sometimes supported by rating scales. The NZMSC recommended the Purdue Scales (Science) and these were being utilised. The end of programme survey of teachers supports this reliance on teacher nomination, with 92% reporting involvement in the identification process.

Schools also used self-nomination in most instances, relying upon interest inventories and an adaptation of the McAlpine and Reid (1996) Teacher Observation Scales outlining common characteristics associated with giftedness. The end of programme survey shows that 81% of the students were given the opportunity to put themselves forward through application for the programme, with 45% receiving school-based support in the application process. Despite this student involvement, focus group interviews revealed that students were often unclear as to why or how they were selected for participation in the programme. Their perceptions ranged from "cos my science teacher wanted to get rid of me" (Student, May 2006, focus group) to "It could be something to do with exams. I think a lot of us got really high marks in science exams" (Student, May 2006, focus group) to many who just didn't know.

Standardised testing was another common practice, reported by the majority of schools. The asTTle tests and Progressive Achievement Tests (PATs) were most often reported, though some schools utilised school-based entrance test scores. Student portfolios, performances, and auditions were sometimes used, with peer and parent nominations seldom reported. Parents completing the end of programme survey verified their lack of input in the process, with only 4 of the 12 parents stating involvement.

At least one school screened applicants and developed a process for selection solely for the purposes of the NZMSC programme. Students were invited to apply through the gifted and talented coordinator, who then screened the applications and interviewed the students. Selection was then made based upon a range of identification methods, and verified through the students' programme application, covering letter, curriculum vitae, and an interview.

In 2006, schools did not, however, always submit full information regarding selected students – and this was needed for student placement in groups and to provide the mentors with insights into the strengths, interests, and weaknesses of their students. As the education director explained in an interview in June 2006, "Certainly it's been an issue and a number of teachers have mentioned to me that … one of the stumbling blocks … was the amount that we expected – the pre-programme paperwork." In 2007 and 2008, a student profile form (to be completed by teachers) was developed by the education director, as well as a clear statement of the expectations for enrolling students. These steps ensured that all information regarding students in the programme was easily collated and shared with mentoring staff.

The information gathered during the identification and application process was vital to the implementation of the programme. It was used for grouping students into research teams, based on their reported strengths and weaknesses, as well as learning preferences. Mentors were provided with a summary report for each student in their team prior to the start of the programme, giving them a sense of the potential group dynamics, range of abilities and interests, leadership capabilities, and so on, to aid them in planning. This information was also used for creating the island groups, again, aiming for a mix of different backgrounds and interests. In other words, the identification was used as a means to an end (not an end in itself) as recommended by the Ministry of Education (2000).

As the programme developed over the three year period, the quality of the information received from schools and students continued to improve, as did the nature of the information sought by the NZMSC. For example, the student application form was refined in 2008 to include:

  1. A Metacognitive Awareness Inventory
  2. Preferences for learning in science, including differentiated strategies
  3. Roles in team work
  4. Leadership styles

This information was used for grouping, teaching, and as a pre-assessment to measure the programme's effectiveness (the section on evaluation further describes this). It was also used to provide professional support and direction for the mentors. For example, analysis by the education director of the Metacognitive Awareness Inventory showed that students had some consistent areas of weakness. The statements were turned around into questions or prompts mentors could use with their teams to reflect upon their investigations. An overwhelming negative response to the statement, "I focus on overall meaning rather than specifics", became "Let's step back from the detail and look at the big picture." The identification information was an important element of the planning and delivery of the programme, as this section has shown. Once students are identified as gifted and talented, the next important step is to provide a qualitatively differentiated programme, as will be described in the next section.

Differentiating for Gifted and Talented Students

Four aspects of teaching and learning should be differentiated to meet the needs of gifted and talented students: content, process, product, and learning environment (Ministry of Education, 2000). The catalyst for differentiation is the teacher, or in the case of the NZMSC programme, the educational director and mentors. This section of the report will describe these elements of differentiation, highlighting the programme's development and effectiveness over the three-year period of funding, as shown through the evaluation.

Content Differentiation

The programme had a broad, conceptual theme: making sense of the marine world aids survival and enriches lives. The theme was interpreted in different ways throughout the programme: for example, whole group sessions introduced the concept through a discussion of the marine world and human interaction with it, explicitly highlighting how these understandings aid survival and make lives richer, placing science in the 'real world'. Laboratory experiments introduced the ways in which marine creatures use their senses to aid their own survival, and this analogy was extended to other real life experiences. For example, some animals use camouflage as a mechanism for survival, just as some gifted students hide their abilities in some situations. Students also extended their own sensory experiences through the tools and equipment used in a marine environment – from the research vessel to lab equipment.

The aim was to teach key ideas and concepts. In their reflections, the mentors showed a shift in their approach to teaching key ideas and concepts; in 2006 only one mentor (of six) reported 'always' teaching these and four 'usually' did, whereas in 2008 four mentors (of nine) indicated 'always' and four 'usually'. Interdisciplinary connections were also being made, and again, the mentors reported a shift: in 2006 two-thirds of the mentors always or usually taught this way, whereas in 2008, all but one did so. This conceptual, interdisciplinary teaching became a major focus of the whole group lectures and discussions as the programme developed, with the mentors and other scientists acting as guest speakers to share their research and its applicability across many aspects of life. One mentor, in particular, discussed the connections between art and science, comparing and contrasting Beethoven's work with that of Darwin. Another mentor was engaged in a multidisciplinary study working with local Māori, and was able to share bicultural perspectives of scientific knowledge and processes. The content of the programme was complex, interrelated, and abstract – and this was enabled by the use of a broad conceptual theme.

This theme also allowed for a range of different marine topics and creatures to be explored. Each year, the mentors selected a topic for inquiry, sometimes based upon their own research in marine studies and other times venturing into another area. A sampling of the topics of investigation shows that each inquiry gave students an opportunity to explore how marine animals use sense and response to survive:

  • The meiofauna of intertidal sand ecosystems
  • The feeding relationships of the southern arrow squid
  • Animals and plants in the Papanui Inlet ecosystem
  • The feeding dynamics of seastars and cockles
  • The relationship between sediment and macrofaunal communities
  • The zonation of seaweeds and its relationship to differences in polysaccharide extracts
  • Hermit crab behaviours
  • Cockle ecology, with particular reference to shell strength

As the above list shows, the content was advanced, with students often utilising university level textbooks and academic journals to develop their understandings. The content was also 'new' for students, with the majority reporting on their PMIs that a positive element of the programme was being given the opportunity to learn about something completely new, interesting, and different. These student comments explain their sense of delight with their new learning:

I knew literally nothing! (Student, PMI, 2007)

I come from inland and wasn't taught anything in school. So now – WOW! (Student, PMI, 2007)

As the second comment shows, for students from 'land-locked' regions, being in the marine environment was a completely new and different learning experience. The laboratory and its surroundings enabled an examination of content that was unfamiliar.

The content was challenging for students of this age and level: classroom observations showed that in over two-thirds of the recorded observations, mentors were presenting "content that challenges students". This was confirmed by the mentors in their reflections, who reported they 'usually' provided this (three-quarters responded this way in 2008). Students also commented on the challenge, with some seeing it as too difficult, particularly the scientific and technical language. This was not, however, true for most students, many of whom enjoyed the challenge and as one commented, "I'm not being taught what I already know" (Student, PMI, 2007). Students also felt it extended their current knowledge through the depth of coverage.

However, depth of learning was not as readily observed by the researchers using the COF (observed less than half the time). Yet, between 2006 and 2008, the mentors reported an increase in their attempts to always provide a depth of coverage. In 2006, no mentors felt they were 'always' emphasising depth, whereas in 2008 nearly half reported they were always doing so, and for the rest, this was 'usually' the case. The students, as shown in their PMI responses, felt that the content was in far greater depth than other experiences, as the responses below show:

Very interesting topic, far more in-depth and engaging than shcool science. (There is a reason for my spelling.) (Student, PMI, 2007).

It's a lot more intense – more info is packed into our brains. (Student, PMI, 2008)

It's great to learn about previously learned topics (i.e., photosynthesis) in greater depth. (Student, PMI, 2008)

An important element of the scientific experiments undertaken was that while the mentors set the stage with background information and some ideas, the experiments themselves were for the most part designed by the students. This enabled the students to also be taught content related to 'how-to skills': hypothesis-making, reliability, validity, problem-solving, and so on. Students were being taught the skills of a scientist, and many commented that 'what' they were learning about that was both positive and interesting was dissection or how to use lab equipment. They made reference to enjoying the practical knowledge needed for engaging in the scientific process.

Based upon the evaluation and concern that the outcomes related to metacognition and team roles were not being measured, a focus on 'thinking about one's thinking' came into its own in 2008. The students had whole group sessions on thinking styles, designing qualitative models of science, Belbin's roles in team work (Belbin Associates, 2007-2008), how the brain works, analogical thinking, and so on. The students' views on this element of content ranged from positive to negative, as these comments show:

I like the metacognition … like it helps me think about what I'm thinking … I reckon it develops your thinking more … it helps you sort things out. (Student, Focus group interview, 2008)

Well it's pretty interesting the whole thinking about thinking. (Student, Focus group interview, 2008)

… like [named Educational Director] often goes off on tangents about thinking about thinking. I mean maybe he's that kind of person who likes random abstract thinking but I hate it … I don't really consider thinking about thinking to be a worthwhile use of my time. (Student, Focus group interview, 2008)

Over the three year period, the mentors' reflections showed a shift towards greater use of modelling metacognitive strategies, providing opportunities for students to think about their own thinking, and having students reflect on their own performance. For example, one mentor explained his own doctoral hypothesis and his thinking behind it as he guided students into their own research investigation. Another mentor commented, "One student, while discussing a problem, said, 'I am thinking about my thinking!'" (MRF, 2008).

Overall, the content of the programme was differentiated in most aspects: abstract and centred on a broad theme; interdisciplinary; in-depth and with breadth; and advanced (Ministry of Education, 2000). Only one content-related indicator of the COF was seldom observed or reported by mentors in their reflections: bicultural and multicultural perspectives of knowledge. This lack of attention to cultural knowledge was, as one mentor reported, highly dependent upon the topic of study; mentors tried to integrate this knowledge when it was appropriate. Another factor which may have influenced this was the cultural make-up of the students and mentors, the majority of whom would identify as Pakeha, and many of the mentors were from overseas – Canada, Germany, the United Kingdom, the United States. One of these 'ex-pat' mentors, however, was a speaker of Māori, identified with her Māori whanau, and was engaged in marine research involving local iwi. Interestingly, on their PMIs, some of the students acknowledged the diverse cultural backgrounds of their mentors, noting this as an 'interesting' content-related aspect.

A final aspect of differentiation was in relation to planning mutually, reinforcing content. In 2006, as was noted previously, much of the planning was undertaken by the educational director and some of this was outlined in the proposal and milestone reports. Over the 2007 and 2008 year, this planning became more detailed and transparent. As he explained in a focus group discussion with the mentors:

I came into this with a bit of a general idea of what was going to happen... I don't know if you noticed, but the itineraries have become a little bit more precise and so for me the itinerary is the plan, it's the structure and strategy. (Educational Director, focus group interview, 2008)

The educational director's planning was around the 'big picture' and mentors were expected to conduct their own planning around the team investigations. In 2006, there was little evidence of their planning, with half the mentors stating that they 'sometimes' planned, and only a few instances of planning evidenced in observations. By 2008, two-thirds of the mentors reported that they 'always' or 'usually' planned. Planning was discussed at length in the 2008 focus group discussion with mentors, revealing that the mentors were taking time to plan the beginning sessions; however, as the sessions progressed and investigations were underway, they tended to step back, allowing the students to do the planning or to work alongside the mentor in planning.

There was strong evidence to support this approach to planning: the observations showed that the most frequently observed indicator was the communication of the purpose and objectives to students (two-thirds of observations). The mentors explained that this was how they began the planning with students: "I recapped it … sitting down and going, 'What are our objectives? How are we going to do that?'" (Mentor, focus group interview, 2008). Another mentor described the importance he placed upon planning the first session, during which he moved students from the broad topic of science to their specific area of investigation, infusing it with metacognitive strategies, and discussing their roles in a team. As the programme progressed, this same mentor was able to really step back and let the team pursue their topic very autonomously. It seems planning was critical in the beginning sessions of the programme, however, there was also a need for this gradual shift towards autonomy and mentor- independence .

Differentiated content included opportunities for students to explore science knowledge and skills, framed by a broader conceptual theme, while concurrently examining a range of metacognitive strategies, how-to skills, and team roles. The content of this programme was closely tied with the differentiated process skills – how students were learning – which is examined in the next section.

Process Differentiation

The learning process of scientific investigation has been described throughout the preceding sections of this report. The framework of the programme was very much focused upon team inquiries, but balanced with whole group discussion sessions and activities. Within that framework, how were students learning? Their responses to this question over the three years consistently highlighted 'hands-on learning' as the most positive process. This was evidenced in their PMIs, focus group discussions, and end of programme survey. The mentors' reflections and focus group discussions verified this approach, with all mentors reporting a consistent use of hands-on approaches (MRF, 2006, 2008). Observations also showed this hands-on learning was an integral aspect of the programme.

It is not surprising that the students reported enjoyment in the 'doing' of science – one would expect a hands-on approach to experimentation. The points of difference in this programme's hands-on learning opportunities and other experiences were the intensity and duration of investigations of advanced content using authentic processes. For a period of several full days, students were examining real scientific problems and mirroring the work of marine scientists. Whether dissecting squid in the lab, dredging the seabed on the marine vessel, collecting seaweed specimens on the beach, observing sea lions, or testing mussel shell strength, the students were mimicking the scientific processes, using equipment and scientific processes, in the marine environment. As one student commented, this was a "pretty practical way of learning" (PMI, 2007).

Other student comments further support this approach. The following is a discussion with four students in a team (Focus group discussion, 2008), highlighting the duration, intensity, and authenticity of the investigations:

Student 1: At school in science the little investigation things we've done have just been very short, 1-2 periods long sometimes. But here it's much more drawn out.

Student 2: Yeah because it means we can test for longer …

Student 3: Yeah get more data and make our results more thorough and more accurate.

Student 4: And it's like you're in the scientific environment … if you were going to be a scientist when you grow up you would see how you would do things and stuff …

The students' experiences of science, prior to this programme, had in the main been confined to school-based science classes (with some experimentation in those), science fairs, and the real world of farming (some students reported animals 'autopsies'). Therefore, their responses are not surprising. In a 2008 interview with two teachers from different contributing schools, the teachers expressed their support for this approach and frustration regarding the constraints of schooling:

Teacher 1: It's something I would fully support as being in addition to straight schooling. It's not something we can do necessarily in our own school because of the numbers …

Teacher 2: I'd like to take this back to the staff... I'm sure a lot of them know that you lose this in day to day teaching – just how amazing these gifted children are at picking up the ball and running with it.

These teachers also commented on the easy access to scientific equipment in a specialist facility as a differentiated factor from school (the section on the learning environment further explores this).

The students enjoyed the shift away from 'set tasks' and traditional schooling, as these comments show:

I like it that there are a lot more hands-on approaches than just copying a whiteboard and jotting down notes. (Student, focus group interview, 2008)

Lots more dissection and practical work, compared to copying notes from school and no discussions or continuation. (Student, PMI, 2008)

Students were engaged in recording their data and analysing the results, but were not limited to paper and pen, having access to computer programmes and software. In 2006, the access to technology was limited, and this became a major point of discussion in the mentor focus group. They expressed frustration at the lack of access to computers. In 2007, this was resolved with the purchase and availability of more computers, as well as some mentors allowing students use of their personal laptops.

Another element of differentiation was the use of collaborative team work, whereby not everyone did everything all the time. Students had choices in the roles they undertook, whether it was setting up an experiment, recording data, analysing the data, or compiling the results for presentation. The mentor reflections showed that over three-quarters of the mentors were always or usually aiming to accommodate individual differences through material selection or task assignments, and all mentors always or usually addressed different modes of learning (2006 and 2008 MRF). In 2006, when asked to reflect upon their use of flexible groupings, only one mentor indicated this to be the case, with the others either never doing this, or not responding to the statement. However, in 2008, eight of the nine mentors responded that they always or usually allowed for flexible groupings within their teams. The observations verified these accommodations for individual differences.

As the student teams progressed through their experiments, there was also a great deal of discussion and debate between them and their mentors. These discussions were highlighted in student PMIs as one of the more interesting elements of how they were learning: one student described this as "big discussions" (PMI, 2008). The discussions were facilitated by mentors who used a range of critical thinking, problem-solving, and metacognitive strategies. In their reflections, mentors consistently reported their use of brainstorming, comparing and contrasting, making judgements and evaluations, defining problems, and exploring interpretations – with all mentors in 2006 and 2008 reporting 'always' or 'usually' facilitating these critical thinking and problem-solving skills. There were, however, positive shifts in their greater use of some strategies: generalising from specific to abstract; synthesising or summarising information; debating points of view and developing arguments; selecting and implementing solutions; and modelling, facilitating, and reflecting upon metacognitive strategies.

Another shift in the process of learning occurred over the three years of the programme: the mentors became better skilled at questioning. In the 2006 observations, the research team noticed a tendency for the mentors (all trained scientists, not teachers) to simply answer student questions, find solutions to problems, and almost over-direct the student investigations. This was discussed with the educational director, who in 2007 and 2008 provided more professional development and support focused on questioning, metacognition, and critical thinking. This response from a mentor, who had been working in the programme for three years, explains her development of questioning skills:

Yeah heaps. The sense of come up and ask you a question … initially you might have a couple of answers just to nudge them in the right direction but its grown so that I'll answer it with a question and go, "well, what do you think?" (Mentor, focus group discussion, 2008)

The other mentors confirmed this example, describing these strategies to questioning: avoiding leading questions; changing the way a question is asked; breaking down a big question into manageable blocks; and sometimes simply admitting that they didn't know an answer (Mentor focus group, 2008). This skilled questioning was observed by a visiting teacher who stated, "He has done the best inquiry questioning I have ever seen in a teacher" (COF, 2008).

Students were for the most part positive about having the opportunities to "think for ourselves" (Student, focus group discussion, 2008) – or as one student wrote, having "to think for once" (PMI, 2007). At the same time, some students struggled with the level of discussions, particularly the 'big words' being used. Some students felt there was "too much talking" (PMI, 2007). Other students struggled with the level of self-direction and decision-making required. One mentor, for example, shared her experience of a student who "every step of the way … wanted basically reassurance on what ever action she is about to take" (Focus group discussion, 2008). However, these students who struggled with or disliked the opportunity to think critically and act independently were by far a minority.

The teaching and learning processes in the marine studies programme were overwhelming considered positive or interesting by the students. Their negative comments on the PMI and in focus group discussions were much more individualistic and often based upon their team's experimental focus: some students didn't like dissecting animals; others disliked the amount of repetition required for testing their results; some complained about the 'smells' in the marine laboratory or the 'freezing' temperature on the beach; a few thought the pace was too fast or the processes to challenging.

This programme has the hallmarks of process differentiation: a balance of independent and group opportunities; higher levels of thinking, including metacognition; problem-solving and problem-finding; open-ended, using problem-based learning strategies; authentic roles, skills and expertise; and the development of research, planning, management, and decision-making skills (Ministry of Education, 2000). The hands-on learning process, facilitated by the mentors, enabled all aspects of appropriate process differentiation, embedded in advanced content and leading to real solutions to real problems – the products of the students' learning.

Product Differentiation

At the end of the secondary school programme, the 'grand finale' was a presentation to parents and whanau, teachers, marine scientists, and other invited guests to the NZMSC. This was an opportunity for students to share their learning with a differentiated product to a wider audience. The Ministry of Education (2000) explains differentiated products as being the result of real problems, requiring a shift in students' roles from consumers of knowledge to producers of knowledge. As has been outlined in previous sections, the scientific inquiries naturally lend themselves to this aim. Students were using real scientific processes in their experimentation, but of equal importance were the processes of presenting their findings, as this section of the report will explain.

In 2006, the students were presented mid-way through the programme with a series of challenges for the presentation of their work. This included a sound challenge, visual challenge, and an oral and visual presentation. Student teams produced a variety of products in response to the challenges: original songs, skits, puppet shows, video, powerpoint presentations, brochures, posters, sound recordings, and even food (chocolate mousse derived from seaweed extracts!). These were self-selected by the teams, apart from the oral presentation of their research, which was an expectation of each team.

As was explained in the programme design section of this report, this set up hindered the overall quality of the projects, and was changed in subsequent years. The SPAFs showed that with each presentation, there were missing elements, and these varied across the team presentations. For example, one team designed a poster for the local beach, warning visitors of the care to be taken in their natural habitat – meaning that many factors of the SPAF (e.g., problem focusing; logic, sequence and transition) were not included, while the appropriateness for the audience was outstanding. Another team was able to clearly articulate their aim, methods, results, limitations, and conclusions, but due to technical difficulties, the powerpoint presentation was not of a high standard. It was concluded that the students simply had too many 'challenges', with insufficient time or resources to create products of a high quality. It was also clear that the programme had not incorporated some teaching of the 'how-to skills' of product development.

Thus, in 2007 and 2008, students were introduced to the expectation of a final presentation at the beginning of the programme, encouraged to work through the development of these throughout the programme, given trial runs and practices for constructive criticism from their peers and mentors, and provided with examples of products for critical discussion (e.g., scientific posters in the NZMSC, powerpoint presentations). The challenges were also refined so that each group produced a scientific poster, an oral and visual presentation, and a 'novel presentation' of their choice. The SPAF was also used to evaluate the products as a package rather than as individual products. As was stated previously, the lack of adequate computers to support final productions was also addressed. Finally, it was decided that the team posters would be placed on the NZMSC website for public sharing.

The students enjoyed the product challenges, especially the opportunity for student choice across a variety of presentation options, as was highlighted in their PMI responses across all three years. Overwhelmingly students reported the presentations as a positive element of the programme, including being part of the audience for other presentations. They enjoyed the opportunity to share their work, as well as learn about what other teams were doing. Students also favourably viewed the technology readily available to them, especially some of the software programmes and the value of these applications in creating presentations.

The students took on different roles within the team as they prepared their presentations, utilising their personal strengths and interests. For example, the student who was writing the science-fiction novel designed a series of characters for a team skit. A student who had a strong grasp of chemistry took charge of explaining the chemistry-related analysis to a wider audience. When one team was faced with a very reluctant public speaker, they designed a role whereby as the other students discussed the project, she would draw the process on the whiteboard. This reluctance and nervousness about public speaking was shared by other students, who felt this was a negative aspect of the product differentiation. As one student wrote, "genii not good public speakers" (Student, PMI, 2007). Having to overcome shyness, nerves, and speaking to an audience of peers, teachers, family, and marine scientists was a real challenge for some.

The mentors facilitated the swapping and switching of roles, as one explained in the 2008 focus group interview:

… about half my group really wanted to swap their roles and the other half … wanted to do what they wanted to do … they had it all sorts and they didn't want to do that other thing … so in the end I went "okay I think everybody should do everything because everybody has got strong points and everyone has weak points and this is a learning experience for all of us." (Mentor, focus group discussion, 2008).

Overall, the students appreciated these chances to try their hands at other roles in the team. As one student explained, "Everybody contributes different ideas … it is good to have other people who have different strengths and weaknesses" (PMI, 2008).

In addition, the mentors provided ongoing feedback throughout the product development; the key to ensuring the quality of the products was this provision of constructive feedback and criticism. This was provided in several ways: in 2006 and 2007, at the end of each day, teams presented their progress to date on all aspects of their investigation, including their presentation. During these sessions they gauged their progress in working as a team, achieving their goals towards completion, explaining their experiment, and so on, in a session with their peers and the mentors. The audience could ask questions and provide feedback. Teams also met individually with their mentor to reflect upon their progress and begin making plans for the next session.

Having the opportunity to hear others mentors' perspectives was valued by the students, and beginning in 2008, this format was changed whereby each group spent time in discussion with a different mentor using a Team Reflection Sheet (devised by the programme director). This was a form of 'shared' mentoring. The purpose was to review each group's progress to date, but also to discuss the team roles each member had played and metacognitive strategies that were used. These reviews were met with mixed opinions: some of the students found the questions tedious, while others enjoyed the discussion; some mentors enjoyed getting to know and work with other students, while others really wanted the time to plan with their own teams. As one mentor explained, a "balance" of the shared mentoring approach, whole group, and individual team meetings would address these issues (Focus group interview, 2008).

An example of the quality of the presentations is best summarised by the results of the SPAF for the 2007 rural teams. The average rating for the team presentations, as assessed by two of the researchers, was four across all indicators [on a scale of 1 (poor) to 5 (outstanding)]. These ratings show that students were able to state their purpose; remain problem focused; use logic, sequence, and transition; and remain action-oriented in their presentations. They also demonstrated an ability to use a high level of diverse resources appropriate to their presentations of their investigations. They were able to appropriately address their audience in their products. Overall their products reflected original ideas (especially the novel presentations). The most outstanding element of their products was their advanced familiarity with the subject – in the oral presentations, some students' abilities really shone through in their explanations and responses to audience questions. As one researcher noted, "They coped extremely well with the follow-up scientific questions posed by the audience" (SPAF, 2007).

Overall, the product differentiation adhered to the principles of qualitatively different student products, as outlined by the Ministry of Education (2000). The students addressed real problems, using real methods and techniques. The products were critically evaluated by themselves, their peers, and mentors in a formative manner. These were presented to an appropriate audience, and the ongoing participation by marine scientists, who showed a genuine interest in the students' efforts, attests to the quality of their work. In a different learning environment, this may not have been so readily achieved. What did the NZMSC offer by way of a safe, responsive physical and psychological learning environment for gifted and talented students?

Differentiated Learning Environment

The physical learning environment of the NZMSC programme has been previously described: students had access to lecture rooms; a teaching laboratory with running seawater and associated microscopes and scientific equipment; and an Aquarium. Additionally students were able to utilise the Portobello Marine Laboratory, the research vessel Polaris , and the local marine environment. Quarantine Island provided accommodation and cooking facilities for overnight stays. The overall physical environment was authentic and specialised, but it was also flexible in that students could move freely throughout the facilities based upon their learning activities. This flexibility enabled the pursuit of learning that was in-depth, supported by high-level materials and equipment. It was dramatically different from a traditional classroom environment, as these student and teacher comments show:

The lab is not like school. It's proper. (Student, PMI, 2006)

… you don't get to work with like live animals and that kind of thing (at school). It's just sit there, copy down the notes and try and just pick it up yourself. Here you actually get out there and do it all. (Student, focus group, 2007)

I think another aspect of the science side of things is the equipment. Here they've got oxygen meters … tanks … the set up, running seawater tanks … I don't have that equipment. The schools don't have the money to buy it and in some cases the money to maintain it and have it adequately stored. (Teacher, interview, 2008)

… here it's completely concentrated on marine biology and so it has … the best equipment, the best kind of places to actually do it. (Student, focus group, 2008)

The physical environment not only facilitated in-depth, hands-on, specialised learning, but it also provided an atmosphere in which students could be themselves and in which it was safe to be gifted. The psychological and social-emotional environment recognised and supported giftedness and talent – and overwhelmingly, students saw this as one of the greatest benefits of the programme.

Over the three year period when students were asked to comment on the learning environment of the NZMSC programme, their responses consistently related to the opportunity to work with like-minded peers in an environment that was warm and accepting, with shared responsibility for learning. The students specifically discussed the differences between their school-based peer group (often mixed ability) and their peers in this programme in relation to pace of learning, effort, behaviour, commitment, abilities, strengths, and interests. As one student stated, "I enjoy meeting people as smart as me" (PMI, 2008). For some students, particularly those from small rural schools, this was their first opportunity to work alongside equally able students, and this was initially threatening, but eventually comfortable. As one student explained, "It can be kind of scary but that's cool" (Focus group, 2007). In 2007, a member of the research team received an unsolicited letter from a parent of a student in the rural group which describes the benefit of working alongside like-minded peers:

I have unlimited praise for the programme that my daughter attended … I am so pleased with the friendships she has developed … and the intellectual satisfaction she has experienced. At last she has realised that there are more students like her. She was accepted by students and staff without discrimination for the first time ever. It was great for her to have had time where she wasn't misunderstood and treated like a freak or bullied by others. (Unsolicited letter from parent, 2007)

This acceptance and enjoyment of like-minded peers was purposefully facilitated with the following key strategies, as highlighted in the final milestone report of 2008:

  • Using a range of identification methods, but targeting a small percentage of the school population (2-3 students from each school's Year 10 cohort).
  • Using a broad-based conceptual theme that relates to understandings of self.
  • Purposeful grouping and re-grouping of students based on strengths, as well as for different activities (e.g., research teams and social teams).
  • Providing hands-on experiences, both scientifically generic and specific to marine science, for small research teams facilitated by postgraduate research students who scaffolded a gradual transition of ownership and independence.
  • Expecting reflection, critique, and communication of science from students as individuals and teams.
  • Including a residential component with shared living responsibilities (e.g., cooking and cleaning), as well as social challenges related to the programme theme.
  • Bringing together students from a range of different schools, and thus, different backgrounds and experiences.

In addition, the programme has developed into a sequential programme with opportunities at Years 6-8, Year 10, and Year 11. It also provides opportunities for complementary programmes for individual schools. Finally, the programme staff see themselves as a "research team that continues to monitor how the programmes are meeting student and school needs" (Milestone Report, 2008, p. 16). The ongoing evaluation of the NZMSC programme and its many complex and interrelated differentiated strategies was a critical component of the evolution and success of the programme as the next section describes.

Evaluating the Programme

As a TDI, one of the expectations of the NZMSC was the development, implementation, and analysis of internal tools of evaluation by which the programme could be monitored in relation to its outcomes. The programme's self-evaluation was complemented by the Ministry-contracted evaluation, and over time both monitoring processes evolved and changed in response to the findings. This section will focus on the NZMSC's evaluation processes over the three year period, highlighting the findings of the evaluative research.

In 2006, the NZMSC utilised a range of methods of formative and summative, formal and informal, quantitative and qualitative, programme evaluations. For example, this included a before and after analysis of student applications and research proposals; a thinking web constructed from the central theme of the programme; mentor observations and reports; parent, teacher and student interviews with the coordinator; and questionnaires for students, parents, and teachers. This was a multi-faceted approach inclusive of the programme's stakeholders, and showed some positive trends, but it was "a work in progress" (Milestone Report, July 2006) and not without problems. The research team observed the administration, collation, and analysis of the evaluation tools was not consistent across the two programmes. In other words, whilst the evaluation was comprehensive in scope, it was patchy in actual collection, analysis, and, subsequently, findings. Another difficulty that arose was in regards to measuring student outcomes related to the programme goals of metacognitive, team work, and social/emotional development.

In 2007, the tools were refined by the NZMSC, including the internal adoption of the PMI, MRF, and COF (which were being used by the research team). The programme's educational director made a major effort to ensure that the administration, collation, and analysis of the methods were more consistent across the two programmes. This was achieved, in part, by the provision of administrative support, but also because other systems of organisation were in place. At the end of 2007, the educational director reviewed the desired outcomes and monitoring by creating a matrix. The result of this self-review showed that while most outcomes were being successfully measured by the tools developed and implemented in the first two years of the programme, they were still not providing adequate evidence of shifts in metacognition, team work, or social emotional development. Additionally, it showed that more frequent assessment of the development of science skills and understanding was needed.

The education director explained in an interview (July, 2008) how the evaluative questions and methods shifted over the three year period. This is evidence of the evolution of the programme, how the questions shifted from broad to specific, and how the tools were adapted and refined to answer those. The NZMSC evaluation became more practical and focused over the three years of the programme.

… the initial thing was a question of whether we had captured them and their interests and motivated them … so there was a sheet we used about the first day to see what was happening … I think the last two years we haven't used that because it became clear that we were just repeating ourselves and that was fine. And the next level was to do with … what elements of it were essential to the learning process and the adventure … what were they actually learning in terms of skills, attitudes, knowledge. What I'm thinking we need to do now is to really ask the question how has your work or attitude or understanding of science and your learning changed … and what skills or attitudes or knowledge will you take and use in your … school, in your life. So basically … just two questions … (Education Director, Interview, July 2008).

Prior to the start of the 2008 programme, the educational director visited Massey University to work with the research team to refine the evaluation tools. This allowed for potential barriers to effective evaluation to be addressed: time; expertise (in terms of access and input); and planning (Programme Director and Coordinator, interview, 2008). A Metacognitive Awareness Scale was included in the student application to be used as a pre- and post-assessment, and students were asked detailed questions regarding their learning preferences, also as a pre-post assessment. Table 7 shows the evaluation methods being used at the end of 2008. It is inclusive of all stakeholders, but also very specific and detailed in its focus, with closer alignment to the programme's intended outcomes.

As the education director explained, "there is a lot less assessment and monitoring … because … if your intention is to aid the evolution of the programme and the programme has evolved then at some point you've got to say, well what is the purpose of the continual assessing and doing it?" (Interview, 2008). This continuous reflection and refinement of the evaluative tools, as well as focused analysis of a comprehensive data set of relevant information, has enabled closer alignment between the programme outcomes and the internal evaluation processes.

Table 7: 2008 Internal Evaluation Methods
Notes:
  1. Pre-post.
  2. Adapted.
Evaluation Methods Stakeholders
Students Mentors Teachers Parents
Application, including Metacognitive Awareness Scale 1


Skills, Knowledge and Attitudes Sea Star


Plus, Minus, Interesting: Differentiation


Classroom Observation Form
2
Focus Group Discussion/Review


Questionnaire
Products (e.g. conceptual diagram, presentations)



As with other aspects of the NZMSC programme, the internal evaluation process changed and developed over the three year period. The education director described a shift in confidence from the first to the final year, explaining that initially there may have been a sense of 'over-evaluation', but that by 2008, "… I've actually got more confidence that we are on the right track … so I'm not quite as anxious about it now" (Education Director, interview).

This confidence enabled the programme coordinators to sharpen the evaluation focus and subsequently make changes to the programme. In the final milestone report (December 2008) the programme development and sources of feedback related to those were detailed. For example, based upon urban student feedback, a residential component was added in 2007 and expanded in 2008. When mentors reported inadequate computer facilities in 2006, improved access was provided for subsequent years. In 2007, mentors expressed a desire to be involved in the residential component so they could gain a fuller sense of the programme, so in 2008 mentors were rostered for overnight stays on the island. Contributing teachers requested more detailed individual student progress reporting in 2007 and this was carried out in 2008. What this shows is that the evaluation of the NZMSC programme led to specific changes in the programme – it was a means to an end, not an end in itself – as recommended by the Ministry of Education (2000).

By the final year of the programme, the directors of the programme were also in a position whereby they recognised the importance of sharing the information they had gained from their evaluations with others. The programme had developed a 'reputation' amongst teachers, schools, students, parents, and communities, and it was felt this was the strongest evidence of the success of the programme, but an objective for 2008 was to gather some "publishable data and set the framework for that really early on" (Programme Director, interview, 2008). There was recognition by the directors that "It would have been good to do that right from Year 1 … we weren't really in the mind set to do that" (Education Director, interview, 2008). This is yet another demonstration of the growth of the programme.

A final factor impacting upon the NZMSC internal evaluation was this Ministry of Education-contracted evaluation. Throughout the three year period, the researchers worked closely alongside the programme directors, consistently seeking their feedback and views of the evaluation during interviews. Several themes emerged:

  • Some of the tools utilised by the research team were adopted by the NZMSC and utilised across programmes.
  • The external input and expectations of the research team assisted in the collection, analysis, and reporting of data.
  • The expertise of the research team helped shape and guide the development of the programme and its evaluation.
  • Professional relationships emerged.

In the final interview in July 2008, the education director also shared two concerns: the fact that the research team was "separate from us" (based at Massey University, as opposed to Otago), but also the recognition that the Ministry-contracted evaluation was coming to an end. As he stated, "I'm concerned about not having you. I'm becoming addicted to it!" These emerging themes demonstrate the value of using an action-based evaluative approach, but also relate to resourcing. The next section of the report further elaborates upon resources to support the NZMSC programme.

Resourcing

The NZMSC programme was supported by human, physical, and financial resources. In regards to both human and physical resources, these were specialised: postgraduate marine science students acting as mentors guided by a highly skilled educator with understandings of gifted and talented education in an authentic marine laboratory setting. The physical resources have been described in the earlier section on differentiated learning environments. Human resources have also been described in previous sections and will be further elaborated upon in the following section on professional support and development. This section of the report, therefore, will focus upon financial resources.

The NZMSC programme was primarily funded by the Ministry of Education (as a TDI), with some costs absorbed by the University of Otago. The TDI funding was utilised to cover expenses related to:

  1. Professional fees (salaries for personnel: directors, mentors)
  2. Professional costs (travel, training, and consultancy)
  3. Operational costs (communication, printing, resources, facilities, equipment, boat rental, transport and accommodation)
  4. Professional development funding (teacher release days)

More than half of the overall funding was used for professional fees; this is because, unlike school-based providers of gifted and talented programmes, the salaries for personnel are not provided by the Ministry of Education, but are reliant upon contracted services. Thus, the programme was almost completely dependent upon TDI funding, with the only costs to students or schools being for some transportation and meals. Over the three years, the NZMSC directors were aware of issues of sustainability (further discussed in a following section) and sought further sources of funding to maintain an ongoing programme.

In 2007, the NZMSC received funding from the Todd Foundation to support the rural Year 10 programme, as well as their participation in a newly developed 'user-pay' Year 11 programme. The NZMSC also receives LEOTC funding and over time more programmes for gifted and talented students (complementary to the Year 10 programme) were being offered through this structure. In 2007 and 2008, the NZMSC sought input from schools regarding their willingness to partially fund student participation. Most schools indicated an inability or lack of interest in providing financial support. In 2008, the research team queried parents and teachers about funding (in an online survey). Twelve of the 14 parents (86%) indicated the Ministry of Education should fund the programme, with comments alluding to parent and school inability to pay full costs. Of the teachers surveyed, most indicated that they did not know if their schools would be able or willing to pay the costs. Although the NZMSC has always had a focus on the sustainability of the programme, it could be the case that in its initial funding, with no substantial costs to schools or students, an unsustainable programme was developed.

In 2008, the NZMSC applied for an extension of funding, in light of the Ministry's review of gifted and talented education initiatives, and received partial funding for professional fees. This funding, coupled with the Todd Foundation grant and funding from the University of Otago, will allow the programme to continue in 2009, but not without costs to students. Rural students will pay a small fee, whereas urban students will pay a much more substantial fee. In addition, the number of overnight stays will be decreased, as will the level of professional development and support provided to schools. The resourcing for the future will no doubt create issues of inequity, as some families will be unable to financially support their children's attendance, and there remains a need in New Zealand for access to ongoing teacher professional development.

Overall, the resources to support the NZMSC programme allowed for the development and implementation of a specialised scientific programme for gifted and talented students. The financial resources mainly supported the provision of expert personnel, a cost factor unique to out-of-school providers of gifted and talented programmes. However, unlike other out-of-school programmes, there were minimal costs to participating students, their families, or schools. Other costs included operational costs and teacher release for contributing schools. The next section details the professional support and development for NZMSC staff, as well as teachers in participating schools.

Professional Support and Development

The NZMSC programmes provided two levels of professional support and development: support and development for its own staff and opportunities for staff from contributing schools. As has been explained previously, the NZMSC employed a part-time education director and part-time programme director. Their roles were complementary and supported the postgraduate students who acted as mentors. These mentors were not experienced or trained teachers, but rather research students in marine science, and their main source of guidance came from the education director who was an experienced secondary school teacher. In addition, the NZMSC provided professional development to contributing teachers by way of observational days while the programme was 'in action' and an annual professional development day. Given the evaluation focused primarily upon outcomes for students, this section mainly discusses the NZMSC staff development and support, and the effects of the contributing teachers' professional development upon the NZMSC programme.

A unique aspect of the NZMSC programme was its utilisation of expert mentors, postgraduate research students, to guide and facilitate the team projects. This role was complemented by whole group seminars and discussions led mainly by the education director, but also inclusive of mentors and other scientists who shared their expertise. The mentors played a critical part in the delivery of the programme and were supported by the educational director. The mentors were carefully selected, based upon several key criteria: passion for and expertise in their discipline; experience in other educational programmes (run at NZMSC) or an interest in gaining such experience; and availability and commitment to the programme's timetable. In addition, many of these mentors were 'young' postgraduate students, in their early careers. None of the mentors were qualified teachers.

This created somewhat of a tension when it came to providing professional development and support. There was some reluctance, particularly on the part of the education director, to turning these young experts into 'classroom teachers' or creating barriers to their passion by setting high expectations of planning, assessing their performance, utilising school-based approaches to teaching, and so on. Therefore, in the beginning stages of the programme, there was limited organised professional development for the mentors. The support they had was mainly one-on-one with the education director:

I go to the mentors and ask them what they are planning, how do they think they are doing … I'm the person … the mentors can come to and say 'oh I need this' … I'm the person that asks the mentors when they're going to do it and how they're going to do it. (Education Director, interview, 2006)

Despite this support, in the focus group discussion at the end of the first year, the mentors asked many questions about the nature of the programme, the characteristics of gifted and talented students, and the best ways of teaching these students. Beginning in 2007, a more structured staff introduction and training was provided which addressed these issues. Also, based upon observational and interview data collected as part of the evaluation, a professional development session on metacognition and questioning was conducted 20072. Overall, the professional development was considered a success, with participating mentors responding positively to the opportunity to learn and share, but the timing of the introductory session clashed with other commitments, so attendance was lower than expected (Education Director, interview, 2007). In preparation for the 2008 programme, greater consideration to timing and earlier mentor selection resolved this problem. Another unexpected form of professional development became the annual focus group discussions with the research team, which focused primarily upon differentiated teaching and student outcomes. This provided a forum for sharing and questioning, feeding-forward and backward, and shared reflection.

Table 8: Mentor Characteristics
Themes
Expertise
Student statements:
  • Experts: very knolwedgeable about marine life. I've learned heaps from them. (2006)
  • Well informed about chosen subject, less general than school teachers. (2007)
  • Being taught by University lecturer type people. (2008)
Mentoring (facilitating learning)
Student statements:
  • They let you do most of the thinking. (2007)
  • Very helpful and cheerful. (2007)
  • Teach and guide understandably. (2007)
  • Genuinely want us to learn. (2007)
  • Very good teaching methods. (2007)
  • Very interesting people who are willing to share knowledge with others so they can learn. (2008)
  • Let us take charge, give us a chance to work it out for ourselves. (2008)
Personable
Student statements:
  • Funny! (2007)
  • Laidback scientists! (2007)
  • They're human! (2008)
  • Really nice and friendly. (2008)
  • Helpful and cooperative. (2008)
Respect for Students
Student statements:
  • They trust us. (2006)
  • They don't treat us like little kids. (2007)
  • Just as interested in stuff as we are. (2007)
  • They are inspiring and handle our capabilities well. (2008)


The combination of one-on-one support, whole group professional development, and focus group discussions, alongside hands-on mentoring experience, was an effective approach. This is evidenced in student feedback across the three years: students viewed their mentors as knowledgeable, passionate, even 'cool', people who respected their students' abilities, differences, judgements, ideas, and so on. Major themes, describing mentors, and arising over the three years, are supported by student statements from the PMIs as shown in Table 8.

While it could be argued that many of these traits are based upon personality, the professional development focused these on meeting the needs of gifted and talented students. The professional support and development shaped these seemingly natural abilities and sharpened other skills, creating not only more effective mentors, but better scientists. In the final focus group interview with mentors (July 2008), they were asked to "share what some of the outcomes have been for you as a mentor, a scientist, a person involved in this gifted and talented programme."

Several themes arose from the discussion with mentors in regard to their own growth from being part of the programme:

  • Learning to communicate science : The mentors described the benefit of being able to communicate science in more common, everyday language. As one stated, "I've really learned how to communicate with people which I definitely apply to my daily life."
  • Questioning, including metacognitive thinking : The mentors admitted to some initial confusion regarding these strategies, but with professional development and support, most felt they were using these successfully. Two of the mentors shared how they use metacognitive thinking in their work as scientists, with one stating, "I've told my group when they say "I'm sick of thinking about my thinking" that it's actually quite useful to me … when I want to apply for funds … I have to get everyone to love everything and make it sexy for the judges … so I apply it in my own research." The other mentor applied metacognitive strategies to her mentoring, explaining, "... thinking about my own thinking and my own way of dealing with problems has been incredibly helpful." The mentors also discussed a growth in confidence to be able to ask questions (rather than giving answers) or tell students they didn't know an answer (even if they did). One mentor said these approaches were helpful in her University teaching of first year students.
  • Letting go : Several of the mentors discussed a shift from initially seeing their students' performance as a reflection of their own abilities as a scientist to coming to the realisation that, "its not about the actual science they're doing, it's about their development as individuals … the science is means to … really get them going and thinking."
  • Planning : The mentors were not consistent in their approaches to planning, with some stating that with experience, they planned more, and others indicating lesser planning. One mentor felt that he over-planned and structured his approaches, saying that he needed to work on "being organised to be able to be disorganised." Another mentor said, "I do have to structure myself a bit more and plan a bit more."

These self-reflections are supported by observational and interview data reported in earlier sections of the report.

The mentoring approach used in this differentiated programme modelled good practice in teaching gifted and talented students. Therefore, an important element of the NZMSC programme was to provide a 'learning laboratory' for contributing school teachers by giving them an opportunity to see the programme in action. Schools were invited to send one teacher to one session annually (with teacher release time paid by NZMSC). In 2006, there was minimal uptake of this opportunity, so beginning in 2007, schools were rostered to attend and this was an explicit expectation of school participation. Also, in 2006 there was little direction or focus, so that in future years, teachers were given the COF to guide their observations. Teachers were also given written information on the programme and an opportunity for discussion with the education director. The purpose in these visits was to enable teachers to 'see' a programme for gifted students, in hopes they would be able to emulate strategies in their own classroom and make connections between their school programmes and NZMSC programmes. It also gave teachers insight into their own students in a different context with a like-minded peer group.

The effectiveness of the teacher observation days is questionable. School attendance was initially variable; selection of teachers was sometimes haphazard (e.g., one school sent a relieving English teacher); participation levels in the actual day varied, with some teachers getting over-involved and others not staying the duration or actively engaging; and there was a lack of planned, purposeful focus (apart from the observation form). However, there was anecdotal evidence that some teachers did enjoy the opportunities presented and gained from those. For example, a rural school teacher (2006) gained funding from her School Board to attend all sessions of the programme, including the residential overnight stays. As a result, she prepared a professional development session for other teaching staff in her school and liaised with two other rural schools to offer a specialised, complementary, follow-up programme onsite at NZMSC. Other teachers commented on the value of seeing others work with gifted students, and particularly the utilisation of questioning techniques. It should be noted that this element of the programme was not part of the original NZMSC proposal, but was negotiated with the Ministry of Education as a means of ensuring outcomes for teachers and schools.

However, the annual professional development day was an integral part of the programme from its conceptualisation. The purposes in this day were to provide teachers with information about the programme (from the perspectives of mentors, students, other teachers, the education director, and the research team); to inform teachers of the characteristics of gifted students; to share appropriate, multiple methods of identification; and to provide information about complementary programmes, such as CREST, Olympiads, and other NZMSC programmes. The professional development days were well attended, with teachers from rural and urban, primary to secondary, schools. Guest speakers, including local school advisers and the TDI coordinator, were involved in each session. The feedback from teachers on the value of this day was positive, especially in regards to identifying students for the programme. Over the three year period, school-based identification methods became more appropriate, comprehensive, and selective (as evidenced in the student application forms and interviews with the education director).

The final piece of professional development, which was perhaps unplanned and unexpected, was in the opportunities for sharing the programme nationally and internationally. For example, the education director presented workshops as an invited guest at the International Organisation for Science and Technology Education (Perth, 2007) and the 4th APEC Gifted in Science forum (Korea, 2008). He was also a presenter at the 10th Asia-Pacific Conference on Giftedness (Singapore, 2008), which resulted in a publication for the Singapore Ministry of Education. In New Zealand, presentations have been made for the Otago Association for Gifted and Talented and at annual TDI hui. All of these opportunities have led to other consultancy work, sharing, and networking. The ripple effects of this sharing of the TDI have benefited the programme in many ways, as well as the professional competence and confidence of the educational director. As the final milestone report states, "… the main benefits have been endorsement that the principles and processes underlying our programmes are congruent with research, models, and recommendations from those expert in the field" (Milestone Report, December 2008, p. 3).

Summary: Programme Development and Implementation

The NZMSC programme began in 2006 as an 'idea' which by 2008 evolved as a differentiated educational programme for gifted and talented students. The key factor in the ongoing development of this programme was responsiveness to monitoring and evaluation (both internal and external) – feedback was sought from all stakeholders, considered, and acted upon. Another important element was the quality of the programme staff, all of whom had an interest in gifted and talented students and the development of appropriate programmes for those students. Their professional development and support, alongside their experiences in coordinating, organising, and mentoring, was critical to the programme's development and implementation. The major focus of the programme was to meet the unique social, emotional, and intellectual needs of gifted and talented students.

Outcomes for Students

The programme had four major outcomes for students. These were the development of:

  • Scientific knowledge, skills and attitudes;
  • Metacognitive, thinking and problem-solving skills;
  • Team work skills; and
  • Self-awareness and understanding.

The previous section described how the programme was developed and implemented in relation to these outcomes. This section will describe further evidence showing that these outcomes were met.

Scientific Knowledge, Skills, and Attitudes

The NZMSC programmes not only developed students' scientific knowledge and skills in the specific discipline of marine science, but also in the broader field of science and its interrelationships with other discipline areas. In the final online survey of the students, their teachers and parents, 47% indicated that the programme had a strong impact upon students' knowledge and skills, 42% indicated a positive impact and only 11% indicated some impact. This is supported by evidence collected and analysed in 2008 by the NZMSC programme showing nearly half (42%) of all written responses to an internal evaluation tool were knowledge statements regarding marine organisms (Milestone Report, December 2008). The PMI analyses over the three years also show that students positively commented about the organisms they were studying, and many also found this interesting. The PMI also indicates that for many students, the scientific knowledge was in-depth, new, and different.

The scientific knowledge gained was not isolated or specific to marine science, as a programme goal was to increase students' understandings of the relationship between perception and response, and this relationship to survival (for both marine organisms and people). Data collected by NZMSC in 2008 and from the evaluation team observations show that in their products (a press release and final presentations) students were able to explicitly connect their scientific investigations with broader issues of survival, as well as global environmental issues. The NZMSC internal evaluation provides some further support, but only 43% of parents and teachers surveyed agreed or strongly agreed that students had increased their understandings of the conceptual theme (Milestone Report, December 2008). The students also showed this increased awareness in their PMI responses and focus group interviews over the three year period. The mentors and education director, in interviews and mentor reflection forms, indicated that students were challenged to relate their scientific knowledge to the 'bigger picture'. As one mentor commented, "Yes, definitely increased their understanding! It resulted in thinking on a wider scale and where our own research fits in, as a part of the whole" (MRF, 2008).

The science of this programme also related to other disciplines, especially as students began applying the scientific processes through experimentation. Students were observed applying content and skills related to mathematics, literacy, computer science, social sciences, and the arts in both their experimentation and preparation of final presentations. Given the wide range of team projects and presentations, it is difficult to measure these specific subject-related outcomes, but is worthy of noting that the programme was interdisciplinary, especially in terms of skills.

Student attitudes toward science were also affected by the programme, primarily by exposure to a specific discipline (marine science) and its related processes. Given the identification processes used by schools, many of the students who entered the programme were identified based upon their school-based performance, so it can be assumed that they came to the programme with a positive attitude towards science. Therefore, it would be difficult to show shifts in their general scientific attitudes; however, what can be shown is that students were positive about the opportunity to undertake what they referred to as 'real' science. The PMIs and interviews over the three years show that the students consistently reported a new sense of appreciation for the scientific process, which they discovered through hands-on science in an authentic laboratory environment, guided by expert scientists.

For some students, their participation in the NZMSC programme clarified their thinking about science and its role in their future education and career plans. This attitude shift went two ways: some students were certain they wanted to pursue a career in marine science, whilst others were certain they did not want to pursue this pathway. However, all students in the end of programme online survey were pursuing science courses in their secondary schooling, and half of the students indicated the pursuit of science-related future study and careers.

Metacognitive, Thinking, and Problem-solving Skills

A major focus of the NZMSC programme was on the development of metacognitive, thinking, and problem-solving skills. In the final online survey of the students, their teachers and parents, 34% indicated that the programme had a strong impact upon students' metacognitive, thinking and problem-solving skills, 47% indicated a positive impact and 19% indicated little or no impact. As has been explained earlier in the report, this outcome became a major goal and focus of the programme in 2008 with concerted efforts in professional development and support for mentors and more explicit teaching and evaluation of these skills. Therefore, it is not surprising that in student responses to the online survey, the 2008 students reported more favourable shifts in this outcome. Further evidence is found in the student PMIs and interviews, whereby in 2008, students made more explicit comments regarding metacognition than in previous years.

In 2008, all students were administered a pre-post survey regarding their preferences for learning science and metacognitive awareness. The metacognitive awareness pre- and post-responses from the students show both gains and declines in their reported usage of metacognitive strategies, as Table 9 below shows. The table provides the frequencies for strategies students reported as the most least commonly used (as indicated by a negative response) by all students in the 2008 programme (rural and urban). As it shows, students' reported use of review, organisational structure of texts, strategy selection, and self-monitoring declined, whereas there was some greater use of focusing on their overall meaning. Although students did report their use of other strategies, such as goal setting and time management, the least commonly reported strategies in pre-survey responses were a major focus of mentors' questioning and facilitation of teams. What this table shows is that students did not show gains in most of the targeted metacognitive strategies: for example, before the programme, 39% were unable to select an effective strategy, after the programme 50% reported this inability.

Table 9: Pre-Survey Strategies and the Percentage of Students Not Using Them Before and After the Programme
Metacognitive Strategy Pre-
Survey
Post-
Survey
I focus on overall meaning rather than specifics. 41% 36%
I know when each strategy will be most effective. 39% 50%
I find myself analysing the usefulness of strategies while I study. 37% 22%
I periodically review to help me understand important relationships. 31% 67%
I use the organisational structure of the text to help me learn. 31% 38%
I ask myself if there was an easier way to do things after I finish a task. 31% 38%


This decline in reported metacognitive strategy use could be the result of several factors. For example, the pre-survey was part of the application pack and students may have thought that a positive answer to questions would place them in a more favourable light. Because metacognitive strategy teaching was an integral part of the 2008 programme, it could be the case that students were actually more aware of the strategies and potential use of those at the end of the programme, and, thus, better able to make an informed judgement of their usage. It could be the case that students were actually showing greater discrimination of their reported strategy usage by the end of the programme.

However, in their end of programme survey, the students indicated that "some aspect of thinking, problem-solving and thinking about their thinking is the most commonly stated 'take-away' benefit of the programme" (Milestone Report, December 2008). Further confirmation of gains in metacognition was evidenced in the student PMIs and interviews, with students indicating their enjoyment of these strategies. Observational and interview data also show students engaged in metacognitive thinking.

The 2008 end of programme survey, conducted by the NZMSC as part of their internal evaluation, confirms the students' growing awareness of metacognitive strategy use. When asked what knowledge, skills and attitudes they had gained and would transfer to school or everyday life, the majority of students described scientific and thinking processes. For example, one student explained that she would use "better, innovative strategies to solve problems" in her everyday life. Another student commented that "Thinking about thinking is always helpful, even outside school" and another found that "It has really made me find new ways to get around problems that occur."

Team Work Skills

The students in the programme worked in teams of like-minded peers to conduct their scientific investigations, and their ability to work as a team member was a major goal of the programme. In the final online survey of the students, their teachers and parents, 34% indicated that the programme had a strong impact upon students' team work skills, 54% indicated a positive impact and 12% indicated little impact. Communication skills were similarly shown to have been developed in the programme, with 48% of parents, teachers and students indicating a positive impact and 23% a strong impact. Observations of the programme in action, interviews with mentors and students, and student PMIs support these perceptions. Students developed skills of cooperation and collaboration, as well as leadership (including shared leadership).

The pre-post survey of leadership styles shows a change in their perceptions of some group roles at the end of the programme. For example, pre-survey results showed that in the local 2008 year group, 41% always took the role of 'chairman' and 50% sometimes did so. Following the programme, only 9% reported this as their usual role, 86% indicated they sometimes took this role, and 23% said they rarely served as the chairperson. Similarly, 52% of students initially indicated they always took on a practical role of getting the job done and 43% sometimes did so; however, at the end of the programme, only 33% always took this role and 67% sometimes did.

Students were also asked to indicate their preference for working – independently, with a partner, or in a small or large group. Again, a shift in perception occurred, with students indicating a stronger preference of working in small groups and scientific teams (Milestone Report, December 2008). For example, 38% of the urban 2008 students indicated in their pre-programme survey that they preferred working in small groups, whereas post-programme 60% indicated this. Most students (95%) in the pre-survey felt the optimum team size for scientific investigations was three to five; at the end of the programme this had decreased slightly to 83%. Interestingly, some students preferred working with a partner or independently after the NZMSC experience. These shifts could be the result of working in a like-minded peer group.

Self-awareness and Understanding

Another important goal of the NZMSC programme was the development of self-awareness and understanding of giftedness. In the final online survey of teachers and parents, 58% reported a strong impact upon students' self-awareness and understanding, and 35% indicated a positive impact. This is confirmed in the NZMSC internal evaluation conducted in 2008, with 64% of teachers and parents agreeing or strongly agreeing the programme had increased their understanding of self, true peers, and communities (Final Milestone Report, December 2008). Furthermore, the end of programme student survey showed that 30% of both rural and urban students would take away knowledge and skills related to team work, leadership, and organisation to apply to their learning ; but 60% of rural students would apply these to their personal lives (Final Milestone Report, December 2008).

This goal of increasing self-awareness and understanding was accomplished by the bringing together of like-minded peers, which overwhelmingly was perceived by all stakeholders as one of the most beneficial aspects of the programme. Nearly half the teacher respondents (42%) ranked working in teams of like-minded peers as the most beneficial aspect of the NZMSC programme; this was especially the case with students from rural schools. When asked if they would recommend the programme to other gifted and talented students, and if so why, all of the student respondents indicated that they would because of the opportunity to meet and work with other gifted and talented students. As one student commented in the online survey (2008), "Meeting others about as bright as I am is cool". Interestingly, about half of the students (52%) had maintained relationships with other participants in the programme.

Observations, focus group discussions and interviews confirmed that the opportunity to work with like-minded peers was a strength of the NZMSC programme, and one which facilitated growth in self-awareness and understanding. This programme extended beyond simply working together academically, in that it included a residential component which created opportunities for social engagement. The gifted and talented students were working, living, and playing together – and this was a unique aspect of the programme. The result of this mix of opportunities, as reported by all stakeholders over the three years, was a boost in confidence, communication, and social skills, as well as, recognition of one's roles in a team situation in relation to his or her abilities.

Being aware of one's abilities better enables clarification of further study and career directions, and this was another outcome for student participants. In the 2008 teacher and parent surveys conducted by the NZMSC, 79% thought students had discovered more possibilities for future careers, with none disagreeing (Final Milestone Report, December 2008). The online end of programme survey confirmed this, with 79% of parents and 92% of teachers indicating it had a positive or strong impact upon future decision-making. As one parent commented, the programme "… focused him in a confusing plethora of choices in the modern world" (Online Survey, 2008). As the previous section on science knowledge, skills, and attitudes shows, students also saw the programme as an aid in clarifying their future pursuits.

Summary: Outcomes for Students

The NZMSC programme was designed to enhance students' science knowledge, skills, and attitudes; metacognitive, thinking, and problem-solving skills; team work skills; and self-awareness and understanding. All stakeholders indicated that these outcomes were achieved. The programme design and implementation was a major factor in ensuring positive outcomes for students, and over its 3-year development and evolution, growth was shown for some students in some outcomes. Another key determinant of these shifts was the ongoing evaluation and responsiveness to evaluation results, with the aim of pursuing intellectual, social, and emotional development for gifted and talented students.

Impact of Programme

The previous sections have provided evidence of the effectiveness of the NZMSC programme. The impact of the programme for all stakeholders is summarised in this section, highlighting the major themes which have arisen from the evaluation. The programme impacted upon its student participants, their schools and teachers, and the NZMSC and its staff – to varying degrees for different stakeholders.

Impact for Students

  • Development of knowledge, skills and attitudes through acting, being, thinking, and feeling like a scientist in a research team.
  • Creation of real solutions to real problems, based upon in-depth, specialised, hands-on learning.
  • Development of relationships with like-minded peers and mentors.
  • Greater understanding and awareness of self, as a gifted and talented individual within a community of learners.
  • Enhancement of metacognitive, thinking, and problem-solving skills applicable in school-based learning, as well as life in general.
  • Clarification of future study and career options.

Impact for Schools and Teachers

  • Enhanced identification processes for schools and teachers which are appropriate, selective, and comprehensive.
  • Increased awareness of the needs of gifted and talented students and how to meet these.
  • Increased human, physical, and financial support for gifted and talented students (by way of the NZMSC programme as part of a continuum of approaches for meeting their needs).
  • Increased opportunities for professional development.

Impact for NZMSC

  • Growth in expertise in the development, implementation, and evaluation of differentiated programmes for gifted and talented students, recognised by external educational providers and organisations.
  • Development of a qualitatively differentiated model programme for gifted and talented students, potentially transferable or deliverable to other groups, in other curricular areas, and in other tertiary settings.
  • Increased effectiveness in evaluating a gifted and talented programme and being responsive to evaluative outcomes.

Impact for NZMSC Mentors and Staff

  • Enhanced ability to communicate scientific knowledge and understandings.
  • Greater awareness and understanding of the needs of gifted and talented students, and how to effectively facilitate those through mentoring.
  • Increased skills of questioning, problem-solving, thinking, and metacognitive strategies, applicable in their mentoring roles, but also in other professional and personal situations.
  • Increased confidence and competence in the development, implementation, evaluation, and dissemination of gifted and talented programmes.

Sustainability

Sustainability can be viewed from several angles: financial sustainability; programme sustainability; and the sustainability of outcomes for stakeholders. This section of the report addresses each of these aspects, beginning with the ongoing financial viability of the programme. As the section on resourcing shows, the programme was primarily funded by the Ministry of Education as a three-year Talent Development Initiative, with the bulk of the funding for professional fees for the directors and mentors. The programme directors were well aware of the implications of reliance upon a short-term funding source and successfully sought further financial assistance from the Todd Foundation, support from Otago University, and the utilisation of some LEOTC funding for gifted and talented programmes in general (not for the Year 10 programme specifically). An extension of funding for the education director's salary was also granted by the Ministry for 2009. However, these funds will not enable the sustainability of the programme without financial input from students, their families and schools, or in its current structure.

The decrease in funding means that the programme, as it was conceptualised and developed between 2006 and 2008, is no longer sustainable. For 2009, the number of overnight stays will be limited and there will no longer be release-paid teacher observation days. However, as the section on programme development and implementation shows, the NZMSC programme for Year 10 students has been documented, evaluated, and refined, with clear and transparent systems for all aspects of its delivery. This means that despite any resourcing issues, the differentiated programme is sustainable as a model for gifted and talented students. The programme directors have developed relationships with other institutions, both within and outside of New Zealand, who have shown an interest in the programme's delivery at the NZMSC and off-site, but whether these come to fruition as viable options to supplement programme costs is yet to be seen.

The final issue of sustainability is in regards to outcomes for students. Given the NZMSC is an out-of-school programme, there is potential for the Year 10 experience to be a one-off, fragmented opportunity. As the previous sections have shown, the outcomes for students are positive, but are they sustainable? From an NZMSC perspective, the Year 10 programme has been extended to a user-pay Year 11 programme and plans are underway for a Year 12 one-on-one mentoring programme with retired or senior academic staff in the marine sciences. Prior to the Year 10 programme students may have opportunities in Years 4 to 8 for specialised programmes, and any school may participate in complementary programmes. In that regard, the NZMSC has developed a multi-year, multi-faceted programme accessible to some students. This continuity and further development of differentiated programmes for gifted and talented students goes some way towards sustainability of outcomes for students.

Yet, for the majority of students who have participated in the Year 10 NZMSC programme, the primary source of sustainable outcomes would be within their regular school programmes. A lack of opportunity to further enhance and develop the outcomes of the NZMSC programme was expressed by many of the students and their teachers over the three-year evaluation. From the students' perspectives, their schools showed little interest in the programme, their experiences onsite or ongoing opportunities – despite the NZMSC's efforts to provide ongoing information and support to schools. Most students explained that there was little preparation or support from their schools. The follow-up from schools was mainly an opportunity to write something for the school newsletter, something one student noted as an "ironic" outcome for a scientific, hands-on experience (interview, 2008). Students also described their frustration in not being well-informed by their schools of other similar or complementary programmes, with some students having to self-advocate in order to attend the Year 11 programme or take part in other opportunities.

The teachers valued the resources, like-minded peer group, intensive hands-on study, and expertise of the mentors, recognising these as unattainable in their schools because of constraints of funding, time, student numbers, resources, locality, and so on. While the uptake on professional development (for observation as well as sessions on identification and the programme) increased overtime, the outcomes for school-based practice are indeterminable. In theory, the professional development and support for schools should have strengthened the sustainability of outcomes for students, and in some cases did so, but overall the NZMSC was not an integrated element of school-based science or gifted and talented provisions. The onus for ensuring the scientific, metacognitive, and personal outcomes for students are further developed and maintained will ultimately fall upon the students themselves.

In summary, the NZMSC programme grapples with financial sustainability despite having developed a systematic, transparent model of good practice for gifted and talented students. The outcomes for students are multi-faceted, addressing academic, social and emotional needs; however, the long-lasting effects may be undermined by lack of school-based opportunities or uptake for complementary programmes. The final milestone report of the NZMSC programme best summarises its need for substantial funding for sustainability:

it has become increasingly evident that there is a huge difficulty in continuing a successful and much appreciated programme without a substantial long term guarantee of central funding … If an initiative has been found to work for the benefit of students, schools and teachers then, at some point, there needs to be considered transition from trialling and development to a committed strategy over a wider base and much longer term (December 2008, p. 16).

Summary: New Zealand Marine Studies Centre Secondary Programmes

The summary of the results of this case study is provided in relation to each of the research questions.

How were decisions around the programme design arrived at? Who was involved in the decision-making process? How has this process impacted upon the sustainability of the programme?

Four factors influenced the programme design: a team approach; previous experiences in designing and delivering educational programmes; responsiveness to stakeholders; and grounding in theory and research. The NZMSC team was co-led by an Educational Director and Programme Director, who worked together to ensure an educationally sound and pragmatically viable and accountable programme. An Advisory Board supported and monitored the decision-making. The Education Director was the key catalyst, designing and facilitating all aspects of the programme 'in action' and working closely with the mentors who delivered it. The key elements from this process which have ensured sustainability are: transparency; documentation; shared roles; and ongoing dissemination of information and results of the programme.

What changes in climate and philosophy have been required for the successful implementation of this programme? How were these changes managed, and how were changes in practice achieved?

Gifted and talented education was not initially a major focus of the NZMSC, but the success of this programme and student enthusiasm for more programmes has led to this being a major focus of the Centre's work. Their existing philosophy was one of facilitating advanced scientific content using hands-on approaches in an authentic learning environment: for gifted and talented students little, if any, change in this philosophy was necessary. The mentors' passion, youth and inexperience as 'teachers' suited the goals of the programme, but professional development and support was provided in response to their needs and programme evaluations.

How appropriate were the identification procedures, curriculum adaptations and forms of assessment in relation to the goals of the programme? How has this contributed to student outcomes?

The programme adhered to the principles of gifted education as outlined by the Ministry of Education (2002) in its acknowledgement of school-based, multi-categorical definitions, multi-method approaches to identification, and differentiated learning experiences. The residential component complemented the academic programme, with both providing opportunities for like-minded peers to learn and socialise together. The programme was able to meet all its intended outcomes for students, and this improved over time in response to evaluations inclusive of all stakeholders and using multiple methods, and especially in measuring metacognitive, thinking, problem-solving, and team work skills. Improvements were also shown in the quality of identification tools and methods used by participating schools, and this was the result of professional development and improved communication. These improvements enabled differentiated learning matched to gifted and talented students' strengths and abilities.

What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme? To what extent has this specific programme design contributed to improved student outcomes?

The design evolved over time, becoming more closely aligned to the intended student outcomes, and in responsiveness to the evaluations. Curriculum differentiation to meet outcomes included:

  1. Scientific knowledge, skills and attitudes were taught using marine science as a vehicle for teaching a conceptual theme (which enabled an interdisciplinary approach) in an authentic scientific environment.
  2. Metacognitive, thinking, and problem solving were explicitly taught in the programme by pre-assessing students and targeting specific strategies based on the pre-assessment results. Questioning became a major focus of professional development and support, enabling improvements in the thinking and problem solving triggered by good questions.
  3. Team work skills were developed by the grouping and re-grouping of like-minded peers based on individual strengths. Team work roles and skills were also explicitly taught.
  4. Self-awareness and understanding was enhanced through the opportunities to work with like-minded peers, and this was perceived by all stakeholders as the greatest benefit of the programme. The academic and residential components complimented one another and enabled this outcome to be strengthened.
How comprehensive are provider initiated student and programme monitoring and evaluations? How do the findings of the monitoring or evaluation inform the programme?

The evaluation changed over time in response to the findings of both the programme-based and external evaluations. An extensive range of formative and summative, formal and informal, quantitative and qualitative programme evaluation tools were employed. This multi-faceted approach, aimed at measuring all outcomes, was also inclusive of all stakeholders. Two key barriers were met and overcome: the administration, collation, and analysis of evaluation data was addressed by employing administrative support and allowing more time for the Educational Director to devote to evaluation; and the measurement of all outcomes (especially process skills and social-emotional) through self-review, reflection, and expert input (by the evaluation team). The evaluation was refined with a shift to fewer tools, which explored broad themes (rather than specific details), and the adoption of new measurements.

What is the evidence for improved student learning and social, emotional or cultural outcomes as a result of participation in the programme?

The NZMSC programme was designed to enhance students' science knowledge, skills, and attitudes; metacognitive, thinking, and problem solving skills; team work skills; and self-awareness and understanding. All stakeholders positively indicated that these outcomes were achieved. The programme design and implementation was a major factor in ensuring positive outcomes for students, and over its 3-year development and evolution, growth was shown. Another key determinant of these shifts was the ongoing evaluation and responsiveness to evaluation results, with the aim of pursuing intellectual, social, and emotional development for gifted and talented students.

How have resources and personnel impacted on the success or otherwise of the programme?

The resources to support the NZMSC programme allowed for the development and implementation of a specialised scientific programme. The specialised physical and human resources resulted in a responsive, authentic learning environment in which a differentiated programme could be facilitated by expert mentors. The financial resources mainly supported the provision of these expert personnel and facilities, a cost factor unique to out of school providers of gifted and talented programmes. However, unlike other out of school programmes, there were minimal costs to participating students, their families, or schools, allowing participation in a programme that met its outcomes for students.

What role has staff professional development played in achieving the programme goals?

The mentors in the programme were not trained teachers, and the philosophy behind the programme did not call for such. This created somewhat of a tension when it came to providing professional development and support. The professional development and support that was offered ranged from one on one (provided by the Education Director) to whole staff focus groups. Over time these whole group sessions focused on the results of the evaluations in order to enhance the programme. The Education Director became more and more involved in his own professional development and there was professional development for contributing schools. The effectiveness of the observation days is questionable; the evidence shows the annual professional days proved more effective as this was responsive and focused.

How well has the programme planning occurred in regard to sustainability of the programme after the three-year period of funding ceases?

The staff of NZMSC was well aware of sustainability issues from the outset of the programme: they are reliant on external funding contracts. Financial sustainability of the programme is problematic despite having developed a systematic, transparent model of good practice. Student outcomes are multi-faceted, but their long-lasting effects may be undermined by lack of school-based opportunities, including complementary provisions. The programmes are still in operation, but to sustain them students are now charged fees for their attendance. Some elements of the programme have been funded for 2009 by the Ministry of Education and with a grant from the Todd Foundation.

As external providers, this programme impacted school-based identification; teacher awareness; increased physical, human and financial resources for gifted and talented learners; and increased professional development. For NZMSC the impacts have been in relation to increased expertise in developing, implementing, and evaluating gifted and talented provisions; the development of a differentiated, and potentially transferable, programme; and increased effectiveness in internal evaluations.

Rutherford College: 'N' List, 'N' Quire, 'N' Rich, 'N' Light

Background

Rutherford College is a decile 5 co-educational secondary school (Years 9 – 13) situated on the Te Atatu Peninsula in West Auckland and serves a multi-cultural community. The school was founded in 1961 and has a current roll of approximately 1200 students and 84 staff. The school has a predominantly New Zealand European roll but with 19 percent Māori and with significant groups of Pacific, Asian and Indian students.

At the time the College applied for Talent Development Initiative (TDI) funding there was a dedicated programme in place for Year 9 and 10 gifted and talented students. The Kaleidoscope Programme (referred to as KAL) offers a qualitatively differentiated programme for two groups of very able students over these two years. The school's 2004 Education Review Office (ERO) report describes this programme as student-centred, integrated, intellectually challenging, and with a focus on developing the skills of critical thinking. The schools' digital prospectus (2007) states that the KAL programme, 'aims to identify and meet individual needs by offering a programme that challenges students and develops the higher level thinking skills that are necessary for future learning'. The programme involves a small teaching team offering 'thematic holistic units of learning'. The KAL classes work on a modified timetable that is aimed at maximising learning opportunities.

This is how the story began, but as this report will show, over the three years of the contract the Rutherford programme evolved in responsiveness to students, and to better align itself with its initial aims.

Research Methodology

This section describes the research team, cycles of research, and data collection methods used in this case study.

All the participant groups in this study were fully informed of the ideas behind and practices employed in participatory action research. Negotiating parameters around the action research process did pose some challenges. Understandably, the TDI director was eager to maximise the opportunities to draw on the expertise of the research team. A central tenet of the action research methodology is collaboration between those being researched and those undertaking the research. It is based on notions for feedback and feedforward. What is sometimes more blurred when undertaking an action research investigation is determining the boundaries to researcher input. The TDI director designed and commenced implementation of the programme with the active involvement of the school's principal and deputy principal. In the first year both these staff members left the school. The TDI director was relatively professionally isolated at this point. Teachers with expertise in gifted and talented education represent a very limited resource. Two of the three researchers had expertise in gifted and talented education, and a third, significant secondary school teaching and administrative experience. For the TDI director, the researchers had knowledge and experience that she potentially had ongoing access to and that could certainly not be replicated within the school.

Action research requires the researcher and researched to be actively involved in all stages of the research process. As a consequence, the action researcher is close to those being studied and as many writers point out, the roles of the researcher and the researched are often blurred. Waterman, Tillen, Dickson and Koning (2001, p.3) state that, 'Action researchers view this intimacy as a means of promoting appropriate change and understanding of practice.' In this context it is important to point out that action research involves both research and practice outcomes.

Typically, on a visit to the school the TDI director would share with the researchers her plans for the programme a term or two in advance. She would then actively invite responses from the researchers to these plans. This is an example of the potential tension between an advisory role and a researcher role. The issue here is arguably less about what is being sought (in this case by the TDI director) and more about the response that is provided. On such occasions, the input from researchers tended to be linked by them to the research investigation. Rather than offering solutions to problems, the response was usually to pursue an issue with a question and to explore ways the research could be used to test an idea or inform a decision.

After each data collection phase, the TDI director and researchers would meet and reflect upon programme developments and where relevant, consider options for change. This involvement became quite substantive over the three years of the research. The researchers certainly did become part of the TDI programme, rather than simply observers of it. Ultimately, however, final programme decisions were the role and responsibility of the TDI director.

Research Team

The research team involved in evaluating the Rutherford College TDI comprised of two academics with extensive research, teaching, and professional development experience in gifted and talented education but no experience as teachers in the secondary school sector. The third researcher lacked the same level of experience in gifted and talented education but brought to the team a depth of understanding of and experience in secondary school education. The researchers worked closely with the programme director, who was also considered part of the research team for this action research.

Research Cycles

For both the research and researched the first year of the programme was very much an exploratory period. The researchers were focused very much on seeking to answer the question, What is going on? In hindsight, although not officially designated as such, this year is best viewed as a pilot year. During this period the research was guided by the following key questions:

  1. What are the specific objectives of the TDI programme?
  2. What philosophical and pedagogical ideas is it based on?
  3. How is giftedness and talent conceptualised?
  4. How is eligibility for involvement determined?
  5. How are students identified for inclusion?
  6. How are individual programmes devised, implemented, and evaluated?
  7. How do the students' programmes interface with the regular school programme?
  8. What is the nature of teacher input and involvement?
  9. What is the nature of parental input and involvement?
  10. How is the overall effectiveness of the programme measured?

Over the course of this year the research team undertook two two-day visits, two one-day visits, and an afternoon/ evening visit. The first visit to the school in early 2006 was introductory in nature, where the researchers met with the TDI director, senior staff, and the group of six TDI students. This initial contact was used to outline the nature and purposes of the research project and in particular to assure staff and students that this was not an 'audit' of the TDI, but a collaborative undertaking with an objective of contributing positively to the programme. Consistent with the research design, time was spent with the TDI director developing a framework for the research study.

The subsequent visits usually began with the TDI director briefing the research team on developments since the previous visit, as well as sharing current issues, challenges and dilemmas. At each visit, relevant documentation was made available to the researchers, including identification data, students' personalised learning plans, progress and achievement information, and student and parent evaluations. Following this, the researchers met and spoke with the students and were able to observe specific projects they were working on or had completed. In 2006, even though the TDI comprised of only six students, their class commitments were such that it was almost impossible to meet with them as a single group. On a number of occasions the researchers outnumbered the sub-group of participants being interviewed.

At the second visit to the school, two members of the research team met with a focus group of 12 teachers who taught students involved in the TDI. The purpose of this meeting was to brief these teachers on the research project and to address any questions or issues that they raised, but also to ascertain their understanding of the nature and aims of TDI project and their attitudes towards and expectations of it. In term three, a follow up interview was conducted with a small sub-group of these teachers. The father of two of the TDI students (twin boys) was also a teacher at the school and as it was felt that his perspectives were quite unique, he was interviewed individually.

In Term 4 2006, the parents of the TDI students attended an end-of-year celebratory dinner where students had the opportunity to present their work and to share their personal reflections on the programme. The research team attended this meeting and outlined their role to parents. They were then able to informally speak with each student's parents. Later in the evening, the parents of the 2006 group were joined by the parents of the students selected for the 2007 intake. This was a much larger group of parents as the cohort of students for 2007 numbered 23, compared with six in the first year. At the meeting the researchers spoke to 2007 parent group about the research project, answered their questions, and were able to learn of their aspirations for the programme, as well as listen to their queries and concerns.

Throughout 2006, the research team had multiple opportunities for involvement into most aspects of the programme. This occasionally extended beyond formally-scheduled visits and included attending events such as parent information evenings, student performances and presentations of student work.

Although the overall aims of the TDI remained the same, the nature and size of the participant group changed dramatically in 2007. This required a change of approach to the research, focusing the research on answering the question, Is it working? In 2006, with three researchers and six student participants, it was possible to obtain in-depth data in a relatively informal and unstructured way. These sessions were semi-structured and very much 'conversational' in nature. The researchers would determine a set of lead-questions to cover the dimensions of the project that were considered to address the broad question, Is it working? The questions would be shared with the TDI director who was given the opportunity to have input. On occasions she suggested additions, amendments or identified areas for investigation that the researchers had not considered. As experienced interviewers, the researchers felt comfortable working from lead questions, which allowed those being interviewed some control of the direction of the interview and an opportunity to raise issues of importance to them.

In 2007, the research team spent a total of 15 days at the school. In contrast to the previous year, the researchers did not visit as a team on all occasions, as the increased size of the group meant that engaging with all of the students at a single point in time was virtually impossible. Each researcher made his or her own arrangements to complete their focus group and case study interviews.

As in the previous year, the first visit involved a meeting between the TDI director and the research team. At this session the TDI director informed the researchers how selection of the group was undertaken and the identification data for each individual student was made available. The TDI director then profiled each individual student and explained the rationale for their selection and their interests, abilities and learning needs, and outlined their planned programme. These briefing sessions, which characterised the introductory aspect of most visits, became an invaluable source of information for the researchers but also provided an opportunity for the TDI director to share and test ideas and to receive input from 'outside' experts.

On this first visit the research team was introduced to most of the new TDI group. Four were part of the previous year's group but this was a first time meeting for the majority. An explanation was given to the students of the aims of the research project and what was being requested of them. As in all introductory sessions, the students were explicitly told that the intention of the TDI evaluation was not to ascertain whether or not the school was meeting the terms of its contract agreement with the Ministry of Education. Rather, the purpose was to investigate the extent to which the overall aims and purposes of this national initiative were being met. It was not concerned with 'assessing' or 'auditing' individual programmes.

At this first meeting it was decided that the student group should be divided into three focus groups. The composition of the groups was determined by the TDI director and every effort was made to ensure that they were parallel in profile and that each included a representative mix of gender, year levels, length of involvement, achieving and underachieving gifted students, and culture. Within each of the three groups two 'case study' students were identified. The case study students were selected to provide representation across the same categories used to determine the composition of the focus groups.The approach was for each of the three researchers to undertake a group interview with a focus group of students, analyse the responses, and to identify areas and issues that could be pursued in more depth with case study students. Each researcher conducted individual interviews with their two case study students from their focus group. These case study interviews were transcribed and returned to the students for any amendments, and then to endorse as an accurate record of their responses. The case study interviews provided a context for exploring in-depth the individual programmes of a sample of six students. Although the researchers had access to this information in documentation form, the interviews gave each student the opportunity to speak about their programme and comment on it.

As was the practice over the three years of this study, the specific areas to be investigated with any group were negotiated with the TDI director and the findings shared with her. All participants were informed that a summary of the findings from the data gathered would be shared in this way. These summaries did not identify individual students, however, it would not have been difficult for the TDI director to link a specific student or students to a particular comment, especially from the case study summaries. Students were assured that they would not be named in these summaries but were informed that there was always the possibility that the nature of some feedback could see them connected to it. They were invited to signal any interview material that they wanted treated with particular sensitivity and/or to have input into how it was shared. This was never asked for.

In term three, after each researcher had analysed his or her data, the researchers met, compared findings and identified themes across groups and individuals. Immediately afterwards the researchers shared the findings with the TDI director.

In term four, each researcher completed an individual interview with the parent/s or caregiver/s of each case study student. During that visit, the TDI director hosted a dinner for the parents and caregivers of the Māori and Pasifika TDI students. The director had established this group for the expressed purpose of offering dedicated support and to provide a forum for issues specific to them. After the meal, this group were invited to share their experiences of, and perspectives on, the TDI programme, and to answer questions from the research team.

The data collected by the TDI director contributed significantly to the researchers' overall evaluation of the programme. She systematically obtained ongoing feedback from the students and their parents/caregivers, and input from teachers was regularly sought. This information was made readily available to the researchers, as were milestone reports to the Ministry of Education. As researchers, it was important that our approaches took cognisance of her schedule of data collection with regard to the total time students, parents, teachers, and others were being asked to give to the evaluation of this programme.

In October 2007, four students from the TDI group presented their perspectives on education generally, and on talent development specifically, to a national hui of gifted and talented education advisers. This hui was attended by one of the research team. In December 2007, a researcher attended the TDI end-of-year function where students shared their individual achievements and experiences over the year and where departing students were farewelled. This evening was attended by students, their parents/caregivers and some teachers, and provided an informal opportunity to gain further insights into the programme.

With each visit, the researchers were able to view work undertaken by students. Most visits to the school included non-scheduled contact time where the researchers could engage informally with the students in their home room. Students would often talk about their work or share projects and products, sometimes prompted by the TDI director, but at others times of their own volition. It was hoped that being in the home room without any formal research agenda would lessen researcher effect and help the students to feel relaxed in the researchers' presence. It also enabled the researchers to observe the students at work and interact with them in a more natural and non-intrusive manner. In November 2007, the research team hosted a luncheon for all the students and three teachers, including the TDI director, to acknowledge their participation in the research. This afforded another informal opportunity to interact with the students and to contribute further to the creation of a climate of mutual trust. Most importantly, these informal sessions gave the researchers unique opportunities to gather first hand data relating to interactions and relationships (student to student as well as director and students), mentoring, school activities, and individual TDI projects.

While there was significant difference in the composition of the TDI group from 2006 to 2007, the differences between the 2007 and 2008 intake were nowhere near as pronounced. The main change was an increase in the number of gifted and talented Māori and Pasifika students. These students were considered to be significantly underachieving and it was believed that they would benefit from a personalised learning programme and from being part of this community of learners. The research team spent a total of seven days at the school in 2008, seeking to determine responses to the questions: Is it working? How do we know?

While it was very difficult to employ similar research approaches from 2006 to 2007, it was much more viable to do this from 2007 to 2008. This was considered important, as it gave the ability to identify change over time and across cohorts that were relatively homogeneous. In 2008 there was a small group that had been part of the programme since its inception, but a much larger group that were in their second year in the programme.

Thus, the procedures used for data collection in 2008 bore a very similar resemblance to those employed in 2007 and the focus remained relatively constant. However, following the 2007 interviews, it was decided not to follow up the group interviews with individual case study interviews in 2008. In 2007 it was found that the case study interviews yielded very little new information from what was obtained in the group interviews with the focus students. There are two possible reasons for this. First, these students are generally very confident and capable of expressing their ideas verbally. Second, it was evident that there was a very high level of trust within the group and a culture of open and honest disclosure had been encouraged. It was evident that the students felt very safe sharing their experiences and perspectives in front of their peers. Sometimes what was shared in the group interviews was of a very personal nature but there was often a sense from observing the reactions of the group that this was not new to them and had been talked about elsewhere.

In 2008, the group interviews were undertaken by the researchers during independently-organised visits. In 2006 and 2007 the researchers visited the school as a group and undertook all the interviews during that time. However, it became increasingly difficult to organise this across the researchers' schedules and the students' timetables. Almost by default rather than by design the lead researcher spent an extended session with the group during his final scheduled focus group visit. Due to timetabling and other commitments, the students tended to arrive intermittently, meaning that over the day both small group and individual interviews took place. The individual interviews were not all with his focus group students. This was interesting and indicative of the relationship that had developed between the researchers and the students. The students who were not part of his focus group simply took the opportunity to engage with the researcher to talk about their activities and what was happening in their lives. Included in these small group interviews were two students who had been involved in the TDI for each of the three years. One student was part of his focus group but the other was not.

This represents the reality of researching in settings such as schools. Even with the best laid plans, circumstances such as timetable changes, unscheduled school events and illness impinge on the research process. In many instances it is not practical and sometimes not even possible to reschedule the session. For the researcher, changes forced upon the process often raise issues around consistency of approach and the validity and reliability of the data obtained. In the situation outlined above, the data were considered but only tended to confirm information gathered by the other researchers in their sessions.

The combination of data collected from parents by the researchers in the previous two years and by the TDI director across all three years was extensive, and it was not considered necessary for the researchers to involve this group again in 2008. The TDI director's processes for obtaining parental feedback were regular and comprehensive and the researchers had had input into the design of these. In previous years, the researchers had used scheduled TDI parent meetings to interview this group. There was no scheduled meeting in 2008 at a time that suited the researchers, and given the existing feedback processes it was decided not to schedule a special parents' meeting.

The researchers continued to meet with the TDI director and examine her records. As in previous years, the data gathered by the researchers was used to contribute to the ongoing planning and delivery of the TDI. The practice of engaging informally with students in their home room continued and the researchers were regularly updated by the TDI director on the students' activities at each visit, and occasionally between visits by email communication.

During the final visit the researchers met with the school's principal, who offered some reflective comments on the programme and explained the school's plans to continue the initiative beyond the three year TDI funding period (the school has received a small amount of funding support to support its continuance in 2009). This session was short due to the pressure on the principal's time in the last month of the school year. However, lead researcher interviewed the principal more formally early in 2009.

In December 2008, in an attempt to capture a holistic picture over the entire period of the project, each researcher independently prepared his or her own reflective comments of their impressions over the period of the evaluation. The data from these three reports informed this final report.

Data Collection Methods

The collection of data focused on the following sources of information:

  1. The TDI director : This initiative was very much the responsibility of one individual. When the proposal for the TDI was submitted it enjoyed the full support of the school's principal and deputy principal. The deputy principal was very closely involved in the design and delivery of the programme during the first year but took leave in 2007 and took up a new position in 2008. The principal involved in proposal development retired in the first part of 2006.
  2. Document Analysis : The research team was provided with copies of all documentation related to the TDI and the analysis of these was critical to the data collection process.
  3. Participant Students : From 2006 – 2008 more than 50 students were involved as members of the TDI. The researchers met with these students on numerous occasions, as a single year cohort, in smaller focus groups, and individually.
  4. Student Projects and Presentations : Over the course of the three years, the students completed a diverse range of projects and presentations, such as film making, musical performances, presentations to teachers, research investigations, and the creation of web sites.
  5. Parents of Participant Students : The researchers met with the parents of the TDI students each year, either as a single group or in smaller focus groups or individually.
  6. Teachers and Principals : Although no teachers had direct significant involvement in the design or management of the TDI, the majority of these students' day-to-day learning took place in 'regular' classes alongside 'mainstream' students. In many cases these were classes that they would have attended whether or not they were part of the TDI. However, numerous students were accelerated and attending classes a year ahead of their same-age peers. All their teachers were considered an important source of information. In addition, the researchers met regularly with the principal as part of the research process. Usually this was an informal courtesy visit that was sometimes used to update him on developments, offer him an opportunity to ask questions of the researchers but also for the researchers to seek specific information. Each year, one formal meeting was organised with the principal for these purposes.

This research primarily used qualitative approaches to data collection. In the first year (2006) with only six students involved in the TDI, the students were interviewed as a single focus group. The researchers met with the students' parent/s at the end of that year. These were single, informal interviews with each student's parent or parents. Teachers were invited to a focus group interview the same year and 12 teachers attended that session. In the second and third years (2007 & 2008), when the TDI group more than trebled in size, each of the researchers was assigned a focus group of one third of the class. Within each focus group, four students were selected for individual case studies. In 2007, the researchers interviewed their case study students individually and subsequently undertook individual interviews with each student's parent/s or caregiver/s.In 2007, the researchers met with parents/caregivers of the Māori and Pasifika students and offered them the opportunity to share their opinions and experiences of the programme.

Employing a wide range of both qualitative and quantitative data collection methods, inclusive of the perspectives of the many key stakeholders in the programme, provided rich insights – and a tremendous amount of 'raw data'. The next challenge was to make sense of it all through careful analysis and recording of this TDI's story. The researchers undertook a thematic approach to analysing the data and used these to compare and contrast the voices between and within the various stakeholder groups. While this was a task primarily undertaken by the researchers, the TDI director was actively involved in informing this process and particularly in assisting the researchers to interpret the information collected. The data was used formatively, that is, it was used to inform programme design and practices. This cyclic approach to data collection and analysis also informed the research process. For example, as themes emerged in one context they could be further investigated in that context or across contexts.

Table 10: Guiding Questions and Themes
Framework: Guiding Questions
How are programmes developed and implemented for gifted and talented students?
Themes:
  • Programme design
  • Definition of giftedness and talent
  • Identification methods
  • Differentiated programmes
  • Evaluation
  • Resources
  • Professional support and development
What are the outcomes for students in relation to the programme goals?
Themes:
    Meeting needs
  • Creating learning environments
  • Planning future directions
What is the impact of the programme for all stakeholders?
Themes:
  • Evidence of growth, development, and change
Is the programme sustainable?
Themes:
  • Future plans and directions
  • Resourcing


The table above summarises the pre-ordinate themes for this case study. The results are presented in relation to programme development and implementation; outcome for students; and the impact of the programme for key stakeholders.

Programme Development and Implementation

According to the College's TDI proposal to the Ministry of Education, a major impetus for building on the KAL Programme in Years 11- 13 came from the programme's 'graduates'. The application states that the proposed 'N' List, 'N' Quire, 'N' Rich, 'N' Light (4 'N') programme would vary from the KAL programme in that it would have a primary goal of identifying and providing for the needs of underachieving gifted and talented students. The stated intention was to include underachieving gifted and talented students from all the school's major cultural groups. The proposal also highlighted other features of the intended programme including:

  • active parental involvement
  • an emphasis in students' social and emotional needs
  • using ICT
  • the provision of ongoing staff professional development

The School's proposal to the Ministry of Education reflects a sound knowledge of the important principles and practices that underpin effective programme provisions for the gifted and talented. In addition, the proposers were conversant with national and international research in this area and effectively drew on this to support their programme design.

The major component of the funding sought (and eventually received) was to provide a programme leader's salary for the three-year period of the contract. An additional one-third (approximately) was to be used to meet the costs of professional development facilitators, teacher relief, conference attendance, administration, specialised equipment, and students attending courses.

The TDI contract between Rutherford College and the Ministry of Education identified the following outcomes of the programme:

  1. Beneficial outcomes in students' motivation and self esteem, and the meeting of their cognitive, socio-emotional, creative, and cultural needs.
  2. Students will be empowered through creation of their own learning environment.
  3. A formal process will be developed to identify gifted and talented students, and in particular underachieving gifted and talented students.
  4. The development of a gifted and talented education professional learning community through targeted professional development to ensure programme sustainability.
  5. An increase in staff confidence and skill in ICT, curriculum compacting, and differentiation in planning, delivering and assessing learning opportunities for gifted and talented students.
  6. Sharing best models of process, practice, and evaluation with other educators.
  7. To provide opportunities for gifted and talented students to have input into planning the future direction of the school.

For the purposes of this evaluation, the major focus was on outcomes for students. The next section explains how these were achieved through the design of the programme.

In essence, the Rutherford College TDI was about personalising learning, although no reference is made to this term in the school's proposal; nor is there reference to the Ministry of Education's initiative that carries this name and was launched the year the Rutherford proposal was developed. However, the Ministry of Education 2006 publication, Let's Talk About It: Personalising Learning, identifies a set of core components of a personalised programme of learning, all of which can be seen within this TDI. These are:

  • Children and young people will know how to take control of their own learning.
  • Parents and family/whānau will be partners in their children's learning.
  • Teachers will have high expectations of every student, know how they learn and adjust their teaching to meet students' learning needs.

The original intention, which has been a constant across the three years, was to identify a group of students with particular abilities and strengths that would benefit from the opportunity to work within a personalised programme. This programme was aimed at enhancing students' abilities, providing them with opportunities to pursue special interests not catered for within the 'mainstream' programme of the school, and to identify and respond to areas of specific need.

In this sense, the Rutherford College TDI is not a single programme but a collection of individual and personalised programmes, developed in consultation with the TDI director, the students, and their parents. Yet in another sense this has, over the period of three years, become an entity in itself and has evolved from a collection of six individuals in early 2006 to a very close-knit and supportive community of learners in 2007 and 2008. There was a deliberate decision made to keep the numbers small in the first year to ensure that systems and processes were carefully trialled and developed. Each year the composition of the TDI group has been significantly different from the previous year's group. By the third year the programme more closely reflected the original goal of including a significant number of underachieving gifted and talented Maori and Pasifika students. Consultation with the Maori and Pasifika communities in 2006 and 2007 had culminated in the refinement of the tools to identify gifted and talented students from both of these groups. This saw more gifted and talented Maori students being selected, and a small group of gifted and talented Pasifika students being added in 2008. Most of these students were considered to be significantly underachieving. This clearly makes for a difference in group character and direction, as the backgrounds, interests, and abilities of the individual members change. The programme has also evolved philosophically, pedagogically, and socially over the three years. These changes have been accompanied by shifts in the approaches taken to evaluating the effectiveness of the programme. The evolving nature of this TDI and concomitant changes in the associated research have been significant.

According to the TDI director, the programme aimed 'to provide opportunities for the students to grow academically, socially, emotionally, physically, creatively and spiritually.' The formulation of each individual programme was a collaborative and negotiated activity. The basis for programme development was each individual's perceived 'strengths and weaknesses'. After a student had been identified, the parents/caregivers were asked to reflect on their child's strengths and weaknesses. Where two parents/caregivers were involved, they were asked to do this independently. The parents, student, and TDI director then met and each parent and the student were given time to share their thoughts related to these two categories. From this, a programme that encompassed all the student's activities both at school and some beyond the school gates was devised. The facilitation and implementation of each programme was the responsibility of the TDI director.

One of the strengths of this programme was the knowledge and experience key staff had in gifted and talented education. The Kaleidoscope Programme (KAL), which offers differentiated education for gifted Year 9 and 10 students, had existed at the school for some years, and the TDI was proposed as a natural extension to this programme. The TDI director had established the KAL programme, as well as being involved with other relevant initiatives within the school, and had a clear understanding of the principles of effective practice in gifted and talented education. The TDI concept itself reflected a knowledge of recent research and contemporary thinking in this area, and specifically that:

  • Giftedness and talent is a dynamic and culturally-constructed concept and means different things to different groups and at different points in time (McAlpine, 2004).
  • Recent approaches to the concept are much broader than simply intellectual and academic abilities and extend to creative, cultural and even spiritual domains (Riley, Bevan-Brown, Bicknell, Carroll-Lind & Kearney, 2004).
  • Gifted and talented Māori and Pasifika students are significantly underrepresented amongst those identified in our schools as gifted and talented (ERO, 2008).
  • Approaches to identifying gifted and talented students need to reflect the more liberal approaches to defining the concept and to culturally-specific interpretations of it (Ministry of Education, 2000).
  • The underachievement of gifted and talented students is disproportionately high and the reasons found for this include a lack of respect for the individual, an unstimulating and unrewarding curriculum, inappropriate teaching methods, and programme inflexibility and rigidity (Whitmore, 1980).
  • Gifted and talented underachievers prefer a 'family' rather than a 'factory' learning environment and may need guidance or counselling to achieve academic success (Delisle & Galbraith, 2002).
  • Approaches to learning that encourage curiosity, independence, autonomy, and creativity have been shown to be best suited to gifted and talented students (Davis & Rimm, 2004).
  • Parents/caregivers should be offered multiple opportunities to be involved in their children's education (Ministry of Education, 2000).
  • Gifted and talented students benefit from spending time with like-minded peers (Robinson, Shore & Enersen, 2007).
  • Gifted and talented individuals often possess unique social and emotional characteristics that need to understood and acknowledged in programming to meet their needs (Clark, 2008).

The level of parental involvement in this programme was extremely high and a key to its success. Many of the students had been in the KAL programme previously, so the parents would have appreciated the efficacy of grouping gifted and talented students together on a fulltime basis. These parents were likely to have welcomed a continuation of a similar approach in Years 11 – 13. The TDI, however, was a very different concept to KAL. The TDI had a much more holistic focus and sought to broaden the range of students' experiences. In some cases it was suggested very able students reduce their subject load to free up time to pursue activities of interest, activities that would not count towards a qualification. Some parents very readily accepted such departures from the traditional school programme; others were much more hesitant.

I have always maintained that successful learning is much more than simply jumping through hoops. These children in particular will follow passions not prescriptions. When I first heard of the philosophy of TDI I knew it was for my daughter. As a parent, having the opportunity to be directly involved was a new experience. (Parent, interview, 2007)<

We were not so keen about him dropping a subject to do something we saw as more of a hobby or fun activity. She [the TDI director] was persuasive and we agreed and we can see some of the benefits now. (Parent, interview, 2006)

At the end of each year, the parents of that year's TDI group met with the parents of the incoming group and at this meeting the new parents were briefed on the programme, heard of the experiences of the TDI parents, and could ask questions of the TDI director and the parents. Parents played an active role in the design of their children's personalised programme. Each programme was a negotiated plan between the parents, the students, and the director. Throughout each year the TDI parents would meet formally as a group with the TDI director where issues could be aired and suggestions made. They were also invited to any presentations, performances, and celebrations that involved the TDI students. Parents' views and ideas were canvassed by the director through a regular cycle of surveys, interviews, and meetings. In 2007 and 2008, a sub-group of Māori and Pasifika parents was established. This was well received by these parents, not only because it provided them with a forum to have more culturally specific input into the programme, but it also facilitated parent-to-parent support.

The benefits of this level of parental involvement are many, not the least of which is advocacy. The parents, as a group, became very strong supporters of the programme and would readily act as advocates for it. In conversations with parents, they exhibited a high degree of conversancy with the programme's core principles and the specific practices and were very familiar with the interests and activities of students across the entire group. In many ways, the TDI became a community of families.

All the data collected from parents showed that they held the TDI director in high regard and they acknowledged her role in the programme's success. The following statement by a Māori parent is indicative of the views expressed by nearly all the parents:

We have talked amongst ourselves and we agree that we have a superb facilitator in her [the TDI director]. Parents need to have confidence in the person because this [programme] pretty much hinges on her. She is the mesh that binds this programme. She offers a protective mechanism, a safety net. She has passion, vision and commitment. She will go the extra mile. She has maximised for our children what this environment offers. It's the quality of the leadership that has made the difference. (Parent, interview, 2008)

This paints a very positive picture of the response of the parents to the programme. This should not be taken to infer that parents did not question, challenge, or criticise elements of the TDI. They did, and the various feedback mechanisms put in place by the TDI director encouraged this. However, the commitment of parents to the TDI concept and their support of the TDI director were unquestionable.

An important point of difference from many other dedicated programmes for gifted and talented students was that selection to this group was not simply on the basis of high achievement. Invitation to be part of this programme was extended to those for whom it was considered maximum benefit would be gained from being in the programme. This concept saw some very high achieving students remaining in the 'mainstream' and some students who had previously shown potential, but limited evidence of high achievement, selected. The increasing trend over time to include more gifted and talented students who were underachieving was a major factor in increasing the heterogeneity of the TDI student group.

The personalised approach, which saw every student with a programme tailored to his or her interests, strengths, and needs, meant that a much greater diversity across these dimensions could be accommodated. The reason many schools cluster gifted and talented students together, whether it be on a part time or full time basis, is usually to maximise the opportunities that a much greater degree of homogeneity offers. As a consequence, many gifted and talented classes provide for a limited range of ability areas. It is generally perceived that a class that includes students with gifts and talents as diverse as science, art, social leadership, specific cultural abilities, and more, becomes so heterogeneous that the benefits of placing the students together are compromised. This is much less of an issue where students are not required to fit a programme, but the programme is required to fit the student. To this end the researchers were not dealing with 'a programme' as such. In fact, for much of the first year, the only time the TDI class came together was when the research team undertook a focus group interview with them as a group.

As the TDI director explained in a conference presentation in 2007:

Secondary schools are traditionally, and by their very nature curriculum driven and some would say increasingly assessment driven … The student enters the system and they fit into it. We suggest that the student comes into our school and we create a programme of learning and a learning environment around their needs … If you start with the needs of the individual student, as opposed to the cohort, you will build a successful programme because you have prioritised needs and looked beyond the traditional systems and curriculum. You advocate on behalf of the student for flexibility and change.

Defining Gifted and Talented

In a general sense, definitions of giftedness can be viewed as conservative or liberal. Conservative definitions tend to be based on only one criterion, such as 'intelligence'. As a result, such definitions accept only a small proportion of children as being gifted (e.g., one to three percent of the population). Liberal definitions tend to be based on broad ranging criteria and usually adopt a more inclusive approach that sees a higher proportion (e.g., 10%–15%) of children as gifted. Worldwide there has been a shift from the narrow, IQ-based notions of giftedness, to more liberal and inclusive models, commonly referred to as the multicategory concept (McAlpine, 2004).

The approach taken to defining giftedness and talent in this school has been multicategorical, focusing on academic, social, emotional, physical, creative, cultural, and spiritual abilities. Such an approach is usually consistent with a view that giftedness and talent is a culturally constructed concept and interpreted differently by different cultural groups. Of particular relevance to this TDI were conceptualisations of giftedness and talent from Māori and Pasifika perspectives. Ongoing consultation with representatives from these communities sought to refine points of difference in how giftedness and talent is viewed, and to incorporate these into the definitions employed to indentify gifted and talented students across cultural groups.

Identifying Gifted and Talented Students

Typically, more liberal approaches to defining the concept demand more multidimensional approaches to identifying abilities or qualities indicative of exceptional ability. More conservative approaches tend to rely on quantitative data derived in the main from standardised test results. Multidimensional approaches will usually include such information but rely more heavily on qualitative data, derived from a wide range of sources. The primary, but not exclusive source of this information is teachers and developing their skill and knowledge base to understand the behaviours associated with exceptional ability and qualities across a diverse set of domains usually demands a comprehensive programme of professional learning. An additional challenge that faced this programme was in familiarising teachers with the characteristics of underachieving gifted and talented students.

In the opinion of the researchers, much of the success of this programme was associated with the attention that was paid to identification. Considerable time was spent before the programme was implemented to developing sets of characteristics that would facilitate 'accurate' identification. The TDI director had read widely and consulted with outside experts to develop these sets of characteristics. This was accompanied by some school-wide professional development focused on the identification of gifted and talented students. The schedules of characteristics were further developed to include gifted and talented Māori students and underachieving gifted students in the second year and further extended to include gifted and talented Pasifika students in the third year. These developments initially involved Maori and Pasifika staff from the school but then extended into their communities. This consultation began in the first year of the programme but became a stronger feature over time.

This programme demonstrated the benefit of funding TDIs with a demonstrated record of effectiveness in gifted and talented education. The approaches to identification were very sound and clearly borne out of the experience with the KAL programme but also from the professional learning staff had undertaken. The approaches to identification were research-informed, contemporary, and relevant. Not only did the procedures reflect a conversancy with relevant theory and research, there was wide consultation, especially in developing sets of qualities and abilities, and appropriate approaches, to the identification of gifted and talented Māori and Pasifika students. Nominations for inclusion in the TDI were invited from teachers, parents, peers, and the students themselves. Individual interviews occurred before the final selections were made.

Teachers were provided with the Purdue Academic Rating Scales (PARS) to guide the selection of students. This scale was developed to assist teachers identify subject-specific ability, rather than having to rely on generic checklists or rating scales (Feldhusen, Hoover, & Sayler, 1989). Subsequently, the generic PARS was used for initial teacher recommendation, but was complemented by other forms of identification. For example, parents of short-listed students completed the Purdue parent checklist. Additionally, checklists of the specific abilities and qualities valued by Māori and Pasifika peoples, then latterly those from Asian backgrounds, were developed. These lists were dynamic in nature and continued to be refined as consultation with representatives from the different groups was extended. Teachers in the school received ongoing professional development focused on the identification of gifted and talented students.

Over the three years, the programme attrition rate has been very low, pointing, at least in part, to effective selection processes. According to both the principal and the TDI director, the skill level of teachers in identifying gifted and talented students has improved significantly over this time. They both consider that sustained professional learning accounts for this improvement.

Differentiating Educational Opportunities

Across the three years that this TDI was evaluated, few students' programmes were the same. This is not surprising, given that the criteria used to design each programme were an individual's strengths and weaknesses. The strengths included areas of interest and achievement, values, and expectations. The weaknesses included areas of concern and anxiety, values, and expectations. These were considered across academic, social, emotional, physical, creative, cultural, and spiritual domains. All personalised plans included a combination of 'regular' and 'special' activities, all included enrichment, and many involved acceleration. In this context, 'regular' refers to subject classes that were part of the mainstream school programme, where there was no requirement for teachers to make special provisions for TDI students. However, while many students may have attended these classes had they not been part of the TDI programme, that is not necessarily the case. Their personalised programmes did not involve an 'add-on' but were developed as an integrated whole. Many students were accelerated in subject areas, typically attending classes a year or more in advance of their age peers. Some took a subject with their same-age peers, but also took that subject at the next year level. The 'special' aspects of a programme were dynamic and diverse, enabling particular strengths and interests to be acknowledged and celebrated within the school environment. This was important for students whose particular areas of talent were not encompassed by the traditional school curriculum.

For example, one Māori student expressed an interest in researching why so many of his Māori peers achieved so poorly, while he and some other Māori students achieved very highly. This research became one of his TDI projects and culminated in a professional development session with staff, and presentations outside the school. Another student, with an interest in video making, but with acknowledged lack of interpersonal confidence and skills, undertook to produce a video with a group of primary school students. Two students received mentoring in music for a year from a member of Auckland University's Department of Music. Their work culminated in the production of a music CD. Another student established his own successful web design business.

The holistic nature of this programme was consistently reported by students and parents as one of its most positive features. Many programmes for students at this level attend to abilities in a quite discrete and disconnected way. This is almost unavoidable at the high school level, where programmes are organised and delivered in such a manner. While the provision of differentiated learning opportunities in individual subject classrooms should be an expectation of teachers at any level, the ability to offer a holistic or integrated programme across subjects, abilities and interests is much more difficult in secondary schools. This personalised approach to learning was very successful in providing for a very wide range of abilities, interests and needs:

All through year 9 and 10 it was just academic homework, academic stuff. Go to class, go home. Do homework. And I didn't have anything else … now I'm doing sport, doing cultural stuff, doing a radio show, doing environmental community stuff and doing more than just academics. It's just shaping me more and allowing me to do more than just academic. So now I have direction. I'd just throw myself at work without really having a purpose other than to get good grades. TDI gave me a lot more options. (Student, interview, 2008)

[The TDI is] basically trying to develop the person holistically and so everyone's first thought is academics … we went to the National Drama school and that was a prime example of trying to develop us holistically. (Student, interview, 2006)

I have seen such a change in my daughter. I had not realised how depressed she had become and how obsessed she was with grades and achieving. It's like part of her person has been unveiled through exposure to another world. Her eyes have been opened, not just through the new experiences she has had, but through the things the others are involved in. (Parent, interview, 2007)

He struggles with school routines therefore the customised programme is ideal for him. Because there is money to support it he has access to the technology he needs to develop his abilities. (Parent, interview, 2007)

In 2007, the Education Review Office (ERO) evaluated the provisions for gifted and talented students in 315 schools and selected seven schools as examples of good practice. Rutherford College was one of this group of schools and their 'Good Practice' report (ERO, 2008) succinctly captures the TDI programme, describing it as:

A holistic programme that offered an integrated curriculum and pastoral care … [where] well-being was paramount and pastoral care was of a high quality … Students could think critically about their gifts and talents, and communicate their needs. They could identify and reflect upon their strengths and weaknesses, and acknowledge both their personal improvement and the positive outcomes for them participating in the programme. (ERO, 2008, p. 32)

Over the last 20 years there has been a marked increase in understanding of and attention to the unique social and emotional needs of gifted and talented young people. As the Ministry of Education (2000) points out, the education of gifted and talented students has historically been driven by concerns about their learning. The consensus of recent advice on programme design for these students is that it must attend to their affective and social, as well as their intellectual and physical wellbeing. The holistic nature of this programme embodies this principle and in designing their programmes these aspects were specifically attended to. As one Year 12 student put it:

I feel I can't hide in this programme. I felt I was exposing aspects of my personality that I didn't want to acknowledge myself, let alone talk about with others. I have to admit I am somewhat socially retarded. Well, maybe that should be 'was', rather than 'am'. Being in this group has somehow given me some skills, and also some desire to be part of things. (Student, interview, 2008)

Although some specific elements of the programme shifted over time, the research data demonstrated that rather than 'drifting' from the original intent, the programme became much more closely aligned to it. Obviously the dramatic increase in the size of the TDI group (from six students in 2006 to more than 20 in 2007 and 32 in 2008) was a factor here. Other changes were in response to the increased diversity of the TDI group in years two and three of the contract, including the inclusion from 2007 of gifted and talented underachievers, and gifted and talented Māori and Pasifika students. Larger group size and increased heterogeneity was reflected in the range of activities that made up the students' personalised programmes but also in the dynamics of the group.

Evaluating Gifted and Talented Programmes

The TDI programme was subject to a regular cycle of formal evaluation that was consistent across the three years. In addition to the data obtained from school and class assessment and appraisal, each individual's progress was formally evaluated by the TDI director on two occasions each year. The primary purpose of this evaluation was to determine the effectiveness of each student's individual programme. This information was used to inform the nature of their programme for the following six months. Twice-yearly the students were asked to evaluate the director's performance.

The researchers viewed and discussed with the director the results from these evaluations and this provided a useful source of data to compare our findings with. A feature of these evaluations was the openness, honesty and frankness of students in sharing their experiences and perceptions. It was clear from the very early stages the relationships that had developed between the director and the students, and between the students themselves, were characterised by a high level of trust. In the view of the researchers, a significant portion of this could be attributed to the personality and the approach of the TDI director. She cultivated a group ethos where she related to the group more as a senior peer than as their teacher. Yet the researchers saw no evidence of students taking advantage of this and accounted for this by the levels of mutual respect that characterised the relationships between the students and the director.

Another feature of the students' evaluations was the level of perception and insight reflected in their feedback. This is not surprising amongst a group of gifted adolescents but unfortunately such rich information is often not tapped into. This disposition, together with the invitation to give it expression, and the knowledge that it can result in actual changes, was a most effective tool in developing and fine-tuning an individual's learning programme.

At the end of each of the three years the parent group were given the opportunity to evaluate the programme and to make recommendations for change. The researchers attended these sessions at the end of 2006 and 2007. These evenings were also characterised by open and frank discussion and while the feedback was overwhelmingly positive, there was no sense that parents felt constrained from offering criticism or suggestions for improvement.

The TDI director saw the role of the research team as important to providing an 'outsider' view of the programme. On most occasions when the researchers visited she would have identified a number of issues or proposed initiatives she sought input on. Our research model meant the researchers were regularly providing evaluative feedback to the director from the data collected. Some of this feedback was to contribute quite significantly to reshaping of the programme. For example, in the first year the researchers felt that an opportunity was being missed to develop a TDI 'community'. As noted previously, in this first year the TDI group was a collection of individuals working on personalised programmes and although they did meet and share from time to time, there was little in the way of a group culture. One of the few occasions the group came together was to meet with the researchers. The students themselves identified the value of having opportunities to interact as a group and quickly saw that they had some common interests, experiences and perspectives. The researchers saw some obvious potential benefits in developing a 'learning community, particularly in providing enhanced social and emotional support. This was shared with the TDI director, as were some of the benefits that could be gained from students coming together as a group on a regular basis. This is a very powerful example of participatory action research. This change in group culture and dynamics was, in the views expressed by almost all students during 2007 and 2008, the most significant benefit of being part of the TDI.

As well as undertaking formal evaluations of the programme the nature of the director's role meant that the opportunities for continuous informal appraisal was much greater than would be available to most high school teachers. The funding provided by the Ministry of Education to support this TDI meant that the director's role was virtually fulltime. There was a high level of engagement with the programme participants both individually and collectively. This factor, together with the skilful facilitation of relationships built on trust, meant that she had an intimate and holistic knowledge of her students. The sustained daily contact she had with the group created a very natural and effective feedback/feedforward loop.

Resourcing

The majority of the budget allocated to this TDI was used to support a director's position. This decision to make this a fulltime role was undoubtedly central to the success of the programme. When teachers, students, and parents were asked to identify the strengths of the TDI, 'the TDI director' was consistently their response. There is little doubt that her knowledge, enthusiasm, and commitment underpinned the effectiveness of the TDI. One Māori parent expressed it this way, "I have complete faith in her. I know that she has the interests of our children at heart. She has the skills needed to work with these students." The ERO (2008) report noted the significance of her work in gifted and talented education in the school. The principal stated, "She enjoys my absolute confidence and her knowledge and passion are such that I can just let her get on with it."

While the TDI director had some support from other staff in developing the programme, the day-to-day organisation and implementation were very much solely her responsibility. She had an intimate knowledge of her students and their families. She worked tirelessly on behalf of her students and especially in advocating for them. As researchers, all observations and interview data left us in no doubt that she possessed some rather unique traits that accounted for much of the success the programme enjoyed. That is not to imply that the programme's success relied disproportionately on a dynamic facilitator for its success but in this context it is almost impossible to separate the programme from the person. The TDI director designed and implemented the programme in manner consistent, not only with her views of how gifted and talented students learn and develop, but also in keeping with her values, beliefs, and style.

There is, however, a body of research identifying the characteristics of effective teachers of the gifted and talented. The findings from this research, which draws mainly on opinions of gifted and talented students, are relatively consistent across numerous studies. Vialle and Siobhan (2008) offer a set of traits that they believe characterise such teachers. These are consistent with how the Rutherford College TDI director fulfilled her role. According to Vialle and Siobhan (2008) effective teachers of the gifted and talented:

  • have insights into the cognitive, social, and emotional needs of gifted students;
  • have skills in differentiating the curriculum for gifted students;
  • employ strategies that encourage higher level thinking;
  • encourage students to be independent learners;
  • provide student-centered learning opportunities;
  • act as a facilitator or "guide on the side";
  • create a non threatening learning environment;
  • are well organised;
  • possess in-depth knowledge of subject matter;
  • have broad interests, often literary and cultural;
  • have above-average intelligence;
  • are lifelong learners;
  • think creatively;
  • possess excellent communication skills;
  • are willing to make mistakes;
  • possess a sense of humour; and
  • are enthusiastic.

While there were clear advantages to the degree of influence the TDI director had in overseeing the TDI programme, the researchers did have some reservations about this that were shared with her on several occasions. This was a demanding role. It involved developing and overseeing a large number of personalised programmes. The holistic nature of the approach meant that there was sometimes intense involvement in the students' social and emotional lives. There was also ongoing commitment to consulting with and informing the students' parents/caregivers. In addition, the TDI director took overall responsibility for the school's gifted and talented programmes. In our opinion, the involvement of an advisory group or a critical friend or a colleague in support would have helped to reduce the pressure that came with this position.

In addition to funding the director's salary, a significant portion of the budget was allocated to supporting students' personalised programmes. The resourcing requirements here were diverse and included such things as the purchase of computer hardware and software, audio-visual equipments, digital and print resources, payment to mentors, travel to seminars and competitions, etc. These represented resources or experiences that were outside what could be provided for by the school.

Professional Support and Development

At the point were this TDI commenced the director enjoyed the close support of a small team of colleagues. This group included the principal, who was less directly involved, and the deputy principal, who was very closely involved. As noted earlier, both these staff members left within a year and although she was still supported by a small group of like-minded colleagues, there was no formal mechanism in place to offer the support that such a comprehensive, complex and demanding role arguably required. For this reason the research team probably assumed a more influential role than might be expected to usually occur. While the researchers could not say that this minimal 'in house' support had a negative impact for the students and while the director was very effective even within this environment, it was clearly a less than ideal situation.

However, the director sought opportunities for professional development and support outside the school and was most effective in doing this. She attended a number of national and international conferences, seminars and workshops over the three years and usually presented findings from the TDI at these. These activities gave the TDI significant exposure and generated widespread interest. She was effective in developing an external professional support network over this time, which added to her knowledge and informed her work. In 2005, when the proposal was developed, and in the first part of 2006, the TDI initiative appeared to be a more overt part of the whole school culture. With the appointment of a new principal and deputy principal during 2006, the TDI continued to be supported but arguably less as a school-wide initiative and more as one of a number of provisions for specific groups of learners at the school.

Summary: Programme Development and Implementation

As this section has shown, the Rutherford programme changed and evolved over the three years of its implementation. What remained steadfast were the goals of: developing personalised programmes for gifted and talented students based on their interests, strengths and needs; identifying and providing for the needs of underachieving gifted and talented students from all the school's major cultural groups; addressing students' social and emotional needs and; actively involving parents in all aspects of the programme.

A key player in the success of this programme was the TDI director – not only in her role, but in the skills, wisdom, and personal characteristics she brought to it. The focus of this Ignite programme was clearly on enhancing the education of gifted and talented students.

Outcomes for Students

The Rutherford College programme included three outcomes specifically targeting gifted and talented students:

  1. Beneficial outcomes in students' motivation and self-esteem, and the meeting of their cognitive, socio-emotional, creative, and cultural needs.
  2. Students will be empowered through creation of their own learning environment.
  3. To provide opportunities for gifted and talented students to have input into planning the future direction of the school.

As the previous section explained, this TDI was highly personalised and individualised. Taking a holistic approach to each gifted and talented student's education necessitated meeting their socio-emotional, cognitive, creative, and cultural needs. For students in this programme, a major outcome was the resulting differentiated approach to their education based upon their individual strengths and weaknesses.

Over the three years, a major focus became to identify and provide for underachieving gifted and talented students. Underachievement can be the result of a myriad of complex factors, not the least of which is one's individuality. Rutherford students who were underachieving were identified for their potential abilities despite the fact their levels of achievement did not match these. Therefore, it is not surprising that these students, their parents, and teachers reported increases in their motivation and self-esteem, which, in turn, positively impacted upon their achievement. The increase in motivation and self-esteem was not limited to those students who were underachieving, but was extended to the majority of students in the TDI who reported the greatest benefit as being part of a like-minded peer group that validated and accepted their strengths, talents, abilities, and qualities.

The students shaped their learning environment through direct involvement in decision-making about their individual programmes, and through involvement in the TDI programme planning. This was strongly evidenced in 2005 when the researchers met with the TDI director and other school personnel who indicated that the Rutherford programme was being designed for students by students. Students' plans were developed in consultation with them, their parents/caregivers, the TDI director, teachers, and outside experts or mentors.

In 2007, the TDI group included students from years 11 – 13. Four students had been part of the programme in the previous year, but the other 20 were new to the programme. In this year the group was given 'form class' status and the TDI director their form teacher. Importantly, the group was allocated a dedicated space, or form room. This room, although relatively small, did have a larger adjoining room where the entire group could meet. The group met as a form class for 20 minutes each day. The creation of a vertical form class saw the allocation of group responsibilities as part of the TDI programme. Each year level had designated responsibilities. For example, the Year 12 students mentored Year 11 students and the Year 13 students were expected to and supported in taking an active leadership role in the school. A key part of the 'bonding' of the group was the TDI camp held early in 2007 and again in 2008.

In interviews with students throughout 2007 and 2008, being part of the TDI community was nominated by almost all students as the aspect of the programme that held the most benefit for them. While the parents as a group were not quite as singled minded in assessing the relative merits of their children's involvement, this feature still rated very highly. This finding was not a surprise to the TDI director, as it was confirmed by her own data. When asked to elaborate why the 'community' aspect was so important one student explained it this way:

I'm different and I know I'm different. I was able to manage in the KAL programme but being put into the mainstream this year would have been like throwing me to the wolves. I do not belong there. Here I am accepted. I am appreciated. My quirkiness is seen as a strength and not as something to ridicule. (Student, interview, 2007)

In a similar vein a male student said, "Finally, my love for poetry is recognised and not mocked" (Student, interview, 2007). Another commented:

We inspire each other. In a way it is slightly competitive but in a very healthy way. We support each other and celebrate each other's successes. On the other hand, you look at what others have achieved and we raise our own expectations. And there's no shame in achieving; but in another way there's no shame in failing from time to time. It's not like your place here is under threat and you have to perform or you are out. I don't think any of us think like that. (Student, interview, 2008)

The most often mentioned positive aspects of belonging to the TDI community were:

  • validation of identity and acceptance of being 'different'
  • engagement with like minds
  • exposure to seeing what can be achieved
  • encouragement from peers
  • common values and expectations
  • personal, social and emotional support
  • exposure to role models
  • cooperative learning and mentoring

A number of students mentioned that the TDI resembled a family, not a school class.

The parents identified similar positive outcomes associated with being part of the group.

There is a sense of responsibility to the TDI. He has turned full circle. He did not want to be part of the TDI but is now committed to it. His attendance has improved. His achievement has improved. He is motivated by his peers. This sort of environment is better for Māori where it's OK to achieve, to stick your head up. (Parent, interview, 2007)

I think she has true friends for the first time in her high school life. These are people who appreciate her for who she is and somehow seem to identify with her. She is excited about school, and that's a very new phenomenon. (Parent, interview, 2007)

There is a strong body of opinion that gifted and talented students learn best when grouped together. The benefits of bringing like-minds together are seen to be both academic and social in nature. The arguments are persuasive and are often used as the basis to establish a fulltime gifted and talented class or classes in a school or to support part time initiatives, such as withdrawal programmes and ability grouping.

The benefits for this approach centre on group homogeneity and this is arguably why the criteria for selection to such classes are relatively narrow and group membership more exclusive than inclusive. This could not be said of the Rutherford TDI group. First, the group included students from three year levels. Second, the students selected brought to the group a very diverse range of interests, abilities and needs. This meant that their individual programmes were different and although there may have been some shared elements across small groups, the programmes were individually determined, designed and undertaken. Third, the students did not all have high achievement in common as for two years the group included underachieving gifted and talented students. Fourth, there was significant cultural diversity across the group. Fifth, the students came from different pathways to the TDI. Some came from KAL, others from 'the mainstream', a small number from a total immersion Māori class. Some had moved to the school after year 10 and not been part of the KAL class.

This is not to imply that there were no common threads that connected this group. They shared the affirmation associated with being selected to be part of a somewhat special or unique group. Along with this came opportunities, support, and access to resources that were clearly the envy of some of their peers that were not selected. Such factors set them apart and made them distinctive as a collective. This group also shared a level of ability or abilities, realised or unrealised, which set them apart. However, the students themselves did not seem to see that it was their abilities or factors associated with being gifted and talented that made the difference. The following statement reflects the expressed opinion of many of the students:

This way of learning should be how schools are organised for all students. It's about forming a close and supportive relationship with a teacher who will go into bat for you. A teacher who knows you, understands you and is willingly to go out on a limb for you. That's what we have. She makes you feel you matter. (Student, interview, 2006)

On the other hand, it is also important to note that much of each of their time at school was spent in regular classes alongside peers outside the TDI. Most of the group reported friendships from within the TDI and outside that group and said that any animosity or negative reaction was generally short-lived.

The fact remains that to these students being a member of a group of like-minded peers outweighed any other advantages. The group functioned in a way that clearly met the needs of these students yet it would be erroneous to suggest that simply grouping them together and offering opportunities to work cooperatively acted as the catalysts. In the opinion of all those who were surveyed for this research, the TDI director was considered the single most influential factor in cultivating this supportive learning and social environment. The positive aspects the students identified as being part of with being with like-minded peers, such the freedom to be different, the stimulation of being with individuals of like ability and common interests, shared values and expectations, did not happen by default but were facilitated by a skilled teacher. A salient question here is the extent to which these advantages could be replicated in heterogeneous classrooms. The TDI students were of the view that many features of this initiative could be incorporated into all classrooms and would not just benefit gifted and talented students but could impact positively on all students.

The students also had some opportunities to have input in the future direction of the school, namely in helping shape a learning environment of acceptance of diversity. This was achieved through showcasing and celebrating notable achievements of this group of students, but doing so with sensitivity, wisdom, and skill to lessen possible charges of elitism.

Across all three years, students and parents, in particular, talked of positive effects of increased motivation and challenge, high achieving role models, personalised programming, an interested and involved teacher, mentors, and peer support and mentoring.

The moment I joined the TDI my marks started improving. I don't know, all of a sudden I can see some purpose to what I am doing. I look at those around me and I think - I can do this. We help each other out and I know if I am struggling there is someone there to help me. (Student, interview, 2008)

I have definitely improved in my academic subjects. Some of the reason for that is that I am now being challenged. It is interesting that I am doing history at my year level and also above my year level. I am doing better in the class that is above my year level and that is because the style of teaching and the content is more at where I am at. (Student, interview, 2008)

The most impressive accounts of improvement were from the gifted and talented students who were underachieving, and from their parents.

His improved attitude to school is remarkable. I cannot believe his level of commitment to his studies - we are seeing homework, which we didn't see before. I see a spark and enthusiasm I haven't seen before. He's also keeping up with his sport and kapa haka but his studies now seem more important to him. (Parent, interview, 2007)

Her marks across all subjects have improved. I think it's partly the pressure of expectation of belonging to the TDI and partly not being seen to not keep up with her peers. Whatever it is we're happy. (Parent, interview, 2007)

Summary: Outcomes for Students

The Rutherford College programme was designed for individual students and this was a major factor in ensuring positive outcomes for them. Over its three-year development and evolution, growth was shown in the mechanisms of identification, consultation, and infrastructure to support individual gifted and talented students as members of a community of learners.

Impact of Programme

Impact for the School

It is very difficult to demonstrate with any degree of confidence a relationship between an initiative involving a collection of personalised programmes and the wider school culture. Rutherford College has for many years now had dedicated programmes for gifted and talented students, primarily the KAL programme. Thus, there is a general acceptance that gifted and talented students demand differentiated responses, commensurate with their abilities, interests, etc. The school staff had been involved in professional learning in this and related areas immediately prior to 2006, when the TDI programme commenced . However, the principal appointed that year preferred supporting enthusiastic staff in areas such as gifted and talented education, rather than 'imposing' professional learning on all teachers. In his opinion, supporting the commitment of a dedicated group of staff will 'grow' an area much more effectively than programmes of school-wide professional development.

There is some evidence that the attitude of teachers towards the TDI and its members has changed. In 2007, both in focus group and individual interviews, the students gave examples of what they perceived to be negative responses from teachers related to their involvement in the TDI. In fact, during this year, negative reactions from teachers were more often cited than negative responses from peers, although examples of the latter were also given. The teacher comments reported by the students included criticism of the concept, cynicism of some individuals' inclusion in the programme, sarcasm about expectations of them as 'TDI' students, and resistance to being asked to accommodate atypical arrangements or special programmes. These responses were not widespread and had been experienced by only a minority of students. In fairness, possibly some of the special arrangements or exceptions teachers were asked to accommodate may not have been considered by them to be in the students' own interest. When the same topic was revisited in 2008 the issue appeared to have disappeared, possibly because over time the TDI had become a more accepted part of the culture. Negative peer comments about students' membership of the TDI also seemed to have almost completely ceased, possibly for a similar reason. The inclusion of a greater number underachieving gifted and talented Maori and Pasifika students may also have shifted perceptions of the TDI from a more exclusive provision to one that was more about matching students to a programme that best met individual needs.

While it is difficult to draw a definitive causal relationship between the impact of this programme and improved school achievement, the evidence provided in the records of achievement kept by the director, coupled with the accounts given by students, parents and staff would certainly point to positive gains here. From 2007 to 2008 Rutherford College has shown significant improvement in NCEA Level 1, especially for Māori students, and in Scholarship results. However, the principal is very reluctant to single out the TDI as contributing to this success. When asked what he considered to be the most important school wide impact of the TDI, he pointed to a school culture that is now more accepting of success.

The achievements of some of the students in the TDI have been impressive. As a school we have celebrated these and shared in what they have done. This has contributed to a culture where it's OK to succeed, where the tall poppy is allowed to raise its head.

The very existence of the TDI undoubtedly conveyed messages about the school's attitude to high achievement, particularly in academic areas. This may have contributed to a shift in culture that made the demonstration and recognition of outstanding achievements more acceptable. The validation of high performance associated with this initiative may have provided the TDI students with more confidence and/or greater licence to publicly display their achievements.

The TDI director has been committed to celebrating and to some extent show-casing the notable achievements of the group. This was consistent with one of the goals of the programme, which was to give gifted and talented students increased involvement in shaping the future direction of the school. There is a delicate balance needed here that is not always easy to achieve. Drawing attention to the successes of a group of gifted and talented students can lead to claims of elitism and privilege, and attract resentment. This does not seemed to have happened to any great extent at this school and our conclusion is that promoting the activities of this group has been done with a high degree of wisdom, skill, and sensitivity.

The impact of the programme for all stakeholders is summarised in below, highlighting the major themes that have arisen from the evaluation. The programme impacted upon its student participants, their parents/caregivers and the school and teachers.

Impact for Students
  • Increased recognition and validation of gifts and talents.
  • Empowerment through participation in the development of a holistic educational programme based upon individual cognitive, social, emotional, creative, and cultural needs.
  • Increased engagement with like-minded peers.
  • Development of stronger relationships with peers, role models, mentors, and teachers.
  • Enhanced understanding of self as gifted and talented.
  • Increased motivation and challenge.
Impact for Parents/Caregivers
  • Stronger home-school/parent-teacher relationships.
  • Development of parent-to-parent support networks.
  • Increased skill in advocating for their gifted and talented children.
  • Involvement in parental support groups, including opportunities for involvement with a culturally-specific focus.
  • Greater awareness of their child's special abilities and qualities.
Impact for Teachers
  • Greater skill in identifying gifted and talented students, including those who are underachieving or come from culturally diverse backgrounds.
  • Development of school-based tools for identification of gifted students, including refinement of appropriate measures for Māori, Pasifika, and Asian students.
  • Greater acceptance of gifted and talented students.
  • Development of a school culture more accepting of success of all types, including academic success.

Sustainability

One of the criteria the Ministry of Education set to assess the success of TDI programmes nationally was the extent to which these could be sustained beyond the period of MOE financial support. What appear to be the most sustainable elements of this TDI programme are:

  1. Comprehensive sets of characteristics of gifted and talented students generally, and of gifted and talented Maori, Pasifika and Asian students, and underachieving gifted and talented students.
  2. Well developed multidimensional approaches to identification for gifted and talented students generally, and gifted and talented Maori, Pasifika and Asian students, and underachieving gifted and talented students.
  3. An holistic model for gifted and talented students that is personalised, built on interests, strengths and needs, and developed in consultation with parents/whanau, teachers and the students themselves.
  4. A gifted and talented education model specific to the secondary school sector.

In 2009 the TDI will operate in a similar way to how is has from 2006 - 2008, although the TDI director will be required to teach classes in addition to coordinating the TDI programme. The school has made a significant financial commitment to support the continuance of this programme. This, according to the principal, reflects the level of support of the staff, senior management and the board of trustees.

While the actual nature and scope of the programme will inevitably change over time, the underlying philosophy and principles that were at the heart of the success of this initiative are likely to be much more enduring. Many schools could use these to inform their approaches to gifted and talented education. Our advice to those who would use the Rutherford experience to inform their own programme design would be to focus much more on these elements, than on the programme itself.

Summary: 'N' List, 'N' Quire, 'N' Rich, 'N' Light

The summary of results of this case study is provided in relation to the research questions.

How were decisions around the programme design arrived at? Who was involved in the decision-making process? How has this process impacted upon the sustainability of the programme?

The programme was initially designed to offer years 11–13 gifted and talented students a differentiated programme. The KAL programme for gifted and talented year 9 and 10 students had been in place in the school for a number of years but there was no equivalent in the years after this. In consultation with the school's staff, the deputy principal and the TDI director designed a framework for a programme of personalised learning. This was a conceptual and systemic framework but each individual programme was designed in consultation with the student, his or her parents and teachers. It was based on strengths and needs, and targeted students who would most benefit from this approach. It was not designed for all gifted and talented students but those who would gain most from this provision. The ongoing involvement of parents, cultural representatives and the students themselves has been significant in sustaining this TDI. The parents have become an integral part of the culture of the programme and their support has been significant in its continuation beyond the original three year term.

What changes in climate and philosophy have been required for the successful implementation of this programme? How were these changes managed, and how were changes in practice achieved?

The programme has changed very little philosophically over the three year period of Ministry-funded support. It has grown in size and become more heterogeneous with the inclusion of underachieving gifted and talented students, and gifted and talented Māori and Pasifika students. However, this was part of the original vision and simply took time to fully implement.

How appropriate were the identification procedures, curriculum adaptations and forms of assessment in relation to the goals of the programme? How has this contributed to student outcomes?

The identification processes were comprehensive and continuously reviewed and refined. These were aimed at selecting the most appropriate students to participate in the programme, not simply on the basis of their ability but because they would gain maximum benefit from being part of the TDI community and the specific approaches taken to teaching and learning. The very low attrition rate over the three years is one example of the efficacy of the selection processes. In many cases attitudes to learning shifted and motivation to learn increased, sometimes dramatically, and the students, their parents and teachers pointed to widespread and sustained gains in levels of achievement as a result. For some, this was clearly as a result of professional development. For others, it was possibly in response to the increased awareness of this group, associated with the school's involvement with the TDI.

What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme? To what extent has this specific programme design contributed to improved student outcomes?

This TDI incorporated both acceleration and enrichment. The programme was designed to build on each student's strengths and interests. This saw students working with mentors, undertaking research or independent projects, attending university classes, taking subjects in advance of their year levels, presenting, performing, etc. Many of these activities would not have been accessible outside this programme and as noted previously parents, teachers and students themselves were in no doubt that their achievement levels had risen as a result of participating in this programme. The support offered by peers in this TDI environment was reported by the students in particular, as a factor in their improved results.

How comprehensive are provider initiated student and programme monitoring and evaluations? How do the findings of the monitoring or evaluation inform the programme?

The TDI director took primary responsibility for evaluating the programme and individual student progress within it. Her approach was systematic and rigorous and involved all key stakeholders. The researchers were given full access to this data. The data gathered were used to inform adjustments at all levels.

What is the evidence for improved student learning and social, emotional or cultural outcomes as a result of participation in the programme?

The evidence of improvements in learning outcomes was evidenced in teachers' records of achievement and in the feedback from all those associated with this TDI. The attributions for these shifts were varied but many centred on more attitude shifts, increased motivation, and peer influence and support. The students, parents and TDI director repeatedly highlighted to the researchers the social and emotional benefits of belonging to the TDI group. A number acknowledged the difficulties that they had experienced socially and emotionally previously and how this had changed since they joined the TDI. This, they said, resulted in greater self-acceptance and increased confidence, which impacted positively on other aspects of their life inside and outside school. The cultural component of this programme was largely centred on the number of Māori and Pasifika students that were included in the group in years two and three. The ongoing involvement of Māori staff and parents in all aspects of the programme and the sensitive and skilful facilitation of the director, resulted in the Māori students and their parents and whanua becoming very strong supporters of this initiative.

How have resources and personnel impacted on the success or otherwise of the programme?

Central to the success of this TDI was establishing the director's position a full time role and to appoint to that role a teacher with knowledge of and experience in working with gifted and talented students. Her salary constituted the largest single resource item but it was also the most critical component to the success of this TDI. Resources were used to support students' programmes and were as diverse as their individual interests and strengths. Many of these could have not been supported without the additional funding that was associated with this TDI.

What role has staff professional development played in achieving the programme goals?

The TDI director took responsibility for her own professional development and over the three years took numerous opportunities, locally, nationally and internationally to increase her knowledge in areas relevant to her role. School wide professional development had occurred prior to and shortly after the implementation of the TDI programme. There was some continuation of this across the three years but it tended to be with staff with an interest in and a commitment to the work of the TDI. According to the principal, this approach of working with enthusiastic staff in the first instance to grow a commitment is more effective than working in a wider context and putting effort into changing the attitudes and practices of potentially ambivalent and antagonistic staff.

How well has the programme planning occurred in regard to sustainability of the programme after the three-year period of funding ceases?

This programme was thoughtfully designed and based on best practice in gifted and talented education. In the opinion of the researchers it is imminently sustainable, albeit in modified form. The core principles underpinning this TDI could translate into effective provisions for all learners and in fact the emphasis on community and relationships is at the core of some recent mainstream developments such as Te Kotahitanga. However, there are also numerous aspects to this programme that have been shown to be particularly effective with gifted and talented learners. Many of these principles and practices could be used to inform a wide range of educational responses to the gifted and talented, from inclusion to accelerated classes, and are not dependent on a model that organisationally resembles this approach.

What has been the impact of the programme on the whole school?

The principal contends that the most significant impact achieved by this programme across the school is the validation of success. While he believes this has probably translated into quantitative gains in achievement, it is the shift in culture that he contends will have the most enduring impact.

Te Manu Aute

Background

In the initial proposal to the Ministry of Education, Te Manu Aute (TMA) is described as being in a strategic alliance with Team Solutions, which is a division of the University of Auckland Faculty of Education. It was proposed that the Talent Development Initiative (TDI) programme would provide a three-year educational and professional development programme identifying and raising the achievement of gifted and talented students in the Arts. The target area was schools within the Tai Tokerau region of Northland. The programme aimed to offer school-based qualitatively differentiated programmes to primary, intermediate, and secondary schools in this area. The handbook Gifted and Talented Students: Meeting Their Needs in New Zealand Schools (Ministry of Education, 2000) was to provide the basis for this work.

Research Methodology

This section describes the research team, cycles of research, and data collection methods used in this case study. It is important to note that at the stage that the research to evaluate this TDI ended, the Ministry of Education funding of the programme still had six months to run. This had a bearing on the methodology, as the finding with the most potential to reshape this project was identified in the first part of the final year and there was no time to monitor its impact, although time existed within the contract period for changes to be made.

Research Team

The research team involved in evaluating TMA comprised three researchers with experience and expertise in gifted and talented education. One researcher is a specialist in music and the performing arts, and another in the visual arts. The programme director was also part of the research team.

Research Cycles

As this became primarily a virtual programme, the TDI director became an important data source and the researchers relied heavily on information from him and from his records. Although the original agreement identified two co-directors, when the research team became involved in mid-20063, there was a sole TDI director. At the first meeting, the TDI director was informed of the principles underpinning participatory action research and discussed with the researchers how it might occur with TMA. He was very enthusiastic about working as a 'team' and it was clear that he saw value in the range of expertise and depth of experience the research team brought to the project. Throughout the two and a half years the researchers were involved they had significant input into the project. The spiral cycle of planning, implementation, observation, reflection, and re-planning characterised the way the research took place. The scope of this project was extremely large and covered a wide geographic area. As the delivery of the programme was primarily virtual, the researchers often had face-to-face contact with participants that the TDI director had only had virtual contact with. In some instances, the researchers observed actual programmes that the TDI director had not visited in person.

The research team met with the TDI director for two days very soon after the programme commenced. This meeting took place at the TMA Centre a short distance from Warkworth. The main purpose of that meeting was to gain an understanding of the initiative and to examine relevant documentation in response to the question, What is going on? None of the research team would profess expertise in technology, so it was essential to become fully conversant with its application here to be able to investigate its effectiveness as a tool for learning. The TMA Centre was the technology hub for the programme and the director demonstrated how the digital and other resources would be used.

At this stage, the director was developing the online resources, building a database of tutors, mentors and performers, communicating with schools, devising a needs analysis schedule, and observation and rating scales for identifying giftedness and talented in the arts. The two researchers with expertise in the Arts played an active role in the development of the observation and rating scales, not only during that two-day visit, but in an ongoing manner.

At this initial meeting, a research plan was devised with the primary purpose of viewing the programme in action. Schools were to be clustered into actual clusters and virtual clusters. Much of the work with the actual clusters was to be face-to-face workshops and seminars. The virtual clusters would work primarily with sessions delivered online. Each researcher was to observe two cluster group workshops and two virtual cluster schools. As well as observing the programme in action, it was planned to interview teachers, students and parents. Specifically, the research team were interested in finding out:

  1. How had the students involved been selected for inclusion in the programme?
  2. What knowledge and expectations did teachers and students have of TMA?
  3. How well suited was the content and delivery to gifted and talented students?
  4. How do the students engage with the content?
  5. How are existing levels of knowledge and skill accounted for?
  6. What are the benefits of the programme for students?
  7. How does the TMA programme articulate with other programmes in the school?
  8. How is the programme internally evaluated?
  9. How does the school evaluate the TMA programme?
  10. How effective was the method of delivery?

There was resistance from schools to travelling to cluster group seminars/workshops and none eventuated in this first year. The researchers met with the TMA director for two days in December 2006 and a revised plan for visiting schools was developed. In the first six months of actual programme delivery, the researchers visited three schools. They also attended a Whangarei gifted and talented education schools' cluster group meeting where the TMA director presented on the TMA programme and the seven models for classroom delivery.

In the second and third years of the evaluation, two questions were simultaneously addressed: Is it working? How do we know? The research team communicated regularly with the TMA director and he was very facilitative, making material available on a regular basis. Access to the website meant materials could be viewed and evaluated as these were added, as could all seminars and workshops.

The school visits that occurred in 2006 did not provide the degree of insight into programmes that the researchers had hoped for. In one school, a failure of technology meant plans to take a programme live did not eventuate and the teacher had to use a previously recorded backup version of the session. A further technological problem saw this session halted part-way through. (This school withdrew from the programme shortly afterwards and a researcher revisited the school to discover the reasons for the withdrawal were staffing issues and technological problems.) In a second school, a clash with another event meant only a very small group of students identified as gifted and talented in the Arts were able to be observed and interviewed.

The researchers were aware that these three case study schools were schools that the director had ongoing involvement with, and may not have been representative of the larger group of participant schools. However, they did represent a mixture of Māori, rural and urban schools, and face-to-face and virtual delivery. In the beginning of the second year of the programme, the researchers had limited knowledge of how schools were actually using the TMA resources. It was hoped that a comprehensive questionnaire that the director had sent to schools would provide that information. However, the response to this questionnaire was very low. There obviously would have been no point to the researchers employing the same method in an attempt to answer this question. Arranging to visit a significant number of schools was proving impractical. There were potentially more than 60 schools as part of this TDI, scattered across a wide geographic area, and the researchers had no indication of the percentage of schools actively engaged in it. It was decided, in consultation with the TDI director, to undertake a telephone survey of all the schools who had signed up for the TDI. Each researcher took responsibility for calling a third of the schools and worked through a set of questions that had been formulated in consultation with the director. The questions were designed to explore how schools were engaging with the programme and their perceptions of it. Approximately 85 percent of the TDI schools responded to the telephone survey. It should be noted that the telephone survey did not collect data on out-of-school usage. The results from the survey were discussed with the director very soon after the data had been collected, collated and analysed. From the results of this survey, six schools were identified for a more in-depth investigation. This was the second visit for one of these schools. Arrangements were made to view the programme in action and to speak with staff, students, and, if possible, with parents.

Data Collection Methods

The data collected was primarily qualitative in nature and was derived from document analysis; observation; a telephone survey; interviews with the TDI director, principals, teachers, and students; online resources and seminar/workshop presentations; student products and presentations; the researchers' end-of-project reflective comments; and milestone and media reports. Documents, in this context, refers primarily to the records kept by the TDI director and includes programme information, rating and observation scales, programme and lesson plans and evaluations, participant feedback and evaluation data, website access data, presentations, and conference papers.

The data was analysed in a manner that informed the following four questions shown on the table below, which also summarises the pre-ordinate themes for this case study. The results are presented in relation to programme development and implementation; outcomes; and the impact of the programme for key stakeholders.

Table 11: Guiding Questions and Themes
Framework: Guiding Questions
How are programmes developed and implemented for gifted and talented students?
Themes
  • Programme design
  • Definition of giftedness and talent
  • Identification methods
  • Differentiated programmes
  • Evaluation
  • Resources
  • Professional support and development
What are the outcomes in relation to the programme goals?
Themes
  • Professional development and support
  • Trial and development of innovations
What is the impact of the programme for all stakeholders?
Themes
  • Evidence of growth, development, and change
Is the programme sustainable?
Themes
  • Future plans and directions
  • Resourcing

Programme Development and Implementation

The contract between the Ministry of Education and the University of Auckland stipulated that this TDI would be delivered between July 2006 and July 2009. The initiative would include up to 64 schools and up to 195 teachers. The schools would be grouped into seven clusters. Three clusters would be actual (involving significant face-to-face programme delivery) and four virtual (with mainly online programme delivery).

The broad objectives of the proposal were:

  1. The development, implementation, and evaluation of school-wide policies/procedure, plans and provisions, which meet the needs of gifted and talented learners in the Arts.
  2. The development of management systems and processes that ensure regular monitoring and review of these policies, plans, identification strategies, and policies.
  3. Quality classroom practice in the Arts through the development of understanding of the nature of giftedness and talent, and the needs of gifted and talented learners in the context of Northland Tai Tokerau schools.
  4. The development, use, and evaluation of strategies in the Arts that identify and address the individual needs of a wide range of gifted and talented learners, to specifically target groups identified in the Te Manu Aute school community needs analysis.
  5. The development, implementation, and evaluation of a range of strategies in the Arts to differentiate the learning experiences for gifted and talented learners through attending to the differentiation of content, process, and product as per the Te Manu Aute Qualitative Differentiation Model.
  6. Planning and implementation of classroom programmes in the Arts that use these strategies, so that the schools' Arts' environments are more responsive to gifted and talented learners.
  7. Initiate and contribute online support and/or development of materials in gifted and talented education in the Arts, and facilitate school development of their own e-learning base to provide sustainability in all learning structures.
  8. The building of professional learning communities in the Arts in the Northland Tai Tokerau region, incorporating cluster and peer support/mentor groups.
  9. Effective student learning and professional development in the area of developing talent in gifted students in the Arts must be collaboratively planned, school-wide and viewed as a long-term process.

Specifically, the Ministry of Education contracted the University of Auckland (TMA) to:

  • Provide professional development in the Arts to gifted and talented education teachers, and Arts' teachers with gifted and talented education;
  • Build on current Northland innovations by Te Kotahitanga, FarNet, and KAWN;
  • Trial and develop activities that utilise online virtual educational structures based on synchronous actual educational structures;
  • Explore innovative ways of using the Arts in gifted and talented education;
  • Develop digital resources for each module/class accessible by teachers and students as appropriate;
  • Use itinerant tutors, guest tutors, and visiting performers to deliver rich learning opportunities;
  • Develop systems and strategies for e-learning in the Arts using synchronous and asynchronous models;
  • Develop integrated policies and sustainable development plans for gifted and talented education in the Arts over teachers, schools, and cluster groups.

Initially the assigned personnel included two co-directors with a.5 time allocation each. Later one allocation was split in two, with one.5 director/arts facilitator and a shared.5 position between two arts facilitators. Most of the budget allocation was to cover professional fees and costs, and operational costs. As the TDI contract objectives show, this was a multi-faceted programme with ambitious aims geared at changing school-based practices in the Arts.The main aim of the programme was to raise achievement in the Arts, using technology to deliver qualitatively differentiated programmes.

The first six months of the programme was mainly devoted to developing the TMA website, disseminating information to schools, and inviting expressions of interest in becoming involved in this programme, and responding with information packages. Considerable work went into developing a needs analysis survey, and identification schedules for selecting students with talent in the Arts. At the same time, work was undertaken to compile a data base of mentors and experts who might contribute to the programme by delivering workshops or seminars that could also be participated in 'virtually' in real time, or accessed as cached files. Some short 'taster' courses were offered to provide schools with insight into what participation in the project would give them access to. Schools were sent a newsletter outlining setting-up procedures and providing contact details, and seven models for classroom delivery of the content. Exemplars of these models were presented online and were also available as downloadable files.

The original notion involved two main methods of delivery: actual instruction clusters (these were to be schools within a 100 kilometre radius of the TMA Centre at Warkworth) and virtual clusters (schools in the Northland Tai Tokerau region more than 100 kilometres from the TMA Centre). The first was face-to-face, and this would involve cluster group classes travelling to Te Manu Aute Centre in Warkworth and working with an expert. These clusters were made up of schools closer to the Centre. The second was online or digital/virtual presentations. These could be workshops or presentations accessed as live presentations in real time. The concept was that the class would be prepared for the session, with the appropriate technology and equipment in place, and could communicate with the presenter/facilitator by using a synchronous function on the rich-media presentation programme (Mediasite) to ask questions or make comments. The presenter/facilitator would be able to view the questions or comments on a monitor in front of him/her and respond verbally. Technically, any number of schools could participate in real time. Alternatively, all participant schools had access to these sessions as cached files and could access and use the files when and in a way that suited them. Using the cached files had the advantages for teachers of being able to preview the content, and to stop, restart, and revisit sessions.

Schools had access to a number of cached presentations that detailed the process for using TMA in the classroom. These included ICT setup (computers and networking with other devices), a system analysis (a capacity check prior to delivery), and classroom integration of webcasting (models to use in the classroom).

At the end of 2006, 42 schools had subscribed to the programme, some for an entirely virtual delivery, others for a mix of virtual and face-to-face delivery. That number increased in 2007 to more than 60. Early in 2007 some issues arose that were to result in modifications to the original programme design. First, the full impact of a fibre optic cable going no further than Orewa meant shifting the server location to Auckland. Second, schools without broadband did not have the band width to receive webcasting. Third, schools were not keen to travel to cluster group workshops, citing organisational, timetabling, and cost issues as impediments to their participation in these. This was a major setback because the cluster group workshops were also to be used for virtual workshops and as recorded resources. In addition, while many experts may be willing to work with a live group and be recorded doing so, being recorded working to a virtual audience would understandably be much less appealing. Fourth, the TDI director was now the sole person responsible for overseeing this programme and for significant periods was the only person directly involved. There were now more than 60 participant schools, all desiring a primarily digital delivery method of delivery.

The lead-in work required prior to the delivery of content to schools was extremely time-consuming. These tasks including promotion of the programme, compiling and distributing informational material; responding to enquiries; developing the TMA website; presenting 'taster' sessions to schools; designing needs analysis forms and observation and rating scales in music, dance, drama and the visual arts; preparing starter units; dealing with technological requirements and challenges; and advising and assisting schools with all aspects of the programme, including accessing online resources.

In the first year, two websites were developed and shared with schools:

  1. The first website: Te Manu Aute (A centre for gifted and talented students in the Arts) which included resources, newsletters, and a help section was designed for dialup access.
  2. The second website: Mediasite which included webcast presentation was designed to be accessed by broadband only.

In the first year, the TMA's resource website had attracted more than 20,000 'hits' and more than 10,000 teaching and learning resources, links, and files had been downloaded through the password protected area in the same period. This online material included information about the TMA programme and the identification of giftedness and talent in the Arts, and presentations/workshops in clay sculpting, kowhaiwhai, Nga Taonga Puoro, music, painting, dance, drama, and tukutuku.

There were two key elements to evaluating this TDI programme. The first was examining the nature of the content that was available schools. The majority of this was accessed via the TMA resources website, but there were also some face-to-face sessions with gifted and talented students. The second, and the more critical aspect, was understanding the efficacy of this content. It was impossible to determine from the presentation website data the extent to which the virtual presentations/workshops were being accessed at the time of presentation (live) and how many were used subsequently (cached). More importantly, it was very difficult to find out how teachers were using the material. Website data indicated that a significant amount of access to the resources website occurred outside traditional school hours. Over 80% of the activity was in the evening or at the weekend, with a significant proportion between 5pm and 10pm.

There were now more than 60 participant schools in this TDI, spread over a wide geographical area (although it was unclear how schools interpreted what that participation meant or how involved they were or intended to be). This would have been a challenging programme to implement with a fraction of that number of schools. With only a.5 staffing allocation to implement and oversee this project, any in-depth follow up to ascertain specifically what was happening in the participant schools was almost impossible. The TMA director developed a questionnaire for this purpose, but the response rate was so low that the findings offered little more than a cursory insight.

The scope of this TDI was large and the programme complex. There were few precedents in practice to make comparisons with and to predict with any confidence how it might evolve and the issues inherent in such a development. This initiative represented pioneering-type work and required a very responsive approach to the evaluation of it. The researchers were actively involved as the TDI director worked through a myriad of challenges. This project exemplifies the close relationship that can often develop in action research between the researchers and the researched. This level of 'intimacy' is held by many to promote appropriate change and understanding of practice (e.g., Waterman, Tillen, Dickson & Koning, 2001).

Defining Gifted and Talented Students

This TDI was designed to offer a differentiated programme of learning for gifted and talented students in the Arts. This reflects a contemporary and multicategorical interpretation of giftedness and talent where it is seen as broader than intellectual and academic ability. The Ministry of Education guidelines to schools (MOE, 2000) does not offer a single definition of giftedness and talent but the examples provided and the approaches suggested are multicategorical in nature and inclusive of music, the visual and performing Arts. This TDI applied the Ministry of Education's broad definition to these areas, viewing gifted and talented learners as those with exceptional abilities relative to most other people and possessing learning characteristics that give them the potential to achieve outstanding performance.

Identifying Gifted and Talented Students

Implicit within this TDI was the notion that exceptional or potential ability in the Arts can detected by focusing on dispositions, interests, behaviours, products and performance. Considerable preparatory work was undertaken to identify the specific elements within these that were salient to the identification of exceptional ability, and to present them in a manner that teachers in particular could use to identify this ability. The TDI director undertook a comprehensive review of the literature in this field. He was particularly attentive to Māori perspectives on giftedness and talent and how these might inform the identification of Māori students with ability in the Arts. The two researchers with expertise and experience in the Arts made a significant contribution to the development of these schedules. Typically, the TDI director would develop a draft schedule and seek feedback on the draft. This feedback was provided tentatively and usually in the form of questions, rather than direct suggestions for change.

These tools can assist teachers identify giftedness and talent in music, dance, drama and the visual arts and represent a unique resource. The teachers surveyed who had used the schedules to identify gifted and talented students in these areas gave positive feedback on both their efficacy and usability. However, most teachers will still need support to effectively use the instruments and how to incorporate them into a multidimensional approach to identification, of which these should be one component.

Differentiating for Gifted and Talented Students

This was first and foremost an initiative aimed at offering a differentiated programme for students gifted and talented in the Arts. With appropriate associated professional learning, the tools developed to identify such students have the potential to make a significant contribution to what is currently available in New Zealand. However, the question constantly revisited by the researchers was the appropriateness of resource material for gifted and talented students. The issue here is one that was raised as problematic in the previous section. There value of these materials for gifted and talented students is completely dependent on how they are used. It is clear from the findings that many teachers were not knowledgeable or experienced in differentiating curricula for the gifted and talented. This resulted in the resources being seen as a programme, whereas, in the opinion of the researchers, the value of this material was as a resource that was part of a programme designed for a specific group whose needs had been thoroughly assessed and carefully planned for. The full benefit of this TMA could only be realised if it was supported by a comprehensive programme of professional learning. In the first instance, this could have been best achieved through a small-scale pilot programme.

In this programme, given its scope and complexity, it was impossible for the director have more than a cursory knowledge of how the materials were being used by teachers. It is not possible to produce generic material for gifted and talented students that is appropriate for even a single year level. What might challenge a gifted six year-old may offer no challenge to a non-gifted nine year-old. What might be appropriate for one musically gifted six year-old may be inappropriate for a same-age peer with ability in the same area. Our observations revealed that some teachers were using these resources in a manner that did not offer real challenge to their gifted and talented students. In fact, some teachers were using the resources with all the children in their classes. In the view of the researchers, the material could form the basis for an effective enrichment programme for gifted and talented students in the Arts. However, the degree of effectiveness is totally dependent on how these are used and whom they are used with.

At the end of 2008, the TMA website included a large and diverse set of resources. It should be noted that the development of resources is ongoing and new initiatives are being trialled. Although the research team had viewed and assessed this material as it was added to the website, a more systematic approach to evaluating the entire set of resources was undertaken in November and December 2008. The researchers assessed the material for its quality, generally, and, more specifically, its appropriateness for gifted and talented students. The two researchers with expertise in the Arts were able to evaluate the resources from an Arts' perspective and provide feedback to inform practice.

When considered from the perspectives above, the material was variable in quality. Some of it was of a very high standard on all counts, and the material designed to assist schools to identify gifted and talented students in the Arts could be used as exemplars in this area. However, the overriding conclusion of the researchers was that much of the material had the potential to be very effective in realising the goals of the TDI, but needed to be further developed to achieve this. That should not be viewed as a negative comment on this programme. The aims of the TMA TDI, as stated in the contract, included terms such as 'trial and develop' and 'explore innovative ways'.

The resources developed for use with gifted and talented students in the Arts were considered to be, at least on face value, most appropriate as enrichment activities. The extent to which these might challenge gifted and talented students was difficult to estimate without knowing how they were being used.

Evaluating the Programme

It has already been noted that the director (and the researchers) faced some real challenges evaluating the effectiveness of this programme. The large number of schools potentially involved in the programme, combined with the demands of a complex and evolving programme and a very low staffing component, were significant obstacles to implementing approaches that elicited quality data that could inform programme development. There was the ongoing provision for teachers and schools to offer informal feedback through email contact with the TDI staff. The director always provided quick and full responses to this feedback but teachers and schools made little use of this mode.

While statistical data was available relating to the quantity and nature of what was accessed online, this data gave no indication of who was accessing the site or downloading resources or for what purpose. The research team were able to provide the director with feedback from their case study schools and he had gathered his own anecdotal data. In the second year of the programme the director sought to address the lack of feedback and sent a comprehensive online questionnaire to all participant schools. As noted previously, the response rate to this questionnaire was extremely low and offered little to ameliorate this situation. The telephone survey undertaken by the researchers provided the first full picture of what schools were involved with the programme, and the nature of this involvement.

It is important to again note that at the time the evaluation of this TDI was completed and the report compiled, the contract had a further six months to run. In the light of the findings from the telephone survey, the director began to implement a comprehensive system of evaluation including online polls, individual and focus group face-to-face and recorded interviews, student evaluations, and SurveyMonkey evaluations.

Resourcing

This TDI required a high level of physical resourcing and this comprised a significant portion of the contract budget. The researchers visited the TMA headquarters in Warkworth and viewed first hand the resources acquired to deliver the programme. None of the researchers would consider themselves experts in the technology used to support this programme and any assessment of the effectiveness or appropriateness of these tools to develop, deliver and support the TDI were arguably limited by this lack of expertise. However, throughout the period the programme was evaluated there was no indication that the TDI was constrained by a lack of access to appropriate hardware or software. Any technological limitations were related to the inadequacy of systems outside the control of the TDI.

The major resourcing shortfall and one that the researchers identified as significantly constraining the programme, was human. As has already been discussed, the staffing component of this TDI was arguably far too low to successfully deliver the programme and this situation was further exacerbated by ongoing difficulties finding and retaining appropriate personnel.

Professional Support and Development

The Te Manu Aute Advisory Committee was put in place to support and guide the work of the programme from its inception. This group was comprised of representatives from a range of backgrounds relevant to the focus of the TDI, including gifted and talented education advisers. Over time, the input from this group appeared to diminish.

The director was extremely active in sourcing material that informed his programme and in developing connections with individuals and organisations undertaking similar work both in New Zealand and overseas. These connections sometimes led to new developments. He also took every opportunity to share the TDI concept and activities with others. This exposure sometimes resulted in worthwhile feedback from people with expertise in areas relevant to the TDI. He was also a frequent participant at gifted and talented, and technology conferences. His permanent position as part of an organisation offering professional development to teachers gave him access to professional support and development in areas related to the TDI.

Summary: Programme Development and Implementation

The Te Manu Aute Talent Development Initiative represents an innovative and contemporary model for supporting gifted and talented students in the Arts. The region that this programme was introduced in is comprised of many small and remote rural schools. These schools would potentially benefit greatly from virtual methods of delivery. The concept and objectives of this TDI received widespread endorsement from teachers, although the actual school engagement with it was very small. A telephone survey of most of the participant schools found that the main reason for this low level of sustained involvement was a lack of support, particularly in the early stages of the project. However, their support for what the project was attempting to do, remained high. The materials that are available to the TDI schools represent a very rich resource that could make an important contribution to an enrichment programme in the Arts. Of particular value to schools are the observation and rating scales. However, the effective integration of these resources into a programme for gifted and talented students in the Arts is still dependent on teacher knowledge and skill.

Impact of Programme

The researchers found universal support for the underlying ideas that TMA was based on, and what the programme aimed to achieve. The major strength of the programme was seen by respondents to be the concept of using the internet to deliver programmes to schools in the Northland Tai Tokerau region, a region with many small, rural, remote, and low decile schools. These schools reported that it was not easy for them to access directly the type of programmes that TMA offered virtually. Another reported strength of the programme was that much of the material was local and designed not only for use in New Zealand schools, but schools in the Northland Tai Tokerau region. The focus on gifted and talented students was supported by some teachers who felt that these students were not only a low priority in many schools, but that it was difficult to meet their needs in small schools. The focus on the Arts was reported by some teachers as potentially meeting a need in their schools. The inclusion of Māori content and pedagogies was also endorsed by a number of teachers. Teachers of some smaller and more remote schools responded positively to the notion of clustering schools on occasions.

Endorsement for the concept and objectives can be seen in the results of the telephone survey of TMA schools. Although the majority of these schools had never actually participated in the programme or else their involvement was minimal or for a short term, almost all thought that the concept and objectives were sound and that if it continued, they would consider involvement. A teacher from one school that had signed into the programme but not actually participated, when asked to comment on the strengths of the TDI said, "The idea itself is a strength." The gifted and talented education teacher from a school that had been involved "minimally" said the school signed up for the programme because it was to be delivered online, offered access to experts in the Arts, and that there would be real time collaboration if needed. The support for the TMA and what it aimed to achieve was supported by teachers in the case study schools. One teacher commented, "The idea is brilliant and it's meeting a real need. This is definitely the way of the future. The kids love it."

It is not difficult to understand why this TDI proposal was successful in attracting Ministry of Education funding. The concept, philosophy, and programme design match Ministry of Education priorities and offer a response to these that is innovative. With the benefit of hindsight, the issue of understaffing is very evident. However, in defence of those who negotiated this contract, it was not easy to ascertain the full scope, or to appreciate the complexity of what was involved from the proposal. In fact, the research team struggled in the initial stages to fully understand how the programme would operate in practice. To some extent, having limited experience of the technological aspects that were part of this TDI was an impediment to achieving this.

The role of the researchers was to evaluate, not to audit, the programme. However, when it became apparent that the effective implementation of this TDI was likely to be undermined by the inadequate staffing allocation, it was suggested to both the director and to Ministry of Education personnel that consideration be given to reducing the scope of the TDI.

This issue was not a reflection on the commitment, dedication, or enthusiasm of the director. However, the technological aspects alone were extremely time-consuming. Added to this was a need to develop resources, deliver and record presentations, visit schools, maintain a web site, etc. The more essential and pressing tasks overtook any in-depth and ongoing investigation into how schools and teachers were actually engaging with the programme.

The researchers needed this information for their purposes, and, in 2008, conducted a telephone survey of the schools that had signed up to be part of TMA. Approximately 85 percent of schools participated in the survey, with most involving interviews with a lead teacher. From those responses only about 10 percent of schools could be described as having any sustained involvement in the programme. Within this group, some had started very recently and others had discontinued participation. Interestingly but not surprisingly, schools with the highest level of engagement were the schools that had had ongoing face-to-face contact with the director or one of the two Arts facilitators. There were three main reasons given by schools who had no, or minimal, involvement, even though they had signed into the programme.

First, some schools clearly needed more support and information, and at the earliest stages, to actually get them started and to generate and maintain motivation and momentum. While this information was provided by TMA in newsletters, emails, webcast, cluster meetings, and downloadable material, the initial process was new to some schools. Some of the issues involved complete misunderstandings about the programme and how it would be delivered.

We enrolled but we didn't manage to participate because it was in Whangarei … We thought TMA would be taken into schools but then I was sure I read in the letter that the children had to be taken to Whangarei. We can't do that at a three teacher school … we don't have time to take the kids and we can't expect parents to travel the 168 km. But the information sounded good but the school doesn't have the resources or money. (Teacher, telephone interview, 2008)

The teacher responsible for gifted and talented education at a primary school (that had signed up for TMA but never participated) stated in the telephone interview, "The information sounded interesting but it was never facilitated. We were never really sure what was supposed to happen. They never got in touch. We never knew anything – nothing happened." The gifted and talented education teacher at another school that signed on but did not proceed with the programme had a similar response: "There needs to be someone to come into the school and to take us through the information so it is more personal, not so we are just being informed by emails." A teacher from a case study school with minimal involvement said, "For us [what was needed] is maybe occasionally to have more face-to-face; a face-to-face networking meeting or networking with other schools maybe once or twice a year just to get some sort of momentum and commitment. We haven't met face-to-face with [the TDI director]." This teacher endorsed the TMA concept, and was very positive about the content, but believed a mixture of face-to-face and online delivery, and support networks of local schools and teachers would have enhanced its effectiveness. A high school gifted and talented education teacher from a case study school that had become very involved in the programme in the previous 12 months, was extremely enthusiastic about what it offered, but she too felt that there needed to be more face-to-face support.

A second, but less frequently cited reason for not starting or for discontinuing, was to do with technology, or a lack of staff with sufficient technological knowledge. One secondary school gave their reason for not participating as the lack of internet access in any area of the school other than the library. Given that this was a programme in the Arts, the principal decided that it was impractical for the school to be involved. A primary school principal said that her school's participation was "random" as, "the passwords sometimes did not work and children and teachers could not find the activities – they simply weren't there." As consequence, the children became frustrated and lost interest. A number of schools reported that it was their own equipment or limitations around internet accessibility in their area that impeded their involvement. This was more of an issue in small and more remote schools.

We actually didn't do it. One we didn't have access to that amount of online material … So we didn't have the ICT capabilities but two, we didn't know what it was and how to deliver it. But we did get lots of emails on a regular basis.

A case study primary school that withdrew from the programme after one year cited issues around their own ICT capability and space as the reasons. According to the teacher involved with TMA, the students became "de-motivated when the technology problems meant access to the live streaming was made difficult." Although the school had been provided with a backup DVD, "(the school's) poor sound and lighting meant the children could not see or hear the presentation adequately. For some time, nine children had to crowd around the teacher's laptop."

A number of schools gave as their reasons for non- or discontinued involvement, issues relating to their own internal organisational arrangements. These included factors such as competing commitments, timetabling, and staff changes. A small number of teachers admitted that the failure of the programme to succeed in their schools probably lay with their own lack of commitment or motivation. Three schools reported that they examined the material and decided that it was inappropriate to their needs. A further three schools said that their students lost interest in the programme and that was the reason for the school unofficially withdrawing.

However, although the level of sustained participation was very small, there was strong support from staff in these schools for the work of TMA. The most 'successful' schools were those where the director or other Arts facilitators had established close connections or made visits in person. This close engagement not only gave impetus to the programme and provided it with practical support, but the expert knowledge of the director gave it credibility.

His knowledge is great and when this is not your area of expertise that's invaluable … he is proactive, excited and knows what the kids like and what we like and he responds appropriately. The students think it's great. The parents think it's great – although I am not sure if this is because their child was selected.

This teacher also provided an example of a change in student behaviour as a result of the programme. One item in the telephone survey asked teachers if they had noticed any shifts in students' attitudes and behaviour as the result of participation in the programme. This teacher spoke of a student whose behaviour had previously caused the school concern but that her involvement in TMA had changed this.

Sustainability

It is important in this section to note that as evaluators of the TDI the researchers fully support the concept and objectives of this innovative programme. A proposal to deliver high quality educational programmes to an area of the country that is geographically isolated, and to a traditionally underserved group within that population, is not difficult to defend. The director is commended for his vision, passion, enthusiasm and dedication to implementing it; for his willingness to provide unobstructed access to all aspects of it; and for the opportunity for the researchers to have input into the programme.

However, in the researchers' opinion it is not sustainable, even if the level of funding was to remain the same. This work was always too large and complex to be managed and implemented with a staffing allocation equivalent to a single fulltime position. The recommendation, made part way through the contract, was for this to become a scaled-down pilot programme. The recommendation remains unchanged in suggesting a way forward. A pilot programme would allow the ideas and practices to be trialed and refined and to potentially form the basis for a large-scale programme in the future.

Summary: Te Manu Aute

The summary of results for this case study is presented in relation to each of the research questions.

How were decisions around the programme design arrived at? Who was involved in the decision-making process? How has this process impacted upon the sustainability of the programme?

The programme design was primarily the work of the TDI Director who brought to the project a substantive background in the Arts and ITE and a growing understanding of gifted and talented learners. An Advisory Committee was formed very soon after the contract was awarded and this group included two gifted and talented advisors. However, this committee did appear to be less involved later on in the project and it is possible that some of the delivery problems that emerged over time may have been picked up by this group and responded to at an earlier stage. The research team, especially the two members with a background in the Arts, had significant input into the content and approaches of this TDI. The Director was also extremely active in sourcing pertinent literature and research and in attending conferences and seminars relevant to the TDI work. He also consulted extensively with the Māori community. It is very difficult to draw any definitive conclusions about how this process has impacted on the sustainability of the programme as it has continued beyond the research of it. The finding that the uptake of this TDI by schools was very low came late in the research project and there is every likelihood that the director has responded to this in a way that enhances its sustainability. The general design of the programme was sound but where more input was needed was in its delivery.

What changes in climate and philosophy have been required for the successful implementation of this programme? How were these changes managed, and how were changes in practice achieved?

A major aim of this TDI was to offer schools a programme in the Arts that was available in real time and interactive. This was made available to schools but the uptake of this was minimal, teachers preferring to used the recorded programmes in a way and at a time that suited their needs. The original intent was to provide cluster workshops on site at the TMA headquarters and using experts to deliver these. Again, schools did not take up this opportunity for reasons that have been tabled previously. The use of mentors, experts and tutors was much less than originally proposed. The use of mentors and experts became less viable with the resistance of schools to cluster workshops and the difficulties encountered recruiting appropriate tutors meant that this became a diminished feature of the programme. The philosophy remained but the director was forced to accept numerous constraints to delivering on this. This meant adjusting approaches to reflect how schools were choosing to engage with the materials, which was more as a resource than in the form of streamed presentations or workshops. It also resulted in the director working more directly with some schools than had probably been planned for. In the opinion of the researchers, the responses to these shifts could have been better managed by an increased staffing component.Simply appointing a person to attend to the administration, management and some of the technical aspects of this project would have freed the director to have had a greater degree of professional involvement.

How appropriate were the identification procedures, curriculum adaptations and forms of assessment in relation to the goals of the programme? How has this contributed to student outcomes?

The identification tools and the supporting material outlining how these should be used are excellent and will be of ongoing value. How teachers have used these in practice is not clear. This TDI aimed to provide a differentiated learning programme for learners gifted and talented in the Arts. The available resources definitely have the potential to contribute to such a programme. However, for most teachers this would be an impossible challenge without the support of a dedicated programme of professional learning. This is a major reason why the uptake of the TDI was low and why there was limited evidence of improved outcomes for students.

What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme? To what extent has this specific programme design contributed to improved student outcomes?

The materials developed as part of this TDI and made available to schools covered all aspects of the Arts curriculum. This provided schools with access to a resource or teaching workshops that would not be readily available elsewhere. However, as noted above, without a parallel programme of professional learning to accompany these materials, the effectiveness of their usage was left to the school and the teacher. From the limited data that was available to the researchers there was little doubt that teachers and students found the workshops and resources interesting and engaging but the research team found very little evidence of improved student outcomes. To achieve this would require much greater attention to supporting teachers in incorporating the materials into a differentiated programme of learning.

How comprehensive are provider initiated student and programme monitoring and evaluations? How do the findings of the monitoring or evaluation inform the programme?

The evaluation of this programme presented some challenges, including the size of the participant groups, the large geographical area the schools represented, and the low TDI staffing component. The obvious way to obtain feedback was through online surveys or questionnaires. However, the response to this method was so low that the data was of very limited value. The Director had comprehensive data from the teachers and students he worked with directly but this was a very small portion of the overall group and the face-to-face involvement atypical of the delivery of the TDI in the majority of the participant schools. The most significant evaluative data on the TDI was that obtained by the researchers in their telephone survey of all schools in the third year.

What is the evidence for improved student learning and social, emotional or cultural outcomes as a result of participation in the programme?

The potential for outcomes across these dimensions exists but the evidence of this is very limited. The researchers observed students engaging with the activities and some clearly acquiring new knowledge and skills. However, these observations were across a small sample of schools, some of which were better supported than the majority. Even within these schools, the content was not always as precisely matched to individual students' needs or abilities. Mention does need to be made of the cultural elements of this TDI and there was clear evidence of enthusiastic engagement by Māori students. Given the low uptake by schools generally, this finding was limited to a very small group of schools but the potential here needs to be acknowledged.

How have resources and personnel impacted on the success or otherwise of the programme?

As has been noted previously, the researchers were unable to find evidence to suggest the resources had achieved any widespread impact across the participant schools, as the uptake of this programme was so low. Where TDI staff, particularly the director, were personally involved in schools, this did have an impact and increased the likelihood that the materials would be more appropriately used. The Director's expertise in the Arts and technology, and his experience working with teachers and students, was valued highly by those he had direct contact with. His enthusiasm for the programme was contagious the researchers were left wondering how much more effective this TDI would have been if he had been able to be released to work more closely with schools and teachers. Almost all of the schools most engaged with this TDI were schools the director had had some face-to-face contact with.

What role has staff professional development played in achieving the programme goals?

The importance of this aspect of the programme was probably not recognised even at the planning stages. The expectation that teachers would be able to effectively access what was offered and translate it into a differentiated programme for gifted and talented students in the Arts, without an accompanying comprehensive and easily accessible programme of professional learning, turned out to be extremely optimistic. This situation was arguably exacerbated by having two areas where teacher professional knowledge is somewhat limited: the Arts, and gifted and talented education. The programme director did undertake his own professional development, but this was not a planned element of the programme.

How well has the programme planning occurred in regard to sustainability of the programme after the three-year period of funding ceases?

Again, the research into the effectiveness of this TDI was completed six months prior to the contract ending. There appeared at that stage to be little attention to its ongoing sustainability. The funding to support this initiative was large and there would seem to be little likelihood that funding would be available to deliver anything other than a very scaled-down version of it. Unfortunately, the potential funding sources for such an initiative would appear to be limited.

What has been the impact of the programme on the whole school?

From the data gathered from the participant schools the researchers saw no significant school-wide impact.

Synthesis of Results: Ignite Evaluations

As the previous sections show, each Ignite programme was different in its outcomes, approaches to reaching those, and subsequently, the data collection methods employed to determine effectiveness. The purpose in this evaluation is not to cross-analyse or compare the Ignite programmes; however, some common themes arise in response to the research questions.

Research Questions: Ignite Evaluations
How were decisions around the programme design arrived at? Who was involved in the decision-making process? How has this process impacted upon the sustainability of the programme?

The programmes directors were the key catalysts for programme design. The programme directors each had a background and experience in the practice and theory of gifted and talented education, as well as their specialised disciplinary and age-level groups of learners. The directors also engaged in their own ongoing professional development and learning. This enabled the decisions around programme design to be based on sound theory and practice, as advocated by the Ministry of Education (2000).

The programmes were designed with input from key stakeholders and evolved, to greater and lesser extents, based upon input from teachers, students, parents/whanau, and other stakeholders. The three programmes highlight the importance of an advisory committee, representative of stakeholders, and able to offer ongoing involvement, support, input, and buy-in. This ongoing engagement with stakeholders in the decision-making processes better ensures programme sustainability.

What changes in climate and philosophy have been required for the successful implementation of this programme? How were these changes managed, and how were changes in practice achieved?

For each of the Ignite TDIs, the philosophy of the programme did not shift over time; but the realisation of the philosophy in practice did shift. The approaches to pursuing the philosophy in practice were adjusted, refined, and enhanced based on both the internal and external evaluations. A key point to note is that each programme's original vision required time to implement, and over three years of implementation the programmes evolved to better align with the intended outcomes. This evolution of philosophy to practice was hindered by a lack of adequate staffing for the TMA programme.

How appropriate were the identification procedures, curriculum adaptations and forms of assessment in relation to the goals of the programme? How has this contributed to student outcomes?

The identification methods were comprehensive, multi-method, and appropriate in relation to the programmes' goals. They also demonstrate the important role of professional development and support in the implementation of identification methods. Professional development – and its ripple effect of increased teacher awareness – leads to evidence of improved outcomes for students; without this, there is little evidence of improved outcomes. Identification linked to curricular differentiation, supported through professional development, and continuously evaluated by all stakeholders can lead to improved outcomes for gifted and talented students.

What aspects of curriculum differentiation have been designed specifically to meet the major objectives of the programme? To what extent has this specific programme design contributed to improved student outcome?

The three Ignite programmes differed in their approaches to differentiation:

  • The NZMSC programme used a specific scientific discipline as a vehicle for exploring a conceptual theme in an authentic scientific environment, enabling like-minded peers to work together using a range of processes and resulting in an original product for an appropriate audience.
  • Rutherford College facilitated the development of individual differentiated programmes incorporating enrichment and acceleration, based on strengths and interests, for individual students who were grouped together for peer support.
  • TMA used all aspects of a curricular area, the Arts, to provide teachers with otherwise unavailable resources that were interesting and engaging; however, the lack of professional development did not ensure differentiation for gifted and talented students in practice.

What this shows is that curriculum differentiation can take many different forms, but must address content, process, and product adaptations based on individual student differences as unearthed through a comprehensive, multi-method process of identification. In order to be effective in practice, this matching of students to curricular adaptations requires teachers who have professional development and support.

How comprehensive are provider initiated student and programme monitoring and evaluations? How do the findings of the monitoring or evaluation inform the programme?

The provider-initiated evaluations aimed to be comprehensive, ongoing, inclusive of stakeholders, and useful for informing programme developments. Internal evaluation of this scope was hindered by different factors, but ultimately related to staffing issues. The programme evaluations show that quality can be enhanced through the refinement and adoption of measures matched to goals, as well as the development of stakeholder trust. Each of these programmes evolved in response to programme evaluations – both internal and external. The three Ignite initiatives all demonstrated the value in external input from a team of researchers using a participatory action research approach.

What is the evidence for improved student learning and social, emotional or cultural outcomes as a result of participation in the programme?

The evidence of improved student learning and social, emotional or cultural outcomes is found in the internal and external evaluation data collected from stakeholders. The extent of gain in these areas varied across the programmes, dependent on the programme goals and implementation. This demonstrates the interrelationships between intended outcomes, implementation, and evaluation – and the cyclical nature of these elements in programme development.

Both NZMSC and Rutherford College demonstrated attainment of learning outcomes for gifted and talented students, and both highlighted positive social and emotional benefits. TMA and Rutherford also evidenced cultural outcomes (specifically for Māori and Pasifika students) through the involvement of students, their whanau, and teachers.

How have resources and personnel impacted on the success or otherwise of the programme?

Each of the Ignite programmes primarily used their TDI funding for salaries for the directors who designed, implemented, and evaluated the programmes. As has been stated previously, the programme directors were key catalysts. What this means is that the financial resources provided specialised, dedicated positions in gifted and talented education. Without adequate resourcing for staffing, programmes will have limited success, as TMA has shown.

What role has staff professional development played in achieving the programme goals?

The programme directors engaged in their own ongoing professional development and learning, including their engagement with the external evaluation. Professional development and support played key roles in the achievement of programme goals for both Rutherford College and NZMSC, using different methods of delivery to key players based on need. In other words, professional development was an integral component to these programmes, but it was also flexible and responsive. TMA did not include a professional development and support component for schools, and this hindered the achievement of some goals.

How well has the programme planning occurred in regard to sustainability of the programme after the three-year period of funding ceases?

The financial sustainability of these programmes is problematic given the funding allocation was primarily for salaries and staffing, and this is magnified by the limited external funding sources for gifted and talented education. Only one of the programmes (NZMSC) actively addressed financial sustainability beyond the three-year period of funding, by seeking external financial support. Additionally, both NZMSC and Rutherford College developed systematic, transparent models which ensure sustainability of principles and practice that are not dependent on specific organisational models of delivery. The ongoing collection of evaluation data also ensured sustainability. In the case of TMA, there was little evidence of planning for financial or programme sustainability.

What has been the impact of the programme on the whole school?

The impact of each programme for schools varied in extent and scope based on each programme's goals. For Rutherford College, there was a shift in school culture to one which placed greater value on success. For schools participating in the NZMSC programme, shifts were seen in identification, awareness, resources, and professional development. There was no evidence of school-wide impacts as a result of the TMA programme.

Summary: Ignite Evaluations

As the synthesised results show, to a greater or lesser degree, in the development and implementation of each of these programmes, there was evidence of:

  • Systems to determine needs of stakeholders. As the programmes were being implemented and evaluated, the systems were refined in an attempt to better match the programme to stakeholders' needs.
  • Commitment to ongoing data collection to determine programme effectiveness.
  • Responsiveness to evaluation processes, both internal and external, as shown in cyclical programme change and improvement.
  • Programme evolution leading to a more effective programme that was more closely aligned with its intended outcomes and goals.

The process of developing a programme and continually refining it in response to evaluative findings was a strength of each of the Ignite TDIs. It demonstrates the fundamental role of programme evaluation: to provide information that can be used to improve and advance programmes for gifted and talented students (VanTassel-Baska, 2004c). The evaluation of programmes is part of the cycle of programme development.

There were shared principles underpinning each of the TDI Ignite programmes, and these reflect contemporary theory and practice advocated in New Zealand (Ministry of Education, 2000, 2002; Riley et al., 2004). Each Ignite programme demonstrated, to a lesser or greater extent dependent upon their programme goals, evidence of:

  • Liberal rather than conservative approaches to defining giftedness and talent.
  • Systematic, multi-method, and inclusive identification methods.
  • A range of outcomes for students: intellectual, academic, social and emotional, and cultural.
  • Qualitative differentiation of content, processes, products, and the learning environment.
  • Interrelationships across programme elements: definitions, identification methods, programmes, and evaluation worked together and mirrored one another.
  • Written documentation and record-keeping.
  • Dissemination and sharing.

These shared programme attributes were enabled by the Ignite TDI directors, all of whom were passionate and committed to the education of gifted and talented students. Each director had a vision not just for their programme, but for students, and they were enthusiastic in their pursuit of making that vision a reality. Over the three year period, each programme director actively sought to enhance his or her professional knowledge and skills in gifted and talented education. As a result, the programme directors for these Ignite TDIs demonstrated growth in their understandings of theory and research, and this was demonstrated in their practice.

The three Ignite TDIs were innovative, putting the core principles of gifted and talented education (Ministry of Education, 2002) into practice, and developed from prior experience and commitment to gifted and talented education, as was outlined in the criteria for selection of TDIs. Innovation needs to be embedded in other programmes and/or develop from existing programmes in order to better ensure sustainability. These programmes also focused on outcomes for gifted and talented students from diverse backgrounds and experiences, who might otherwise be overlooked: this too reflects the intentions of the 2006-2008 TDI funding pool. These three programmes met the primary aim of the TDI funding which was the development of innovative approaches in gifted and talented education that result in improved outcomes for gifted and talented students. Another aim of the TDI funding was to conduct research into the impact of innovative approaches on learning and teaching, and that was the overarching purpose of this study.

Footnotes

  1. This session was conducted by a member of the research team who shared preliminary findings and from those provided professional development.
  2. Note: This TDI began in July 2006 and was not finished at the end of the evaluation contract. It completed its contract with the Ministry in June 2009, approximately 6 months after the completion of data collection for the evaluation contract.