Summary

Introduction and objectives

The Arts in the New Zealand Curriculum requires secondary schools to provide rich learning experiences that will benefit student understanding in the key learning areas of the four strands of each Arts discipline (Dance, Drama, Music and the Visual Arts). The Secondary Schools Arts Coordinators Project, by facilitating such experiences, is intended to: enhance student understanding and achievement in arts education in the four disciplines of The Arts in the New Zealand Curriculum; encourage students to develop a better appreciation of the Arts; support and enhance school participation in Arts/cultural competitions or events; support links with the school's Arts community; and (where an Arts Coordinator is employed) provide coordination and administrative support for the arts curriculum.

The Project is divided into two strands. Strand 1 provides an annual allocation for individual state and integrated schools which can be used to provide enriching Arts opportunities and/or to employ an Arts Coordinator to facilitate these opportunities. Strand 2 involves National Secondary Schools Arts Coordinators providing additional information for schools, through a web site and hard copy sent to schools, about opportunities in Dance, Drama, Music, and Visual Arts in each region. The project is in its third year, with an increasing number of schools participating in each round.

The purpose of this Evaluation is to assess how effective the Project has been in supporting Arts teachers and the Arts curriculum. Specific research questions focused on:

• implementation of the Project, including who is involved in the Project;
• effectiveness of Strand 1 of the Project in relation to the impact on teaching and learning;
• effectiveness of the National Secondary Schools Arts Coordinators (Strand 2);
• resourcing of the Project;
• effectiveness of milestone reporting (which schools provide to the Ministry); and
• suggestions for future improvements to the Project.

Method

The Evaluation of the Secondary Schools Arts Coordinators Project had a number of different components:

• Compilation of a profile of schools involved in the Project in 2003 and a comparison with a profile of eligible schools (that is, all state and integrated schools with Year 9 to 15 students) that were not involved in 2003.
• Analysis of a 50% sample of May 2003 milestone reports to the Ministry of Education from schools involved in the Project. The analysis covered Arts discipline, type of opportunity/activity, Achievement Objectives, student participation, outcomes, and cost for the 1,024 opportunities/activities reported across the n=127 schools.
• A survey of a 50% sample of schools involved in the Project in 2003, involving Principals, Arts Coordinators, and Heads of Departments (HODs)/Teachers in Charge of the four Arts disciplines: Dance, Drama, Music, and Visual Arts.
• A total of 369 questionnaires were received - 94 Principal questionnaires (a response rate of 61%), 83 Arts Coordinators questionnaires (59%), and 192 HODs/Teachers in Charge questionnaires (37%). The response rates for HODs/Teachers in Charge of each Arts discipline ranged from 28% for Dance to 45% for Visual Arts.
• The lower than expected response rate for the survey may place some limitations on the accuracy of the data. It is not, however, possible to quantitatively determine what these limitations are.

Implementation of the Project

In 2003, all state and integrated schools with Year 9 to 15 students (N=434) were eligible to apply for funding through the Ministry of Education's Secondary Schools Arts Coordinators Project (the Project). Comparisons between the schools participating in the Project in 2003 and the schools that are not participating in 2003 show that, while a wide range of schools are participating in the Project, small schools, rural schools, composite and special schools, low-decile schools, and schools with high proportions of Māori or Pacific students were less likely to be involved in the Project in 2003.

Most of the schools involved in the Project in 2003 had been involved for two or three years - 43% had been involved for two years and 37% for three years.

Analysis of the milestone reports showed that, on average, fewer Year levels were included in any one opportunity/activity in Visual Arts than in the other three Arts disciplines. This suggests that opportunities/activities offered in Visual Arts were more targeted at students in specific Year levels.

Almost all schools involved in the Project in 2003 had employed an Arts Coordinator. Arts Coordinators were most commonly managed by the school Principal or the overall Head of Arts. However, some were managed by the HOD of one Arts discipline, which suggests a possible alignment with that discipline. The Arts Coordinators' role generally involved all or most aspects of the role identified by the Ministry of Education. Notably, however, HODs reported that they were not always consulted regarding the role of the Arts Coordinators in the school.

Data from the analysis of the opportunities/activities described in the milestone reports and the Principal questionnaire reveal that Dance and Visual Arts have received less focus through the Project than have Drama and Music. This may reflect either the opportunities/activities available to schools or schools' choice to give greater focus to Drama and Music.

The data from the milestone reports suggest that, in the main, the Project supports large-scale initiatives, such as school productions and performances, together with arts-related competitions. The types of opportunities/activities offered under Visual Arts, however, showed a different pattern, with exhibitions - either external or of the students' own artwork - the main focus. Reports from the responding HODs/ Teachers in Charge show that, over all the Arts disciplines, three of the four most commonly offered types of opportunities/activities involved students viewing a performance, an exhibition, or a workshop or demonstration by dancers, actors, musicians, or artists. The most common type of opportunity/activity that involved the students themselves was a performance, production, or exhibition of their work, based at the school.

Half of the responding HODs reported either that their school did not have an Arts policy for involving artists/performers in classroom opportunities, or that they did not know whether such a policy existed in their school. However, where such an Arts policy existed, the majority of HODs had had input into the policy.

In making decisions about what opportunities/activities to provide to students, Arts Coordinators and HODs/Teachers in Charge of the Arts used information from a variety of sources. Information from the National Secondary Schools Arts Coordinators was a key source of information for Arts Coordinators but was used less frequently by HODs. While the Arts Coordinator was a major source of information for HODs, HODs also heavily used their contacts in the Arts community.

A small majority of HODs assessed students' needs when deciding which opportunities/activities to provide to students, and two-thirds of HODs focused on particular learning objectives "most of the time" or "always". However, given that the Secondary Schools Arts Coordinators Project relates specifically to the curriculum, these proportions might be expected to be higher.

For many of the opportunities/activities described in the milestone reports, no Achievement Objective information (strand and level) was identified. This may indicate that some schools are taking an opportunistic approach to the Project - identifying what opportunities/activities are available and fitting these into the classroom programme, rather than planning opportunities/activities around the classroom programme needs.

Effectiveness of Strand 1 of the Project

Overall, the generally high level of agreement among Principals, HODs/Teachers in Charge, and Arts Coordinators with each of a number of statements presented indicates that respondents believed that the Project was supporting implementation of the Arts curriculum in the school. In particular, respondents consistently reported that involvement in the Project has broadened the range of Arts-related opportunities offered in the school or has provided a range of Arts-related opportunities.

However, the level of HOD/Teacher in Charge agreement with statements about the Project supporting the implementation of the Arts curriculum varied according to how general the statement was and whether it related to supporting them personally to implement the Arts curriculum. For example, HODs almost unanimously supported the statement that involvement in the Project had been worthwhile in supporting the Arts curriculum; however, agreement with the statements that involvement in the Project had helped HODs to teach the relevant component of the Arts curriculum and that the Arts Coordinator had assisted them by providing the information needed to support the curriculum was not as strong.

Principals, HODs/Teachers in Charge, and Arts Coordinators predominantly believed that having an Arts Coordinator in the school had freed up Arts teachers and/or reduced their workload. A less clear-cut response from HODs/Teachers in Charge, however, suggests that the degree to which Arts Coordinators have reduced the workload of Arts teachers may vary from school to school.

HODs/Teachers in Charge reported using a mix of assessment processes to evaluate each opportunity/activity provided through the Project, in terms of gains in student achievement, knowledge, skills, and understanding. However, respondents most often reported using an assessment process that evaluated the appropriateness of the opportunity/activity in meeting specified curriculum objectives.

The large majority of Principals and HODs/Teachers in Charge agreed that the Project has increased the range of students in the school exposed to the Arts or has exposed students to new Arts-related skills or techniques. HODs/ Teachers in Charge were less likely to agree that Project opportunities/activities had fostered student awareness and appreciation of Māori art forms and that student understanding of the relevant Arts discipline in relation to diverse cultures had been promoted.

Data on student outcomes from the survey reflect the opinions of those surveyed rather than actual data on any learning outcome gains for students. The survey respondents were generally of the view that gains had been made in student learning outcomes. However, this should be considered in light of the fact that little student outcome information was provided via the milestone reports and that respondents' most frequent method of assessing gains in student outcomes as a result of the Project was to evaluate the appropriateness of the opportunity/activity provided in meeting specified curriculum objectives.

Arts Coordinators and HODs/Teachers in Charge were consistent in their view that the Project had assisted schools to develop or strengthen their links with the Arts community.

A high level of agreement was recorded by Principals and Arts Coordinators about general benefits of the Project for the school. It should be borne in mind, however, that the Arts Coordinators' perspective may be influenced by such factors as perceived benefits to themselves and a desire to retain their position.

Effectiveness of National Secondary Schools Arts Coordinators (Strand 2)

Arts Coordinators made extensive use of the information provided by the National Secondary Schools Arts Coordinators (NSSACs); however, Heads of Department made less use of these materials, perhaps relying on the Arts Coordinator to undertake this role. Specifically, the large majority of Arts Coordinators, but only two-fifths of HODs/Teachers in Charge, had accessed at least one of the NSSAC web sites. However, half of the HODs/Teachers in Charge had received hard copy information from at least one of the NSSACs.

Overall, Arts Coordinators and HODs/Teachers in Charge found the NSSAC web sites to be user-friendly. Also, about half of the respondents found the web sites and/or hard copy information they had accessed to be useful.

To gauge the appropriateness of the information provided by the NSSACs, Arts Coordinators and HODs/Teachers in Charge were asked to rate the extent to which the NSSAC web sites and/or hard copy information they had accessed had provided information that met their various information needs. Generally, for each of the information needs listed in the questionnaire, HODs/Teachers in Charge rated the extent to which their needs were met more highly than did Arts Coordinators. This may reflect differences in the ways HODs and Arts Coordinators used the information received. Results regarding specific assistance to rural schools in coordinating tours and visits suggest that the NSSACs provided such assistance on a limited basis, although Arts Coordinators and HODs/Teachers in Charge had differing views. This does not necessarily mean, however, that the NSSACs did not help rural schools in other ways.

Resourcing of the Project

As reported earlier, 1,024 opportunities/activities were described in the milestone reports to the Ministry of Education that were analysed by the Evaluation Team. Although the cost of each opportunity/activity described in the milestone reports was either not specified or unclear in nearly two-fifths of cases, where it was specified, the cost of each opportunity/activity varied considerably - from $0 to $8,000.

The average (mean) cost was $432 per opportunity/activity, while the median was $171. A small proportion (3%) of the opportunities/activities listed in the milestone reports were funded by students.

The level of funding given to schools involved in the Secondary Schools Arts Coordinators Project varies according to the size of the school rather than the cost of the opportunities/activities they are planning. The amounts of funding allocated to the 309 individual schools involved in the Project in 2003 ranged from $2,096 to $11,916. The mean amount allocated per school was $4,463; similarly, the median was $4,066. The total amount of funds allocated from the Project to schools at the time of the Evaluation in 2003 was $1,378,963.

The Ministry of Education has the expectation that schools will supplement the funding provided through the Project with funding or resources from other sources. The most common types of resourcing that responding Principals said their school would contribute in 2003 (over and above that provided through the Project) were equipment/materials for use in Project opportunities/activities, school funds towards Project opportunities/activities, and workspace/computer for the Arts Coordinator. Nearly half of the schools also contributed towards wages for the Arts Coordinator.

Effectiveness of milestone reporting

As a condition of receiving the funding as part of the Project, schools provide two milestone reports each year to the Ministry of Education. Arts Coordinators report significant involvement in the preparation of milestone reports for the Secondary Schools Arts Coordinators Project. Two-thirds of surveyed Arts Coordinators indicated that they write all or most of the report, and an additional one-sixth write parts of it.

Given that the milestone reports demand some level of curriculum knowledge, the Ministry expected that the Arts Coordinator would complete the reports after pulling together the information required from the Arts HODs at the school. It appears, however, that not all schools are doing this, as only half of HODs/Teachers in Charge indicated that they were involved in the completion of the milestone reports. This proportion indicates that some HODs may not be taking responsibility for providing the data for milestone reporting.

Ultimately, the milestone reports are intended to allow the Ministry to gauge, on an ongoing basis, the effectiveness of the Project in improving student outcomes in the Arts. Of interest to the Evaluation, therefore, is whether schools have provided the required information on the milestone reporting template supplied. Our analysis shows that reporting in each of the milestone reporting areas varies. The most accurate information provided relates to which Arts discipline is covered by the opportunity/activity and the type(s) of opportunities/activities offered - each reported on in at least 94% of cases. At the other extreme, there was a considerable degree of non-reporting on curriculum achievement outcomes for students (only 26% reported) and curriculum level of the opportunity/activity (47% reported).

One interpretation of these findings is that the Arts teachers who are completing the milestone reports may not be linking the opportunities/activities offered to their classroom programmes. However, considering these findings in light of the results from the survey of schools suggests that the main reason for variability and lack of clarity in the reporting of curriculum-based information may be the considerable involvement of Arts Coordinators in completing the milestone reports.

Of most concern here is the lack of information provided in the milestone reports on learning outcomes for students. It is possible that the way this information is asked for and described in the milestone report template (and the exemplar provided) needs to be clarified to assist the staff who complete the forms.

Suggestions for future improvements to the Project

The most common suggestions from Arts Coordinators and HODs/Teachers in Charge for improvements to the Secondary Schools Arts Coordinators Project were for more time or hours to be allocated to the Arts Coordinator and/or more funding to be given to support the role. Because the amount of funding allocated to schools through the Project is not large (a mean of $4,463 for the year), schools have to choose between using the money to cover a small number of hours for the Arts Coordinator, using it to fund opportunities/activities, and using it for a combination of the two. Because of this, although the Ministry expects the funding to cover all disciplines, it is possible that some schools may choose to focus the funding on certain Arts disciplines and/or to use the funding for only part of the year.

Comments regarding the level of funding and the other suggestions for improvement are made in the context of general overall support and appreciation for the Project among survey respondents.