Executive Summary

In 2001 the Government released Export Education: A Strategic Approach (Ministry of Education, 2001a, 2001b). One of the "Key Initiatives" in this document was the development of a mandatory code of practice aimed at institutions with full fee-paying international students (IS). The Code of Practice for the Pastoral Care of International Students was subsequently developed, Gazetted and then published (Ministry of Education, 2002a). In mid-2002 a team from the University of Auckland was contracted through UniServices Ltd to evaluate the implementation of the Code to mid-2003. The team was already evaluating two professional development (PD) initiatives relating to IS programmes. While the two evaluations were 'aligned' in late 2002, mainly because the focus of the PD initiatives was increasingly on aspects of the Code, two separate reports were requested, and have been submitted (cf. Peddie et al., 2003).

Goals of the Evaluation

It is recognised by both the Ministry and the Evaluation Team that the original reporting date of June 30, 2003 meant that some aspects of the implementation of the Code would not have had sufficient time to allow for valid and reliable conclusions - especially in terms of the appeals procedures.

Nevertheless, there were some important issues which can be (at least initially) evaluated before the end of June, 2003.

The focus of this evaluation was to assess whether the introduction of a mandatory Code was making a positive contribution towards the quality of the educational experience of IS in New Zealand educational institutions. Also, is the Code making a positive contribution to the government policy set out in Export Education?

Primarily, the evaluation aimed to determine:

• What (if any) issues arose from the process undergone by all types of educational institutions applying to be signatories to the Code;
• What positive changes have so far arisen from the introduction of the Code in terms of the broad areas of:

1. Marketing

2. Orientation programmes

3. Programming/teaching-learning

4. Pastoral care

5. Homestay/accommodation provisions

6. Other factors associated with the internationalisation of institutions and their communities

There were a number of more specific evaluation questions related to these broad aims. Section One of the full report details these and offers other introductory comments.

Timeline of Events

The evaluation period began about half-way through the six months allowed (under the Code's transitional arrangements), for institutions to become signatories. Section Two of the report details a number of 'events' affecting the two linked evaluations. Of critical importance to this evaluation were:

• The date by which institutions had to be signatories to the mandatory Code (30 September, 2002);
• New legislation enabling the payment of a new "Levy" for each IS, affecting the definition of an "international student" and as a consequence requiring the registration of (mainly) language schools offering courses of less than three months (December, 2002);
• A subsequent process to revise the Code, both to ensure that it conformed with the new legislation, and to reconsider a number of areas covered by the version operating in 2002 (March to July, 2003).

Methodology

The evaluation used a form of "realist evaluation", which is outlined in Section Three. In brief, this includes consideration of:

- "Content", what the intervention - the Code - entails;

- "Context", the (political, economic, and other factors affecting the intervention;

- "Mechanisms", the attitudes and beliefs of those participating in the intervention;

- "Outcomes", but the discussion of outcomes intrinsically includes the processes used.

The procedures used were almost completely linked to the PD evaluation. Even before the Code evaluation contract was signed, much of the data-gathering had related to the Code, while subsequent interviews and other data-gathering 'visits' always covered both evaluations.

In all, over 80 data-gathering visits were made. These included:

• Thirty-eight visits to institutions across sectors and across New Zealand, in which 68 individuals supplied data face-to-face (or in a few cases, by questionnaire);
• Attendance at 13 PD seminars, in most of which material related to the Code figured prominently, and a good deal of informal discussion was also possible;
• A number of other interviews with specialists in various areas, and attendance at several seminars/other events relating to IS.

The vast majority of the institutions visited were purposively selected as representing good practice, either because their Code applications were considered "exemplary", or because regional PD facilitators identified them as worthy of a visit.

Content, Context and Mechanisms

Section Four analyses the key aspects of the intervention content, the context in which the intervention of the Code implementation took place, and the enabling and disabling 'mechanisms' (attitudes and beliefs) held by those in institutions.

By way of summary, the content was the introduction of a mandatory Code, with a fixed time frame, associated costs, and a substantial application process. The context included the policy contained in Export Education, the recent rapid rise in IS numbers against a background of global trends in this and related areas, the neo-liberal approach adopted by successive governments in recent years, funding issues, and factors such as the SARS virus and the strong Kiwi dollar.

The mechanisms were a mix of positive (enabling) and negative (disabling). The former emerged in terms of the professional and genuine desire of almost all institutions to ensure quality programmes. The latter included resentment over a centrally-imposed Code with considerable time and money compliance consequences, and some more naïve beliefs about the future - which could be characterised as, "There's no problem, they'll just keep coming".

Becoming a Signatory

The evaluation found that the process of becoming a signatory was a largely, but definitely not always, positive experience. Some primary schools with very few IS in particular resented the need to formalise in policies and other paper requirements what they thought were already good internal processes for IS. But, while the process did tend to be time-consuming, close to half of the respondents stated that being required both to review their current policies and to develop what they recognised as necessary new ones was a valuable exercise. But another large group believed that approved templates of the more "technical" policies relating to such matters as fees remission, indemnity and grievance policies should have been available on the Ministry's website from the start of the process.

The evaluation found that both the PD programmes and the work of the Code Administrator's office made very positive contributions to the process by which 940 Code signatory applications had been processed by the 'closing date' of 30 September, 2002. It might be noted that by the end of August, 2003, this number will have exceeded 1300.

The Code: Effects on Institutions and Students

Section Six covers quite a wide range of issues emerging from the evaluation.

While the full report contains the details, the following represents the key features of this section of the evaluation.

First, interview questions asked about each of the major areas of potential change as a result of the introduction of the Code (cf. Goals of the Evaluation, above).

The data suggest that 'good' institutions were affirmed in the policies they already had, while others made positive changes. Not unexpectedly, the area of teaching and learning was least affected, as the Code does not specifically address standards in this area.

Next, some "unresolved issues" are noted and briefly discussed. These were:

• An on-going concern that one Code cannot cover the differential needs of so many different sectors and programmes (a view the evaluators reject);
• A number of specific issues addressed to the Code Administrator - but many of these should be resolved by the revised Code and the new Guidelines being developed;
• The roles of other Government agencies, especially that of the Immigration Services;
• The mixed impact of the PD and the Code on regional organizations - some responded very positively, a few reportedly did not;
• The relatively frequent lack of a strategic or business plan relating to IS, notably in the school sector, but also found elsewhere;
• The need for Code provisions to be better understood by non-specialist staff, both teachers and general staff;
• Clearer information about what needs to be provided to the Code Administrator if policy and/or other changes occur within a signatory institution.

There were, next a variety of responses about how the Code should be monitored, both internally on an annual basis (as the Code requires) and externally (through a system to be developed by the Code Administrator). Very few clear patterns emerged, especially in terms of external monitoring - though the private sector saw it as something that could easily be accommodated through their current annual NZQA reviews, provided specialist staff were involved.

The effect of the Code on complaints to the International Education Appeal Authority (IEAA) was also investigated. While the number of complaints since October has more than trebled over the number received in the year 2001-2002, the evaluators agree with the IEAA and the Code Administrator that this is very likely to be because of the vastly increased publicity effected by the Code about grievance procedures. But the evaluators warn that such a trend cannot be allowed to continue, and that appropriate changes need to be made when there are several complaints about an institution.

Finally, the Ministry's handling of the Code review process to date is praised, for the 'extra' (to Code requirements) consultation offered, and the responsiveness to particular sectoral concerns.

Conclusion

Section Seven summarises some of the key issues noted here, and offers a series of recommendations. These are summarised in the next section of the report.

Overall, however, the evaluators consider that the introduction of the mandatory code has had a positive effect on programmes for IS in all sectors and throughout New Zealand.