New Zealand Alumni Survey: Experiences, Attitudes and Engagement

About the Survey

Survey Scope and Design

Survey Scope and Reach

The goal of this survey was to investigate alumni’s attitudes and expectations vis-à-vis their alma mater and New Zealand itself. Designed to serve as an assessment tool for alumni’s experiences, connections, communication preferences, and engagement levels, the survey focused on alumni living outside of New Zealand.

The survey was sent out to alumni from all eight of New Zealand’s state-funded universities: the Auckland University of Technology, Lincoln University, Massey University, the University of Auckland, the University of Canterbury, the University of Otago, the University of Waikato and the Victoria University of Wellington. Not only did its comprehensive nature result in it being the first ever nation-wide alumni survey, but it also gives insights into global alumni perspectives despite its national scope.¹

The survey was conducted in a web-based format. Despite the unavoidable instances of respondent self-selection – e.g.; a bias toward alumni who made their e-mail address available to their alma mater, and alumni with Internet access, etc. – utilizing an online format proved to be the most sensible as well as cost effective way to generate a large number of responses from New Zealand universities’ globally dispersed diverse alumni body.

In total, more than 17,500 alumni were invited to participate in the survey.  With 3,417 responses (3,210 qualified responses after adjusting for residence outside New Zealand), the overall response rate of 19.5% was significantly higher than expected; similar online surveys often attract response rates of 5-15%.

Given the fact that a larger respondent pool increases precision only to a limited extent – and eventually not at all – large alumni programs like the University of Auckland’s were requested to downsample their alumni pool before sending out invitations. Without such a step it would have been easily possible to reach 5,000 or more survey responses.  

Graph 1: Survey Responses

Notes: Survey Responses by University (Total and Rate). All Alumni: 3,417.
Source: New Zealand International Alumni Survey.

Four universities achieved unusually high response rates of more than 20 percent. In the case of the University of Otago, as many as 27.8 percent of survey invitees chose to participate. Given the remarkable willingness of alumni of New Zealand universities to take part in this survey, response rates under 10 percent have to be considered as an outlier.

Survey Design

The survey was conceived as a customized and institution-specific online survey from the outset.  A web-based survey format has several advantages: Unlike phone interviews, international web-based surveys are not subject to time zones and they are inexpensive.  In addition, web-based surveys have a truly global reach (if desired) and invitees are able to decide when they want to participate.  Universities were provided with sample invitation texts and the URL (website address) for their respective survey. 

The survey comprised twenty questions: Six mandatory questions concerning alumni demographic information, eleven (close-ended) multiple choice questions focusing on alumni preferences and behaviour, two open-ended questions regarding scholarships and general comments, and one hybrid question (Question 10) which asked alumni to choose from a list of answering options while also allowing them to contribute personal commentary.

With the exception of Questions 18 and 19, all close-ended questions provided respondents with the opportunity to opt out by selecting “Other”, “Don’t know/Does not apply”, etc. Respondents had the option to skip any question starting with Question 7.

The survey’s multiple choice questions were presented in two formats: For one, questions with nominal answers (i.e. either/or – questions with answering options that are distinct from one another), and second scaled response question which allowed respondents to specify the intensity of their opinions (e.g., very positive, positive, neutral, etc.). 

From a methodological standpoint, allowing for scaled responses can be of great interpretive assistance because oftentimes answers derived from dichotomous questions (i.e. Yes/No or Agree/Disagree) fail to communicate a respondent’s concise feelings about a certain issue.

Among the survey’s voluntary close-ended questions, i.e. Questions 8 through 19, all but three were scaled response questions. The others, i.e. Questions 10, 18 and 19, were general multiple choice questions which asked respondents to select one or more options from a list of distinct nominal answers. For example, alumni were asked to state their concrete expectations with regards to a regional alumni event (Question 19): Do they prefer to socialize with fellow alumni? Do they want to listen to a lecture? Do they want to meet up with university leaders? etc. In this case, results were nominal, i.e. answers were neither related to one another nor were they gradable (nominal data: either/or distinctions).

After devising the initial draft for the survey, ICG worked with representatives from both the New Zealand Ministry of Education and all eight New Zealand universities to optimize questions’ focus, wording, and response option mix, as well as to detect potential sources of error.

In addition, substantial efforts were made to harmonize different naming conventions (e.g., for degrees or academic units) as much as possible to allow for a comprehensive roll-up analysis of the eventual eight individual surveys². 

Ultimately, though, a number of questions remained subject to customization in order to provide alumni with specific touch points.  For example, instead of using “university magazine” the actual title was provided. By balancing out overall structural harmonization with the customization of specific response options a high level of comparability managed to be retained. 

The survey was designed to be completed in less than twelve minutes, which the vast majority of respondents achieved.

Survey Execution

After a two week-long revision process performed in collaboration with both the New Zealand Ministry of Education and all eight universities, the survey’s final draft was conveyed to the universities who were in charge of the alumni invitation process.

In order to gear the survey toward the desired target population – alumni living outside of New Zealand – the universities made an effort to exclude New Zealand residents from their list of survey invitees. 

In a second step, in order to ensure a high level of representativeness, some universities were advised to either e-mail the survey invitation to all alumni (who qualified), or to first run a random sample (at agreed-on sample levels) of the existing list of qualifying alumni.

Universities’ approaches effectively depended on the number of international alumni with e-mail addresses: With 1,342 qualifying alumni, the Auckland University of Technology was requested to contact all alumni, whereas the University of Otago was asked to sample 2,710 alumni out of its qualified pool of 6,716 international alumni.  

Overall, out of 17,541 total survey invitation e-mails, 1,180 e-mails (6.7%) bounced back due to incorrect or outdated e-mail addresses and/or other technical problems. In total, 3,417 alumni responded to the survey of which 3,210 responses were deemed qualified while 207 responses were disqualified as ineligible owing to respondents residing in New Zealand; nearly half of the disqualified responses were caused by an invitation list error committed by a single university. The data set this report draws on therefore is based on a total response pool of 3,210 international alumni, split into former international students (1,590) and New Zealand expatriates (1,620) ³.

Alumni offices were given the liberty to select the precise date of sending out the e-mail invitation to their respective alumni in order to allow them to embed this invitation into other programming measures.  Therefore, both the survey start dates and end dates differed between universities. Common to all surveys was the duration of exactly three weeks.  On 8 May 2009, the last remaining open survey was closed.

Methodological Considerations

Self-Selection Bias

One of the most challenging topics in survey research is the problem of sampling and selection bias. A biased sample will lead to biased results unless such a bias is expertly accounted for and/or de-layered. And though it is next to impossible to exclude all potentially bias producing methodological distortions, being aware of this bias both from analyst’s as well as a report reader’s perspective should increase confidence in a survey’s results.

In terms of self-selection dynamics and the inherent biases, it should be noted that the respondent pool only encompasses responses fulfilling all below selection criteria:

• Alumni living outside of New Zealand (not related to survey’s web-based nature)
• Alumni whose e-mail addresses were retrievable in a university’s database
• Alumni who actually received the e-mail invitation (i.e. a valid e-mail address)
• Those alumni who opened the e-mail
• Those alumni who made an effort to participate in the survey

A brief case study of the Victoria University of Wellington (VUW) serves to illustrate the above issue of self-selection bias:

• VUW has recorded 74,980 alumni in its database. These alumni constitute named alumni, i.e. they merely represent a part of the total pool of all students that ever graduated from and/or attended VUW. The exact size of the total alumni pool is unknown
• Of these 74,980 alumni, 7,855 alumni, or 10.5 percent, are currently residing outside of New Zealand (assuming the database entries are up-to-date)
• Of these 7,855 alumni, a little less than half, i.e. 3,704 or 47.2 percent, have their e-mail addresses captured in VUW’s alumni database
• Of these 3,704 alumni, a random sample of 2,116 alumni was invited to participate in the survey
• 161 invitation e-mails bounced back due to incorrect addresses and/or other technical problems, resulting in 1,955 alumni being reached
• Of these 1,955 alumni, a total of 345 or 17.6 percent, participated in the survey
• Another 27 responses had to be disqualified because the respondents did no longer live outside of New Zealand
• Ultimately, 318 VUW alumni survey responses qualified for the data analysis

Taking the various above referenced instances of (self-) selection bias into consideration, the explanatory picture emerging from this report’s data analysis will be characterized as much by the limitations of the data utilized as by the this survey’s unique scope: As the first ever nation-wide alumni survey this report provides an unprecedented opportunity to assess New Zealand universities’ global alumni footprint and gain a comprehensive picture of the likes and dislikes of alumni of all backgrounds.

Observation Technique

This survey constitutes a one-off, backwards as well as forward looking observation of international alumni living outside of New  Zealand. The advantage of such a broad observation time frame is that it allows respondents to weigh in on their diverse experiences as students and then alumni, and to evaluate these experiences in the context of their expectations and connections to their alma mater and New Zealand.

A disadvantage of the survey’s one-off character is that it essentially only offers a snapshot of alumni attitudes and behaviour as of April to May 2009. Only a longitudinal study in the form of repeated observations (i.e. surveys) would allow for tracking deviations in alumni attitudes and behaviour over time.

Diverse Alumni Pool

The composition of survey participants in terms of their demographic and educational backgrounds was highly diverse along multiple factors, including age, gender, subject studied, nationality, etc.  For example, some survey participants’ student days date back to the early 1950s, whereas others are recent graduates.

These factors – labelled in a simplifying sense “demographic” factors – in themselves as well as in the various possible combinations were expected to hold a significant influence on respondents’ preferences and experiences.  Therefore, the first six questions were mandatory close-ended questions asking respondents to specify their current country of residence, their nationality, their age, their gender, the school/department/faculty/college they attended while enrolled at a New Zealand university, and any degree obtained from New Zealand institutions.  

Though questions were devised to be as straightforward as possible, at times their close-ended character could not account for a variety of complex real-world arrangements: Some alumni held dual citizenship, a few were reluctant to state their gender, and others were affiliated with academic units that no longer exist, and thus struggled with identifying appropriate responses.

Since the first six demographic questions in the survey were mandatory (i.e. responses were required in order to be able to move to the next question), respondents were offered quasi-opt out options by choosing “Other,” or “Do not want to share”.  In another question, clear guidance such as in the case of country of residence was provided by asking for an alum’s country of primary residence. This way, respondents with multiple residences received clear instructions on which country to choose. In the case of nationality, alumni were purposefully left to choose which nationality they wanted to identify in case of dual citizenship.

Given the highly diverse alumni body, all subsequent questions (Questions 7 through 20) were interpreted in the context of key demographics: While this report lays primary emphasis on the differences between New Zealand expatriates and international alumni, other variables such as alumni age or school affiliation were also taken into consideration, especially with regards to event participation and preferences as well as technology usage.  

Close-Ended Multiple Choice Questions

Questions 8 through 11 asked alumni to look back and evaluate their experiences as students in New Zealand. After the introductory set of compulsory demographic questions, this section featured entry-level questions which, given their generic nature, were designed to prepare respondents for more specific questions later on.  

In the case of Questions 8 and 9, answer options were tailored to the respective question: Both questions required alumni to rate their educational and student service experience as either “very positive,” “positive,” “neutral,” “negative,” “very negative” or “Don’t know/Does not apply”. Question 11 inquired about New Zealand’s hospitality (as perceived by alumni) and offered five reply options, ranging from “definitely” to “definitely not”, while allowing respondents to opt for answering “Don’t know/Does not apply”, or outright skipping the question.  

With the exception of Question 10, response options were scaled with the intent to capture the intensity of respondents’ answers. From a methodological standpoint, this can be of great importance because oftentimes answers derived from dichotomous questions (i.e. Yes/No, Agree/Disagree) fail to communicate respondents’ concise and/or granular feelings about an issue. Scaled response questions by contrast offer a wider range of answers which provided respondents with the opportunity to express themselves in a more nuanced way.

The multiple choice format of Question 10 (“As a student in New Zealand, who was an important part of your social life?”) resembled that of Question 5 (degrees obtained in New Zealand) in the sense that alumni were asked to check all answers that applied. In addition, Question 10 included an open-ended part which invited alumni to personally name all of the groups of people they deemed important.

Questions 12 through 15 sought to find out about both the extent and the nature of alumni’s connection with their alma mater. Following a series of backward-looking questions (i.e. Questions 8 through 11), these questions assessed alumni’s communication habits and their perceived connection to both their alma mater and New  Zealand.

The rationale behind these questions was to explore alumni’s connectedness in an abstract way (i.e. overall connection with their alma mater), as well as in the form of concrete reciprocal communication efforts (i.e. “Have you attended an alumni event?” / ”Do you prefer receiving updates via e-mail?”).   

The third section of the survey – Questions 16 through 19 – asked alumni to express their availability and willingness to support their alma mater in the future. Questions 16 and 17 asked alumni whether, and in what way, they could imagine supporting their alma mater or whether they were willing to participate in a New Zealand-themed event.

Though both questions followed a scaled-response logic, their dichotomous character (possible answers: “Yes,” “Maybe,” “No,” “Don’t know/Does not apply”) was intentionally employed in order to yield more tangible results for alumni programming efforts. It should be noted that when asking alumni for support, probing for overly high granular responses do not only have the potential to irritate respondents, but also to result in data which cannot be analyzed with clarity.

Questions 18 and 19 were close-ended multiple choice questions which asked alumni to identify their expectations regarding two types of events: New Zealand-themed events and regional alumni events. The absence of opt-out answers (e.g. “Other”) can be explained by one of the most fundamental principles of alumni programming, namely to avoid creating false expectations.

Each of the questions’ eight answering options had been previously discussed and agreed upon by the universities and the New Zealand Ministry of Education. Allowing alumni to deviate from these options would have sent the wrong message by implying the possibility to generate a wish list of sorts.

Open-Ended Questions

The survey featured two open-ended questions: Question 7 inquired about scholarships awarded by the New Zealand government and Question 20 called for general comments. While Question 20 was intended to seek colour commentary, Question 7 was meant to assess the ability of New Zealand government scholarships to drive international alumni affiliation with the country of New Zealand.

Though several international alumni received scholarships from their alma mater, the heterogeneous nature of these awards nullified the feasibility of a meaningful analysis.  Therefore, the report focused on the two biggest New Zealand sponsors for international alumni in this survey: the New Zealand Agency for International Development (NZAID), and New Zealand Official Development Assistance (NZODA).

To ascertain about the number of relevant scholarship recipients, all 882 entries for Question 7 were manually reviewed for NZAID/NZODA properties. In total, 126 scholarship recipients, or 7.9 percent of all former international students, were identified as having received financial assistance through NZAID/NZODA programmes.

To contrast NZAID/NZODA scholarship recipients’ opinion of New Zealand with that of alumni who received scholarships from institutions abroad as well as those who did not receive a scholarship, all 126 alumni were matched with Question 15 which inquired about respondents’ perceived connection to New Zealand. Ultimately, the comparison confirmed the widely shared assumption that scholarships have a notable impact on affinity levels.

Notes on the Report’s Data Analysis

Differentiating International Alumni and New Zealand Expatriates

While this report largely splits its analysis into either former international students (further on referred to as international alumni) or New Zealand expatriates (referred to as New Zealand alumni), the survey itself was designed without betraying any focus or specific verbiage which would have indicated to either that they were in any way targeted.  To the contrary, slight ambiguities in wording were accepted during the survey design phase in order to arrive at a common, equitable design. 

This proved to be fortuitous since eventual returns amounted to a relatively even split between 1,590 international student alumni and 1,620 New Zealand alumni (qualified responses).  Both groups are of particular interest to New Zealand universities’ alumni programming owing to their – assumed – distinct characteristics and preferences.

Outliers, Lacking Responses, and Non-standard Behaviour

When dealing with large complex datasets which contain small sub-samples, the role of outliers has to be taken into consideration.  In this regard, both Lincoln and Massey Universities showed significant deviations from the average response patterns.

With only 33 qualified answers and a response rate of 4.0%, Lincoln University’s survey responses just cleared the threshold of statistical relevance. However, given distribution patterns within the respondent pool it is suggested that the survey results not be utilized, owing to their lack of representativeness.

Massey University recorded the second highest number of survey respondents (708 qualified responses). A sub-group of this considerable number of respondents were alumni who had taken part in Massey’s distance education program (i.e. extramural students).  Their experiences and attitudes therefore differed from those of on-campus students.

As a result, the question-to-question response rates of these students jumped up and down in a unique, no-standard manner – driven by these students opting out from answering questions which did not relate their experiences.  Typically, question response rates do not recover from such an opt-out behaviour.  Yet these Massey alumni behaved differently by continuing the survey at high rates. 

Demographic Data Discussion

Questions 1 to 6 were directed at discovering alumni demographics information: Country of residence, nationality, age, gender, the academic unit alumni attended while enrolled at a New Zealand university, and the degrees alumni obtained from New Zealand institutions.

Questions 1 and 2 offered the widest range of multiple choice answers – countries of primary residence (Question 1) and nationality (Question 2) offered 31 answer choices each (30 countries to select from and “other”).  The report’s data discussion focused on the eleven most frequently selected countries. Since these eleven countries accounted for 83.5% of respondents for Question 1 and 82.6% for Question 2, the remaining 21 countries were aggregated to simplify the graphic data display.

Results from Question 2 allowed for the eventual disaggregation of former internationals students (1,590) and New Zealand expatriates (1,620).  Complex scenarios such as dual citizenship or changes in citizenship were not taken into consideration owing to limitations of the survey format.

Devising the response options for Question 5 was rather difficult owing to the multitude of degree categories and naming conventions not only between universities, but also within universities over time.  The discussion of degree attainment thus must be considered in this challenging methodological context.

Question 6 asked survey respondents to name the academic unit (school, college, faculty, college) they attended while enrolled at a New  Zealand university. Similar to Question 5, structures, naming conventions, and historical (in-) consistencies posed challenges.  In order to allow for a standardized data discussion, further roll-up scenarios were necessary. For example, the University of Canterbury’s College of Arts, Faculty of Arts, Faculty of Music & Fine Arts and the University  of Otago’s Division of Humanities Departments were both aggregated under the Arts and Humanities umbrella.

Open-Ended Questions

The survey contained two questions (Questions 7 and 20) which invited responses in an open-ended, free text format.  These responses necessitated a different analytical approach relative to the analysis of the other 18 questions.

Question 7 probed for scholarships awarded by the New Zealand Government. The 882 responses for this question were manually reviewed for evidence of NZAID and NZODA funding. Eventually, 126 scholarship recipients were identified as having received such financial assistance. 

The statistical validity of responses to this question, however, is not as high as the validity of response to Questions 1 to 6.  Question 7 was not a mandatory response question and the large amount of irrelevant, ambiguous, mislabelled, or unidentifiable responses provided by survey respondents proved challenging.

The survey’s final question, Question 20, invited survey respondents to share any comments they deemed relevant. Responses predictably touched on a vast array of subjects. Despite the wide range of responses, a high level clustering of responses was attempted. However, no statistical analysis was performed.

Close-Ended Questions

Overall, the survey featured eleven close-ended multiple choice questions. Two forms of multiple choice questions were utilized: First, regular questions such as an either/or question with answering options that were distinct from one another; and second, a scaled response question which allowed respondents to specify the intensity of their opinions (e.g. “very positive”, “positive”, “neutral”, etc.).

Some questions featured multiple sub-questions which probed for further details on alumni’s readiness to connect with and support their alma mater. Since not all sub-questions within a question were thematically aligned (mostly a function of listing a large number of sub-questions), the data analysis process made an effort to group sub-questions by relevant theme and, if possible, to filter salient messages.

All voluntary close-ended questions (Questions 8-19) offered opt-out scenarios, with the exception of Questions 18 and 19. Both questions were intended to identify alumni’s expectations regarding two types of events: New Zealand-themed events and regional alumni events. The absence of opt-out and/or free text answers was driven by the need to avoid creating false expectations, i.e. alumni assuming by stating specific demands that such demands would become a reality.

Visualizations and Graphic Displays

Nearly all data discussion in this report is augmented by a graph depicting the discussed data, or a summary thereof. The choice of charts – e.g.; bar charts, pie charts, etc. – was driven by the desire to select the format easiest to comprehend. Charts represent relative values such as percentages, not total numbers.

The colour coding in the graphs follows a predetermined logic: Shades of grey, blue, green and red are applied in a certain order, typically moving from the left to the right or the bottom to the top. Whenever colour-coded formats yielded suboptimal results in terms of illustration and legibility, they were omitted.