4 - Data and method

This study uses unit record data of around 3,200 university staff who participated in both the 2003 and 2006 PBRF Quality Evaluations. This data was supplied by the Tertiary Education Commission.³ The staff examined include those who were employed at universities in the July 2006 Quality Evaluation and who held the academic rank of lecturer, senior lecturer or associate professor in the 2003 Quality Evaluation.⁴

The analysis is restricted to those who participated in both the 2003 and 2006 Quality Evaluations, given that the academic rank of these staff in both 2003 and 2006 is required to work out if they were promoted. Also note that the staff members included in the analysis are limited to those who had evidence portfolios examined in the 2003 Quality Evaluation. Only those staff who submitted evidence portfolios provided details of their research output.

As there are two possible outcomes for staff, ie, they were or were not promoted between the 2003 and 2006 Quality Evaluations, logistic regression is used to analyse whether various staff characteristics are associated with the likelihood of staff being promoted. The advantage of using regression analysis for a study of this kind is that it holds other factors constant, while examining the association between the likelihood of being promoted and the factor of interest.

The logistic regression model takes the following form where P_Promoted is the probability that a staff member was promoted:

where the dependent variable is the natural logarithm of the odds of a staff member being promoted, X is a vector of explanatory variables, β is the coefficient of the explanatory variables in logit form, μ is an error term and i = 1 to n staff members.

The variables included in the regression model (X_i) are discussed below.

2003 PBRF QUALITY CATEGORY

The quality category assigned to staff in the 2003 Quality Evaluation is included in the model. This is the result that would have potentially impacted on their chance of promotion in the following three years. The quality categories awarded in the 2003 Quality Evaluation ranged from an ‘A’ to an ‘R’, with a ‘A’ representing the highest level of peer-assessed research quality. In general terms, an ‘A’ quality category indicated that a researcher was producing research that was esteemed internationally. A ‘B’ quality category indicated that research was recognised at a national level for its quality and a ‘C’ quality category indicated the quality of research was recognised at a local level within their tertiary education organisation. An ‘R’ quality category indicates the staff have been assessed by the panel as not having met the requirements of a ‘C’ quality category. The reference category in the regression analysis is a ‘C’ quality category.

If the quality of research does have a positive association with the likelihood of promotion, then we would expect to see staff who received ‘A’ and ‘B’ quality categories in the 2003 Quality Evaluation having a greater chance of being promoted than staff who received a ‘C’ or ‘R’ quality category.

RESEARCH OUTPUT

The number of research outputs submitted in evidence portfolios in the 2003 Quality Evaluation is included as a proxy for the quantity of the research output of staff. If the quantity of research output influences promotion, we would expect to see a positive association between the quantity of research output and the likelihood of promotion.

Some caution should be used as this variable does not capture all research output. Rather, it represents all the research outputs (to a maximum of 50) that staff produced in the assessment window between 1998 and 2002 that they considered to be ‘quality’ outputs and so would enhance their evidence portfolios. Also, this count of research outputs does not weight for type of publication – for example, a book receives the same weighting as a journal article. So is a somewhat crude measure of quantity.

AGE

The age of staff as at June 2006 is included as an explanatory variable in the model. Also included in the regression model is the square of age, given that a non-linear relationship between age and likelihood of promotion is expected. In other words, the likelihood of promotion may well increase initially with age, but then decline.

GENDER

Also included in the regression model is the gender of staff. The reference category is males.

EXPERIENCE

This variable is included in the model to control for the experience that an academic has. If a staff member was employed for the first time in an academic role after 1 Jan 2000 (either in NZ or overseas) or required to do degree teaching for the first time from this point, they could be nominated by their tertiary education organisation as a ‘new and emerging’ researcher. Obviously, staff who are relatively inexperienced may be less likely to be promoted in the short term.

This variable takes a value of 1 if the staff member was identified as new and emerging and a value of 0 if they were not.

INSTITUTION

The institution staff were employed at in 2006 is included as an explanatory variable with multiple categories in the regression analysis. Different universities may have different promotional processes and so the likelihood of promotion may be affected. The University of Auckland is the reference category.

PBRF SUBJECT PANEL

The broad PBRF subject panels are also included as a variable with multiple categories. The reference category is Biological Sciences.

2003 JOB TITLE

This is the academic rank of a staff member at the time of the 2003 Quality Evaluation. This variable includes all lecturers, senior lecturers and associate professors. The analysis excludes professors (as they are already at the highest rank) and those whose 2003 job titles mean that they can’t be assigned to a rank on the academic career path. Given that there are substantially fewer associate professors and professors than senior lecturers and lecturers, one might expect that the likelihood of promotion diminishes with higher academic status/rank.

The results of the logistic regression analysis are presented in the form of odds ratios. An odds ratio is the odds of an event happening divided by the odds of the opposite event happening. For example, suppose that 400 senior lecturers were promoted and 200 were not. The odds of a senior lecturer being promoted are 400/200 = 2, or 2 to 1. In other words, the chances of a senior lecturer being promoted are reasonably good. To give another example, suppose that 500 lecturers were promoted and 1,000 were not. The odds of a lecturer being promoted would be 500/1,000 = 0.5, or 1 to 2. The chances of their being promoted are therefore significantly lower than for senior lecturers.

For continuous explanatory variables (AGE and RESEARCH OUTPUT), an odds ratio of greater than 1 indicates a higher likelihood of achieving a higher quality category as the value of the explanatory variable increases and an odds ratio less than 1 indicates a lower likelihood.

For categorical explanatory variables, the odds ratio compares the likelihood of achieving a higher quality category compared with the reference category. An odds ratio greater than 1 indicates a higher likelihood of achieving a higher quality category compared with the reference group, while an odds ratio of less than 1 indicates a lower likelihood.

Footnotes

3. See http://www.tec.govt.nz/upload/downloads/pbrf_dataaccess_final.pdf for more information on the process for accessing information from the PBRF Quality Evaluations.
   
4. The categories used in this analysis are based on simplified job titles as defined by the Tertiary Education Commission. For more detail on how these aggregated positions were determined see Çinlar and Dowse (2008).