Summary

The primary objectives of this evaluation report were to:

• examine whether the eCDF achieved its purpose of building the e-Learning capability of the tertiary education system by investigating the identifiable outcomes of the round 1 eCDF projects including exploring factors that contributed to their success or failure
• scope the state of e-Learning capability in the tertiary sector as a whole; and
• assess, on the basis of the above, the need for further funding to improve e-Learning capability in the tertiary education sector.

In May 2003, Cabinet authorised the Tertiary Education Commission Te Amorangi Mātauranga Matua to administer a pool of funding for capability development initiatives in the tertiary education sector. One of these initiatives was the E-Learning Collaborative Development Fund (eCDF), which had the specific purpose of building the e-learning capability of the tertiary education system through a series of contestably funded collaborative projects.

Such capability building for e-learning was to be in the form of producing and/or disseminating information and communication technology "tools and knowledge" that would support tertiary education organisations' (TEOs') ability to deliver e-learning education programmes. In particular, the eCDF encouraged a consolidated approach to projects whereby TEOs would "share e-learning costs and systems rather than replicating each other's investments" and would actively collaborate with each other on projects in order to provide "demonstrable benefits" to each other and, potentially, to the sector as a whole.

The primary purpose of this evaluation was to evaluate the impact of the eCDF to inform future investment decisions on e-learning in the tertiary education sector.

The evaluation's objectives were to:

• investigate the identifiable outcomes of the round 1 eCDF projects to examine whether the eCDF is achieving its purpose of building the e-learning capability of the tertiary education system
• explore the factors that contributed to the success or failure of the round 1 eCDF-funded projects
• scope the state of e-learning capability in the tertiary sector as a whole
• assess, on the basis of all the above, the need for further funding to improve e-learning capability in the tertiary education sector.

Findings: Assessment of e-learning capability outcomes of round 1 projects

In assessing the e-learning capability outcomes of the round 1 eCDF projects, the evaluation found the following.

• All round 1 projects successfully delivered required outputs, although they varied considerably in the extent to which those outputs involved the effective sharing or dissemination of information to the wider sector.
• The most apparent and lasting capability outcomes of the round 1 projects have been the:
  • raised profile of e-learning as a significant issue for participating institutions' strategic management and planning processes
  • fostering of an active community of practice among e-learning experts and leading practitioners across the university and polytechnic sectors.
• The knowledge and use of specific eCDF projects and their outputs are significantly greater in eCDF institutions (ie. the universities and larger polytechnics) than in non-participant institutions, and are greater in the polytechnics than in other institution types.
• Neither the eCDF nor particular projects are widely known or applied in private training enterprises (PTEs), and neither the eCDF nor particular projects have any profile outside the population of providers already engaged in e-learning activity.
• Capability building through the eCDF in tertiary institutions has been greatest with regard to development in strategic policy, operational systems and technical infrastructure (especially learning management systems), and least with regard to staff development. While teaching staff report relatively high levels of uptake of professional development in technical skills in e-learning, no eCDF institution rated its teaching staff's e-learning capabilities as high overall. Staff and managers of e-learning programmes report that staff development, especially in relation to developing a better understanding of effective e-learning pedagogy, is the area of greatest need.
• eCDF projects cemented some existing partnerships among several tertiary institutions and promoted efficient common practice in them, more than it created new collaborations. Most of the organisations involved in the round 1 projects defined 'collaboration' as information sharing and an agreement to share the benefits of the projects after they had been implemented, rather than as a method for developing projects from the beginning.
• The eCDF projects have been a significant conduit for collaboration among, and building professional capability within, the specific community of e-learning managers and e-learning experts in eCDF institutions nationally. Professional development projects such as Flexible Learning Leaders in New Zealand, policy-focused projects such as the New Zealand E-Learning Quality Standards, Framework and Guidelines and the E-Learnz Collaborative Tool Box, and the three open source software platform projects (Open Source Courseware Initiative New Zealand, Open Source e-Learning Environment and Community Platform, and the E-Learning XML Editor) were the most widely reported examples of successful collaborations in this regard.
• The sustainability of projects varied largely, depending on the nature of the capability-building goals set for a given project. The most 'sustainable' of the projects were those that had a product development focus or some security
  12 Evaluation of the E-Learning Collaborative Development Fund
  of ongoing funding from other sources, such as equivalent full-time student–based funding. The least sustainable projects tended to be those with a professional development focus, where the time or financial resources for achieving a critical mass was not in place, expected alternative sources of funding were not forthcoming, or planning or resourcing for expanding the scope of the project beyond a small group of immediate beneficiaries was insufficient.
• The great majority of eCDF institutions had a well-established profile in e-learning before the projects. Consequently, the eCDF was more successful in assisting early-adopter institutions to extend their range of existing e-learning capabilities than in creating new or widespread capability in institutions with little or no e-learning experience.

Findings: Factors contributing to the success or failure of round 1 projects

The evaluation found the following were key factors contributing to the success of round 1 eCDF projects.

• The extent to which senior management in the institutions showed a genuine interest in, and commitment to, the project and e-learning as a strategic goal generally.
• The extent to which an e-learning 'champion' or small group of 'champions' was present in participating institutions and had leadership abilities.
• The extent to which the collaborative nature and purposes of the project were supported by clear thinking, planning and formal procedures, especially with regard to the parties having transparent, mutual understandings of the extent to which the project was to focus on 'collaborative effort' or 'collective benefit'.
• The extent to which a known group of collaborators worked on the project and took a team approach with technical, teaching and management staff actively working together and given plenty of opportunity for knowledge sharing.
• The extent to which goals and time-frames were realistic. Projects were successful insofar as the project goals and deliverables were aligned and compatible with the existing strategic goals of the participating institutions and based on a sound and realistic knowledge of existing e-learning capability within participating institutions. Relatively few projects were achieved, or indeed achievable, in the one-year time-frame initially designated.

Findings: State of e-learning capability across the tertiary sector

In scoping the state of e-learning capability across the tertiary sector, the evaluation found the following.

• Strategic management capability in e-learning is being built in the sector, but more in the larger tertiary education institutions (TEIs) than in the smaller TEIs, wānanga, or PTEs. E-learning capability is particularly low among PTEs. Web-enhanced and web-based courses account for about a fifth of all courses offered nationwide. Universities and the larger polytechnics dominate the provision of such courses.
• Technical infrastructure capability is being built, especially in the TEIs, but not necessarily using interoperable platforms and more in relation to transmissive, mono-media technologies than in relation to multimedia, discursive technologies.
• The full range of students is starting to use all forms of online courses in tertiary institutions, but not in every field or in identical proportions. Enrolments in online courses are increasing at a faster rate in web-based courses than in other modes of course, but these still represent a small minority of enrolments and courses overall. Pacific students are represented in all types of online learning in the same proportion as their proportion of the total student group. Māori students are represented in web-based and web-supported courses, but are under-represented in web-enhanced courses in relation to their proportion of the total student group. Most other under-represented or disadvantaged student populations tend not to enrol in online courses, but when they do it tends to be in either web-supported or fully web-based courses.
• Content capability is starting to be built, but unequally among different disciplines and fields of study. Three fields of study account for three-quarters of the online courses available to students nationwide, and significantly more courses with online components are offered in traditional academic fields than in vocational, trade or general fields of study.
• Professional e-teaching capability is being built, but slowly, and it is not widespread among staff in most tertiary institutions. Professional and pedagogical capability among teaching staff is less widespread than technical capability.
• Students across the sector are experiencing higher pass rates in web-enhanced courses and lower completion rates in fully web-based courses than in other types of course.