e-Learning Maturity Model (Version Two) New Zealand tertiary institution e-Learning capability:

Informing and guiding e-Learning architectural change and development

Publication Details

This research report commissioned by the Ministry and undertaken by Victoria University of Wellington and e-Learnz Inc is a follow up to Determination of New Zealand tertiary institution e-Learning capability. It builds on this report by taking the ideas of process capability maturity and using them as a foundation for a form of benchmarking. This is explicitly intended to improve the quality of e-Learning for the benefit of students, staff and institutions.

Author(s): Dr. Stephen Marshall, University Teaching Development Centre, Victoria University of Wellington.

Date Published: July 2006
Released on Education Counts: 16 November 2009

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Executive Summary

Institutions and individuals will always have the ability to choose to invest time and other discretionary resources in innovative, unique learning and teaching projects involving the use of technology. The goal of the e-learning Maturity Model (eMM) is to assist educational leaders in changing organisational conditions so that e-learning is delivered in a sustainable and high quality fashion to as many students as possible. As noted by Fullan:

"The answer to large-scale reform is not to try to emulate the characteristics of the minority who are getting somewhere under present conditions … Rather, we must change existing conditions so that it is normal and possible for a majority of people to move forward" (Fullan, 2001, page 268)

The eMM takes the ideas of process capability maturity and uses them as a foundation for a form of benchmarking explicitly intended to improve the quality of e-learning for the benefit of students, staff and institutions. Software process maturity models have been found effective in supporting the transfer of good practice between projects (Herbsleb et al. 1994; Lawlis et al. 1995) and in answering the questions (SECAT 1998):

  • Is the organisation successful at learning from past mistakes?
  • Is it clear that the organisation is spending limited resources effectively?
  • Does everyone agree which problems within the organisation are the highest priorities?
  • Does the organisation have a clear picture of how it will improve its processes?

These are all extremely relevant questions for educational organisations to ask in the light of increasing use of technology and the consequent need for improved and evolved internal systems and procedures.

Since its initial conception (Marshall and Mitchell, 2002), the eMM has evolved with the support of colleagues in New Zealand and internationally and the financial assistance of the New Zealand Ministry of Education. This has allowed for an initial pilot assessment of capability across the New Zealand sector (Marshall, 2005) and the current work to create version two and make the eMM more usable by institutions and researchers conducting their own assessments.

The substantial revision to create version two of the eMM which is presented in this report and the associated documents has resulted in a substantially improved methodology and process set that should enable useful international comparison between New Zealand tertiary organisations and institutions based in many other countries. The international workshops and other collaborations being developed in Australia, the United Kingdom and the United States offer a significant opportunity for future improvements in both the quality of e-learning and our understanding of what drives that quality.

This summary presents the key outcomes of the current project. More detailed information and the full reports are available from the eMM project website. These documents, combined with the eMM web interface for collecting, analysing and displaying capability assessments mean that any institution can now assess their own capability and start the process of improving their own e-learning capability maturity.

Methodology Changes from Version One of the eMM

The eMM has evolved since its initial conception (Marshall and Mitchell, 2003), this evolution was informed by an initial assessment of capability in the New Zealand sector (Marshall, 2005), extensive consultation and workshops with colleagues in New Zealand, Australia and the UK, and an extensive literature review examining a wide set of heuristics, benchmarks and e-learning quality research (Marshall, 2006). As well as a significantly improved set of processes and practices, the current version of the eMM differs most significantly in the change from levels of process capability to dimensions (Marshall and Mitchell, 2006; see below) and the renaming of the Coordination process area to Support to address confusion over the separation from the Organisation process area. The description of the model that follows is of the most current version as at the publication of this report. It is anticipated that future work (see below) will see the model further evolve, particularly as the evidence base grows.

Key eMM concepts

Capability is perhaps the most important concept incorporated in the eMM. Capability, in the context of this model, refers to the ability of an institution to ensure that e-learning design, development and deployment is meeting the needs of the students, staff and institution. Critically, capability includes the ability of an institution to sustain e-learning delivery and the support of learning and teaching as demand grows and staff change. Given the large investments made in e-learning both by students and institutions it is essential that delivery be robust and reliable and able to cope with changes in the personnel involved, growth in the number of students, changes in technology requirements and skills, as well as the increasingly hostile and challenging digital communications environment.

The assessment of capability in a complex area such as e-learning is difficult and necessarily involves reducing large amounts of detail into a broader overview that supports management decision making and strategic planning. It is inevitable that this approach will fail to single out the subtle nuances and innovative work of individuals that motivate teaching staff to work on individual projects. Institutions and individuals will always have the ability to choose to invest time and other discretionary resources in innovative, unique opportunities.

The framework used in this analysis is based on the CMM (Paulk et al., 1993) and SPICE (El Emam et al., 1998; SPICE, 2002) methodologies. The underlying idea is that the ability of an institution to be effective in a particular area of work is dependent on their capability to engage in high quality processes that are reproducible and able to be sustained and built upon. The characteristics of an institution that enable high quality processes are to some extent able to be separated from the details of the actual work undertaken that will vary depending on particular circumstances. This separation means that the analysis can be done independently of the technologies selected and pedagogies applied, thus allowing for a meaningful comparison across the sector.

This separation of technology and other details means that it is potentially meaningful and useful to apply this approach to the assessment of the ability of institutions to engage in e-learning. The rapid growth in the technologies being used, the ways that they are being applied across an ever widening group of academic disciplines and the evolving skills and experience of teachers and students means that e-learning is a moving target. Any benchmarking approach that presumes particular e-learning technologies or pedagogies is unlikely to meaningfully assess a range of institutions within a single country, let alone allow for useful international collaboration and comparison, particularly over an extended period of time.
Processes

Building on the SPICE model, the eMM divides the capability of institutions to sustain and deliver e-learning into five major categories or process areas (Table 1). The key difference from the original SPICE model is the introduction of the Learning area, which replaces the Customer/Supplier area used in software engineering.

Within each of these areas are a number of processes, derived from the research literature on e-learning quality, experience from eMM assessments, and consultation with the sector through workshops. Processes define an aspect of the overall ability of institutions to perform well in the given process area, and thus in e-learning overall. The advantage of this approach is that it breaks down a complex area of institutional work into related sections that can be assessed independently and presented in a comparatively simple overview without losing the underlying detail.

Table 1: eMM version two process categories (revised from Marshall and Mitchell, 2003)
Process Category Brief Description
Learning Processes that directly impact on pedagogical aspects of e-learning
Development Processes surrounding the creation and maintenance of e-learning resources
Support Processes surrounding the oversight and management of e-learning
Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle.
Organisation Processes associated with institutional planning and management


An obvious requirement of this model is that the processes chosen are based on empirical evidence and represent 'common truths' about e-learning capability:

"are there common practices or ways of creating e-learning resources and learning environments that are accepted, useful and able to be described in a way that others can adopt them and improve their own e-learning capability?" (Marshall and Mitchell, 2003, page 4)


The processes used in version one of the eMM were developed from the 'Seven Principles' of Chickering and Gamson (1987) and 'Quality on the Line' benchmarks (IHEP 2000) as outlined in Marshall and Mitchell (2004). These had the advantage of being widely accepted as guidelines or benchmarks for e-learning delivery (Sherry, 2003), however extensive feedback through the workshops and from collaborators in New Zealand, Australia and the UK as well as the experience of applying the first version of the eMM identified a number of additional aspects of capability that needed assessment and this has now resulted in version two of the eMM (Marshall, 2006).

It is important to stress that the focus of the high level processes is on the outcomes or characteristics of successful, or capable, e-learning delivery. This means that some specific concepts, technologies or activities may appear to be absent when reading the list of processes (Table 12). Examples are the requirement that an institution have an e-learning strategy or use a single learning management system (LMS). In both of these cases the desired outcome is not that these specific things exist. In the case of an e-learning strategy the desired outcome is that an institution guides its investment and energies systematically and in line with defined learning and teaching goals that are clearly communicated to its staff and students, an e-learning strategy is a convenient and sometimes effective means to achieving this end.

The rapid uptake of LMSs by institutions internationally reflects the way that these systems provide a cost effective mechanism for administering e-learning courses and students combined with an adequate mechanism for aggregating content and communication facilities in an authenticated environment. It is widely expected that more effective ways of achieving these outcomes will become available as alternatives to the current generation of LMS software is developed.

In both of these cases the desired outcomes are sufficiently pervasive and important for institutional elearning capability that they are reflected in the assessments in multiple ways, using the eMM concepts of Dimension and Practice explained below.

Dimensions of capability

A key development that arose from the application and analysis of the first version of the eMM is that the concept of levels reused from the CMM and SPICE was unhelpful (Marshall and Mitchell, 2006). The use of levels incorrectly implies a hierarchical model of institutional improvement where capability is assessed and built in a layered and progressive manner.

The key idea underlying the dimension concept in contrast, is holistic capability. Rather than the model measuring progressive levels, it describes the capability of a process from synergistic perspectives. An organisation that has developed capability on all dimensions for all processes will be more capable than one that has not. Strong capability at particular dimensions that is not supported by capability at the other dimensions will not deliver the desired outcomes. Capability at dimensions one and two that is not supported by capability in the other dimensions will be ad-hoc, unsustainable and unresponsive to changing organisational and learner needs. Capability in dimensions three, four and five that is not complemented with similar strength at dimensions one and two will be unable to meet the desired goals and liable to fail.

In thinking about the relationship between the five dimensions it is helpful to consider them arranged as in Figure 1 below. The matrix of boxes used on the left to display summaries of process capabilities is helpful when performing comparisons within or between institutions but it can imply a hierarchical relationship that is misleading when interpreting individual process capability results.

Figure 1: eMM Process Dimensions


Each of the five dimensions reflects an aspect of process capability from a single perspective:

Dimension 1 (Delivery) is concerned with the creation and provision of process outcomes. Assessments of this dimension are aimed at determining the extent to which the process is seen to operate within the institution. It is important to emphasize that institutions can have extremely effective processes operating within this dimension, but in the absence of capability in other dimensions there is risk of failure or unsustainable delivery and wasting resources through needless duplication.
Dimension 2 (Planning) assesses the use of predefined objectives and plans in conducting the work of the process. The use of predefined plans potentially makes processes more able to be managed effectively and reproduced if successful.
Dimension 3 (Definition)  covers the use of institutionally defined and documented standards, guidelines, templates and policies during the process implementation. An institution operating effectively within this dimension has clearly defined how a given process should be performed. This does not mean that the staff of the institution follows this guidance.
Dimension 4 (Management)  is concerned with how the institution manages the process implementation and ensures the quality of the outcomes. Capability within this dimension reflects the extent of measurement and control of the outcomes and the way in which the practices of the process are performed by the staff of the institution.
Dimension 5 (Optimisation) captures the extent an institution is using formal approaches to improve capability measured within the other dimensions of this process. Capability of this dimension reflects a culture of continuous improvement.
Practices

Each process in the eMM is further broken down within each dimension into practices that define how the process outcomes might be achieved by institutions. These practices are either essential for the process to be successfully achieved or just useful in supporting the outcomes of the particular process. The practices are intended to capture the key essences of the different dimensions of the processes as a series of items, derived from empirical evidence, the e-learning literature or expert practitioner judgement, that can be assessed easily in a given institutional context.

Capability assessment criteria

The practices provide the eMM with the ability to retain the flexibility of a generic set of high level processes while still being able to incorporate specific detail to reflect the growing body of evidence describing effective use of e-learning technologies and pedagogies. An additional mechanism for ensuring that the model is able to evolve is through the assessments of capability which are made for each process on each of the five dimensions listed in Figure 1.

When conducting a process assessment each practice is rated for performance from not adequate to fully adequate (Figure 2). The ratings at each dimension are done on the basis of the evidence collected from the institution and are a combination of whether or not the practice is performed, how well it appears to be functioning, and how prevalent it appears to be. This provides a useful future-proofing mechanism as performance that is currently fully adequate may not be so in three or four years as technologies evolve and experience in e-learning grows.

Figure 2: eMM Capability Assessments (based on Marshall and Mitchell, 2003)


Once each practice has been assessed, the results are averaged as a rating for the given dimension of the process. Practices listed in bold are essential for the achievement of the process outcomes and are used primarily to make the capability assessment, with the other practices used when making a choice between two possible assessments. In the example shown in Figure 3, the assessment for dimension one would be largely adequate, although the two practices with lower assessments indicate where additional attention could usefully be focused. A purely mechanical process with a mathematical summation has been deliberately avoided in order to provide enough flexibility within the model for differences of pedagogy, technology, organisational culture and national culture.

Figure 3: Example eMM Capability Assessment


It should be noted that experience of applying this type of assessment in the field of software engineering (SEI, 2004) and with the first version of the eMM (Marshall, 2005) suggests that most, if not all, institutions initially assessed will show a low level of capability for the processes selected. This is not surprising as one of the drivers for the model in the first place is the widely held perception that e-learning could be implemented more effectively and efficiently in most institutions.

eMM Process and Practice Literature Review

The nearly one thousand practices defining capability for each of the thirty-five processes of version two of the eMM are informed by an extensive review of the research literature. This detailed review is provided in a separate document the eMM Version Two Process Guide (Marshall, 2006b), which is available from the eMM project website.

Project Deliverables

The work undertaken in this project is summarised in the following sections, additional detail is available in the full project report.

Revision of the eMM Processes and Practices 

The eMM has been extensively revised based on information, feedback, collaboration and analysis from the following sources:

  • the results of the New Zealand eMM version one pilot (Marshall, 2005);
  • feedback from the participants in the international workshops presented earlier in this report;
  • work with the University of Manchester e-Learning Research Centre and with Professor Paul
  • Bacsich in connection with the UK Higher Education Academy's 2006 e-learning benchmarking
  • pilot exercise; and,
  • a substantial literature review conducted under the current 2006 New Zealand Ministry of Education Tertiary E-Learning Research Fund supported research described in the current report.

Changes to the eMM Resulting from the First Sector Assessment 

The processes that constituted version one of the eMM are listed in Table 2 below. These were used to perform the initial capability assessment of nine New Zealand institutions reported in Marshall (2005). It was noted in that report that the initial evaluation identified some limitations with the these processes: 

"Applying these tests and examining the results presented in this report, it is clear that the initial set of processes used can be improved. This is not unexpected; the SPICE process set was developed through a number of 'trials' conducted over several years and the eMM processes are likely to require considerable refinement." (Marshall, 2005, page 14).

In that work, the following recommendations were made: 

  1. Processes L1, L7 and E5 should be combined into a single process describing the use and maintenance of learning objectives as guides to student learning. 
  2. Processes C6, O4 and O9 should be combined into a single process describing the need to support student use of technology through advance warning of its use and opportunities to practice and develop confidence prior to engagement in course work. 
  3. Processes E1, E2, E3 and E4 should be combined into a single process describing effective evaluation of courses in an e-learning context. 
  4. Processes C5 and C8 should be combined into a single process describing the support available to students in the use of technology while engaged in course work.
  5.  Process C1 should be moved into the Development process area. 
  6. Processes O6 and O7 should be combined into a single process describing the provision of administrative information to students. 

Following these recommendations reduced the set of forty three processes to thirty four. However, review of the literature suggested that as many as one hundred additional processes could potentially be incorporated. Merging this wider set and evaluating their utility was therefore undertaken as the first stage of revising the eMM. 

Accreditation Criteria, Standards and Heuristics Mapping 

An initial assessment of the literature on e-learning standards and heuristics describing the various institutional factors that result in effective e-learning design, development, delivery and maintenance identified 377 individual items, resulting in the identification of forty two potential processes. Each of these was then matched with the eMM version one process set and correspondence with either a process or a dimensional characteristic of one or more processes was undertaken. When no match could be made, the item was examined further and then expressed as a process within the evolving set.

Changes to the eMM Resulting From the International Workshops

Three workshops were held internationally following the publication of the first sector evaluation (Marshall, 2005). These were held in Melbourne in association with an ACODE (UK Higher Education Academy (HEA) announced that it was funding a e-Learning, benchmarking project.

Version Two of the eMM Process Set 

The outcome of this redevelopment work is the version two eMM process set and the associated practices. There are thirty five processes incorporating nearly one thousand practices. The full details of these processes is supplied in the separate eMM Version Two Process Guide (Marshall, 2006a) and the summary description of how to apply the methodology is provided in the eMM Version Two Process Assessment Workbook (Marshall, 2006b), both of which are available from the  eMM project website.

Tables 2 and 3 list the processes in version one and version two of the eMM respectively. In comparing Tables 2 and 3 some differences are clearly apparent, the Evaluation process area has shrunk considerably, and the Coordination and Support process area is now labelled Support. Less apparent is that the process set is now much more focused on aspects of capability that present particular challenges to institutions engaged in e-learning. This has seen a number of processes in the Organisation area in particular being collapsed into process O8. 

The language used in the process descriptions has been revised substantially to remove, as much as possible, any inclusion of words that imply particular ways of performing the process, or that are attempting to summarise concepts such as quality. This latter issue is problematic as the process should be done excellently, that is the goal of the eMM. Determining whether or not an institution is currently excellent is the outcome of the assessment. Use of words such as 'proper' or 'clearly' in the first version of the eMM are unhelpful as they are not open to consistent interpretation. The intention with version two has been to remove as much as possible of the value judgement to the level of the practices and to minimise interpretation at that level as much as can reasonably be done. 

The greater definition of the five dimensions and re-examination of the process set has seen much of the Evaluation process area extracted into aspects of the fourth dimension of other processes and a reemphasis on the need to gain feedback from both staff and students. The greater clarity in the differences between dimensions one and two also now makes the resulting assessments more discriminating, and based on the evidence from the sector reanalysis presented below, reduces the significant correlation in capability in these two dimensions observable in the first pilot evaluations (Figure 4 below). 

Table 2: eMM Version One Processes
Learning: Processes that directly impact on pedagogical aspects of e-learning 
L1.  Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and programme requirements
L2.  Student interaction with teaching staff and other students is an essential characteristic and is facilitated through a variety of ways 
L3.  Teaching staff clearly communicate how communication channels should be used during a course or programme 
L4.  Teaching staff manage student expectations over the type and timeliness of responses to student communications 
L5.  Feedback to student assignments and questions is constructive and provided in a timely manner 
L6.  Students are instructed in the proper methods of effective research, including assessment of the validity of resources 
L7.  Learning outcomes for each course are summarised in a clearly written, straightforward statement 
L8.  Assessment of students communicates high expectations 
L9.  Student work is subject to clearly communicated timetables and deadlines 
L10.  Courses are designed to support a diversity of learning styles and to ensure accessibility 
Development: Processes surrounding the creation and maintenance of e-learning resources 
D1.  Guidelines regarding minimum standards are used for course development, design and delivery 
D2.  The reliability of the technology delivery system is as failsafe as possible 
D3.  Learning outcomes, not the availability of existing technology, determine the technology being used to deliver content 
D4.  Technical assistance in course development is available to teaching staff 
D5.  Teaching staff are encouraged to use technical assistance when (re)developing courses 
D6.  Teaching staff members are assisted in the transition from classroom teaching to online instruction 
Coordination & Support: Processes around the day-to-day management and support of e-learning delivery
C1.  A centralised system provides support for building and maintaining the e-learning infrastructure 
C2.  Students have access to sufficient library resources that may include a 'virtual library' acessible through the WWW 
C3.  Teaching staff and students agree upon expectations regarding times for student assignment completion and staff response 
C4.  Students are provided with hands-on training and information to aid them in securing material from a range of sources consistent with the discipline or subject.
C5.  Students have convenient access to technical assistance throughout the duration of the course/programme 
C6.  Students are provided with detailed instructions regarding the electronic media used in a course prior to commencing it 
C7.  Students are able to practice with any technologies prior to commencing a course 
C8.  Questions directed to student service personnel are answered accurately and quickly 
C9.  A structured system is in place to address student complaints 
C10.  Instructor training and assistance continues through the progression of the online course 
C11.  Teaching staff are provided support resources to deal with issues arising from student use of electronically-accessed data 
Evaluation: Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle. 
E1.  The programme's educational effectiveness is formatively and summatively assessed with multiple, standards based, and independent evaluations
E2.  The programme's teaching/learning process is formatively and summatively assessed with multiple, standards based, and independent evaluations
E3.  Summative data such as enrolment numbers, completion rates, and costing is used as a measure of effectiveness within course/ programmes
E4.  Success of technology/innovation used as a measure of effectiveness within course/programmes 
E5.  Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness 
E6.  Instructional materials are reviewed periodically to ensure they meet programme standards 
E7.  Teaching staff capability in making the transition from classroom to online teaching is formally assessed during training 
Organisation: Processes associated with institutional planning and management 
O1.  A documented set of formal criteria are used to determine access to funding and other resources which support course and programme (re)development
O2.  A documented technology plan is in place and operational to ensure quality of delivery standards 
O3.  A documented technology plan is in place and operational to ensure the integrity and validity of information delivered, collected and stored
O4.  Before starting a programme, students are advised of any particular requirements of that programme to ensure they possess the personal and technical skills needed for that programme
O5.  Students are provided with supplemental course information that outlines course objectives, concepts and ideas 
O6.  Students are provided with supplemental course information that outlines admission requirements, tuition and fees and other relevant information
O7.  Students are provided with supplemental course information that outlines requirements for additional resources such as books or other materials
O8.  Students are provided with supplemental course information that outlines student support services. 
O9.  Before starting a programme, students are advised of any particular technological requirements of that programme to ensure they have access to the minimal technology required by the course design
Table 3: eMM Version Two Processes
Learning: Processes that directly impact on pedagogical aspects of e-learning 
L1.  Learning objectives are apparent in the design and implementation of courses 
L2.  Students are provided with mechanisms for interaction with teaching staff and other students 
L3.  Student skill development for e-learning is provided 
L4.  Information provided on the type and timeliness of staff responses to communications students can expect 
L5.  Students receive feedback on their performance within courses 
L6.  Research and information literacy skills development by students is explicitly supported 
L7.  Learning designs and activities result in active engagement by students 
L8.  Assessment of students is designed to progressively build their competence 
L9.  Student work is subject to specified timetables and deadlines 
L10.  Courses are designed to support diverse learning styles and learner capabilities 
Development: Processes surrounding the creation and maintenance of e-learning resources 
D1.  Teaching staff are provided with design and development support when engaging in e-learning 
D2.  Course development, design and delivery are guided and informed by formally developed e-learning procedures and standards
D3.  Explict linkages are made in the design rationale regarding the pedagogies, content and technologies chosen 
D4.  Courses are designed to support disabled students 
D5.  All elements of the physical e-learning infrastructure are reliable, robust and sufficient 
D6.  All elements of the physical e-learning infrastructure are integrated using defined standards 
D7.  Resources created are designed and managed to maximise reuse 
Support: Processes surrounding the support and operational management of e-learning 
S1.  Students are provided with technical assistance when engaging in e-learning 
S2.  Students have access to a range of library resources and services when engaging in e-learning 
S3.  Student enquiries, questions and complaints are collected formally and managed 
S4.  Students have access to support services for personal and learning issues when engaging in e-learning 
S5.  Teaching staff are provided with pedagogical support and professional development in using e-learning 
S6.  Teaching staff are provided with technical support in the handling of electronic materials created by students 
Evaluation: Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle
E1.  Students are able to provide regular formal and informal feedback on the quality and effectiveness of their e-learning experience
E2.  Teaching staff are able to provide regular formal and informal feedback on quality and effectiveness of their e-learning experience
E3.  Regular formal independent reviews of e-learning aspects of courses are conducted 
Organisation: Processes associated with institutional planning and management 
O1.  Formal criteria used to allocate resources for e-learning design, development and delivery 
O2.  Institutional learning and teaching policy and strategy explicitly address e-learning 
O3.  A documented specification and plan guides technology decisions when designing and developing courses 
O4.  A documented specification and plan ensures the reliability, integrity and validity of information collection, storage and retrieval
O5.  The rationale for e-learning is placed within an explicit plan 
O6.  E-learning procedures and which technologies are used are communicated to students prior to starting courses 
O7.  Pedagogical rationale for e-learning approaches and technologies communicated to students prior to starting courses 
O8.  Course administration information communicated to students prior to starting courses 
O9.  The provision of e-learning is guided by formal business management and strategy 


Updated Sector Assessments 

The substantial revision in the eMM that has generated version two has resulted in a substantially improved methodology and process set that should enable useful international comparison between New Zealand tertiary organisations and institutions based in many other countries. In order to enable this, it is necessary that the analysis presented in Marshall (2005) be performed again, using the current methodology and process set. 

The results of that re-analysis are presented in detail in the full report and summarised below in Figure 5. It is important to emphasise that for most institutions this does not reflect a new analysis, but rather a re-analysis using the evidence collected in 2005. For some institutions this is still an accurate description of where their capability currently lies as a year is not sufficient time to change a complex institution. In some instances changes are starting to occur and it is likely that this analysis slightly underestimates their capability overall. Two new institutions have been included in the current data set and these have also be analysed with both eMM versions for consistency. 

Comparing the results for the two versions in Figures 4 and 5 demonstrates the impact that the new methodology has had. In particular the focusing of emphasis on e-learning capability over general aspects of learning and teaching has seen the results decline across the sector. This is most noticeable in the Organisation and Evaluation process areas, where the absence of a large block of capability based on traditional systems and delivery approaches has had a visible result.

More subtle is the greater independence between the assessments for dimensions one and two that was noted above. This has seen the capabilities of most institutions decline across most processes, although the relative strengths and weaknesses noted in the first analysis (Marshall, 2005) are still present. Universities A, B, E and Polytechnics Z, Y and W are still clearly more capable than the remaining institutions although the greater definition now makes the relative strengths in Support (University A) and Learning (University B) more apparent. The greater strength of the five institutions appears to be a consequence of having employed dedicated staff who have as their job both an operational and a strategic responsibility. Not all of these five institutions are strong in the strategy and planning processes (O2, O5 and O9) nor in dimensions three and five, but there is a clear pattern of planned capability (dimension two) throughout the process assessments. Only nine assessments of more than partially adequate at dimension two are made for the other institutions (Universities C, D and G, Polytechnic X) and four of those are in process O8 which is strongly driven by traditional administrative systems. This compares to seventy nine assessments of largely or fully adequate capability from the institutions with dedicated e-learning staff. 

As well as the relative strengths of institutions a number of patterns observed in the previous analysis remain and some additional general observations at the dimension, process and practice level can be made. Across the entire sector there still remains very little evidence of capability at dimensions four and five. This reflects the general absence of evidence collection informing a systematic and strategic engagement with e-learning. What evidence of capability that was seen at dimensions four and five was generally disconnected from the governance and leadership of institutions and there was no evidence of e-learning yet driving or responding to organisation changes. 

Looking at the six completely new processes (Processes L3, L7, D7, O2, O5 and O9) it is clear that these have exposed additional weaknesses in many institutions. Process L3 illustrates the gap in the empowerment of students as autonomous and independent e-learners, and this probably connected to the general use of passive, transmission based pedagogies rather than active engagement of students (Process L7) and the lack of flexibility in the range of assessments and activities commonly used in e-learning (Process L10). 

The absence of formal support for reuse and development of systems to store and manage e-learning resources (Process D7) is perhaps surprising given the high costs of developing resources and the prominence of learning objects in the e-learning research literature. This appears to reflect an ongoing adhoc approach to teaching design and development, with, particularly university staff seeing teaching as an independent activity undertaken in isolation by individual staff. This intuitive and informal approach is also apparent in the weak capability assessed for Process O5 across most of the sector.

The lack of impact of e-learning on the governance and operation of institutions is evident in the capabilities assessed for Processes O2 and O9. There is very little evidence of systematic updating of learning and teaching policy to reflect the differences and challenges consequent to the use of e-learning nor is there much evidence of business goals and strategies driving investment in e-learning infrastructure. The lack of information provided to students across the sector on how the institution uses e-learning to support their learning (Processes O6 and O7) remains clearly evident.

The weakness in the support of disabled students is still apparent (Process D4), however the separation of Evaluation processes into student (Process E1) and staff (Process E2) aspects has exposed the systemic disregard of staff in current evaluation and feedback activities across the sector. The lack of evidence based practice and the use of guidelines, templates and case studies to support the work of teaching staff is still apparent. At the practice level there was almost no evidence of institutions capturing research-based evidence of successful e-learning technology or pedagogy use. The Evaluation area process capabilities now convey clearly the lack of systematic incorporation of e-learning into institutional evaluation and review procedures. 

Professional development, training and support while of high quality, is generally optional and staff are not specifically encouraged and provided opportunities to improve their own skills. There was little evidence of rewards or other incentives for teaching staff to invest their time in developing and improving teaching, and thus student learning, in general, let alone investing the substantial time needed for e-learning. 

Systems and infrastructure for e-learning remain dominated by the use of traditional LMS facilities, with little evidence of the use of standards, formal planning or the systematic analysis and design of e-learning infrastructure driven by strategic goals. At the level of practices there was little evidence of students or teaching staff being asked to provide their perspectives. The absence of any formal risk analysis and planning is also a significant concern given the increasing, if largely unremarked, dependence of institutions on their e-learning infrastructure. 

Support for students and staff remains sharply disparate with University A and Polytechnic Y standing out well from the rest of the sector. The one exception remains libraries who, more than any other internal group appear to understand the changing requirements of students and staff as electronic information use grows. 

The full report provides a more detailed analysis of the capabilities assessed across the sector for each process. As with the first version of the eMM, the identities of the individual institutions remain strictly confidential as do details of the analysis that might allow their identification.

Figure 4: eMM Version One New Zealand Teriary Sector e-Learning Capabilities

See PDF Page 18.

Figure 5: eMM Version Two New Zealand Teriary Sector e-Learning Capabilities

See PDF Page 19.

Web-based Data Collection and Analysis System 

One of the goals of the current project was to make self-assessments of e-learning capability possible. The creation of the detailed eMM Version Two Process Assessment Workbook (Marshall, 2006b) provides a mechanism for enabling self-assessments as well as supporting the work of external assessors. These paper materials are intended to be complemented with a web-based system for conducting and (under conditions of strict confidentiality and anonymity) sharing analyses, capable of providing guidance for institutions and also acting as a mechanism for ongoing comparison and analysis of the sector as a whole. 

The revision of the eMM that has resulted in version two, and the detailed lists of practices in particular mean that it is feasible for a web-based data entry and analysis process to be undertaken. An initial system with a basic set of functionality has been created and is described below. It is intended that this system will evolve over the coming year as more data is entered by participating institutions. Access to the system is restricted and accounts can be obtained by emailing the project lead: Stephen Marshall or by completing the account creation form on the e-Learning Maturity Model project website.

It is important to emphasise that, while self-assessments can be conducted using the eMM Version Two Process Assessment Workbook (Marshall, 2006b) or this web system, substantial organisational change is unlikely to occur without external assistance and critical evaluation of the institutional capability by an independent assessor. The self-assessment process is, however, useful in gaining an appreciation of how the eMM works, the types of evidence needed to determine capability and whether or not major strengths or weaknesses are likely to be identified by a fully independent assessment. 


The web system is designed to support four types of use: 

  • Users interested in finding out details of the capabilities assessed in the sectors represented, initially limited to New Zealand institutions, this will ultimately include the ability to compare between sectors internationally; 
  • Users interested in finding out about the eMM methodology and process by exploring a hypothetical example (The University of the South Pacific) in detail; 
  • Users performing self-assessments of their own institution or organisation; 
  • Users performing external assessments of other institutions or organisations. 
Figure 6: eMM Web System Version 2 Assessments Screen

Figure 6: eMM Web System Version 2 Assessments Screen

Future Work

The eMM has already made contributions both in the understanding of the e-learning capability within individual New Zealand tertiary institutions and across the New Zealand sector as a whole, and internationally as part of a wider scholarship around institutional change and the contribution made by e-learning benchmarking. 

Within the New Zealand sector, anecdotal feedback from the participating institutions suggests that the analysis provided by the eMM has been accepted and is starting to support a systematic and strategic improvement in the quality of e-learning provided to our students. A future goal is to gather empirical evidence of the impact of the eMM and to thus improve the ability of the model to motivate sustainable and effective change. 

An obvious future direction is the expansion into other countries, providing the potential not only of international benchmarking of e-learning but also the sharing of exemplars and ideas nationally and internationally. From the beginning, the eMM has been intended to provide a mechanism for articulating and sharing those critical things that enable technology to facilitate improved student learning outcomes: 

"the model might form the basis for an ongoing discussion within the e-learning community with a view to identifying the key practices necessary for achieving improvements in elearning activities" (Marshall and Mitchell, 2002)


Finally, it is hoped that the current web system will ultimately provide a mechanism for collecting ideas and contributions from collaborators and colleagues interested in the goal of improving e-learning capability. The outcomes of this research are freely available from the project web site  in the hope that this will facilitate investigation and further development internationally.

References 

  • Alexander, S. and McKenzie, J. (1998). An evaluation of information technology projects for university learning, CUTSD and DEETYA, Australia. 
  • Bacsich, P. (2005). Theory of Benchmarking for e-Learning.
  • Bain, J. (1999). Introduction to the special edition. Learner-centred evaluation of innovations in higher education. Higher Education Research and Development, 18 (1), 57-75. 
  • Baruque, L.B. and Melo, R.N. (2004). E-Learning Governance: How to Control and Secure e-Learning Systems. Proceedings Second PGL Database Research Conference.
  • Bates, A.W. (2000). Managing Technological Change: Strategies for College and University Leaders. Jossey-Bass Publishers, San Francisco, USA. 
  • Bates, T. (2001). The continuing evolution of ICT capacity: The implications for education. The Changing Face of Virtual Education. (pp. 29-46). G. M. Farrel. Vancouver, Canada, The Commonwealth of Learning. 
  • Blandin, B. (2004) Are e-learning standards neutral? Proceedings CALIE 04: International Conference on Computer Aided Learning in Engineering Education, Grenoble, France. 
  • Boyle, T. (2003) Design principles for authoring dynamic, reusable learning objects. Australian Journal of Educational Technology 19(1):46-58. 
  • Buckley, D.P. (2002) In pursuit of the learning paradigm: Coupling faculty transformation and institutional change. EduCause Review January/February 2002:29-38. 
  • Bush, V. (1945) As we may think. The Atlantic Monthly 176(1):101-108. 
  • Chickering, A. and Ehrmann, S. (1996). Implementing the seven principles: Technology as lever. AAHE Bulletin, Oct:3-6 
  • Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin 1987, 39(7), 3-7. 
  • Choat, D. (2006). Strategic Review of the Tertiary Education Workforce: Changing Work Roles and Career Pathways. Scoping Report prepared for TEC by Education Directions and Martin Jenkin & Associates 
  • Collis, B. and Moonen, J. (2001). Flexible Learning in a Digital World: Experiences and Expectations. Kogan Page, UK. 
  • Conole, G., Oliver, M. and Harvey, J. (2000). Toolkits as an Approach to Evaluating and Using Learning Material. Proceedings of the Learning to Choose, Choosing to Learn: 17th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Coffs Harbour Australia, Southern Cross University. 
  • Conole, G., Oliver, M., Isroff, K. and Ravenscroft, A. (2004) Addressing methodological issues in e-learning research. In Proceedings of the Networked Learning Conference 2004, Lancaster University, UK. 
  • Cuban, L. (2001) Oversold and underused: Computers in the classroom. Harvard University Press, Cambridge, MA, USA. 
  • Cunningham, S., Ryan, Y., Stedman, L., Tapsall, S., Bagdon, K., Flew, T. and Coaldrake, P. (2000) The business of borderless education. Department of Education, Training and Youth Affairs, Commonwealth of Australia. 23 
  • Curtis, B., Hefley B., & Miller S. (2001). People Capability Maturity Model® (P-CMM®) Version 2.0, Software Engineering Institute, Carnegie Mellon. DfES (2003) Towards a unified e-learning strategy. Department for Education and Skills, UK 
  • El Emam, K., Drouin, J-N. and Melo, W. (1998). SPICE: The Theory and Practice of Software Process Improvement and Capability Determination, California: IEEE Computer Society. 
  • ELRC (2005) Benchmarking. Briefing paper, The e-Learning Research Centre, The University of Manchester, UK. 
  • Fernando, S.D., Dahanayake, A.N.W. and Sol, H.G. (2005). A Holistic Maturity Model for IT Aligned University Education: IA-LMM. Proceedings of the IASTED International Conference on Education and Technology, July 4-6, 2005, Calgary, Alberta, Canada. 
  • Freisen, N. (2004a) Learning objects and standards: Pedagogical neutrality and engagement. In Proceedings of the IEEE International Conference on Advanced Learning Technologies. Joensuu, 
  • Fullan, M. (2001). The new meaning of educational change (3rd ed.). New York: Teacher's College Press. GAO (2003) Military transformation: Progress and challenges for DOD's Advanced Distributed Learning programs. United States General Accounting Office report GAO-03-393. 
  • Graham, C., Cagiltay, K., Lim, B.-R., Craner, J. and Duffy, T.M. (2001): Seven Principles of Effective Teaching: A Practical Lens for Evaluating Online Courses. The Technology Source, March/April 2001.
  • Griffith, S., Day, S., Scott, J. and Smallwood,R. (1997). Progress Made on a Plan to Integrate Planning, Budgeting, Assessment and Quality Principles to Achieve Institutional Improvement. AIR Professional Files Online No. 66 Winter, 1997. http://airweb3.org/airpubs/66.pdf Accessed 15 Jan 2006 
  • Hagner, P.R. (2001): Interesting practices and best systems in faculty engagement and support. Paper presented at the National Learning Infrastructure Initiative Focus Session, Seattle, USA. 
  • Hanna, D. E. (1998). Higher education in an era of digital competition: Emerging organizational models. Journal of Asynchronous Learning Networks, 2 (1), 66-95. 
  • Harris, K. (2004). Use a Maturity Model to Make the Most of E-Learning. Gartner Research Note DF- 22-3036 . h
  • Herbsleb, J., Carleton, A, Rozum, J., Siegel, J., & Zubrow, D. (1994). Benefits of CMM-based software process improvement: Initial results, Software Engineering Institute, Carnegie Mellon. 
  • Humphrey, W. S. (1994). The personal software process. Software Process Newsletter, Technical Council on Software Engineering, IEEE Computer Society, 13 (1), SPN 1-3. 
  • IEEE (2002) 1484.12.1 IEEE Standard for Learning Object Metadata.
  • IHEP (2000) Quality on the line: Benchmarks for success in internet-based distance education. The Institute for Higher Education Policy. 
  • Inglis, A., Ling P., & Joosten, V. (1999). Delivering digitally: Managing the transition to the knowledge media. London, Koogan Page. 
  • Ison, R. (1999) Applying systems thinking to higher education. Systems Research and Behavioural Science, 16, 107-112. 
  • Karelis, C. (1999). Education technology and cost control: Four models. Syllabus Magazine, 12 (6). 
  • Katz, R. N. (1999) Dancing with the devil: Information technology and the new competition in higher education. Jossey-Bass, San Francisco, CA, USA. 
  • Kenny, J. (2001). Where academia meets management: A model for the effective development of quality learning materials using new technologies. In G. Kennedy, M. Keppell, C. McNaught, & T. Petrovic (Eds.), Meeting at the Crossroads. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education. (pp 327-334). Melbourne, Biomedical Multimedia Unit, The University of Melbourne. 
  • Kenny, J. (2002) Managing innovation in educational institutions. Australian Journal of Educational Technology 18(3):359-376. Kirkpatrick, D. (2003). Supporting E-learning in Open and Distance Education. Proceedings of the 17th annual AAOU conference, Bangkok, Thailand. 
  • Laurillard, D. (1997). Applying Systems Thinking to Higher Education. Milton Keynes, Open University. 
  • Laurillard, D. (1999). A Conversational Framework for Individual Learning Applied to the Learning Organisation and the Learning Society. Systems Research and Behavioural Science, 16, 113-122. 
  • Laurillard, D. (2002) Rethinking university teaching: A conversational framework for the effective use of learning technologies, 2nd Ed. Routledge Falmer, London, UK. 
  • Lawlis, P. K. Flowe, R. M. & Thordahl, J. B. (1995). A correlational study of the CMM and software development performance,. Crosstalk: The Journal of Defense Software Engineering, 8 (9), 21-25. 
  • Lima, P.S.R., de Brito, S.R., Fonesca, O. and Favero, S. e E. L. (2005). Personalizaco de Interfaces para Ambientes Virtuais de Aprendiazigem baseados na Construcao Dinamica de Comunidades. Proceedings CLIHC'05, October 23-26, 2005, Cuernavaca, Mexico. 
  • Love, D., McKean, G. and Gathercoal, P. (2004). Portfolios to Webfolios and Beyond: Levels of Maturation. Educause Quarterly 27(2):24-37. 
  • MacGregor, R.C. (1993) Are we recognising the organisational impact on educational software design? Australian Journal of Educational Technology 9(1):59-68. 
  • Manford, C. and McSporran, M. (2003). E-Learning Quality: Becoming a Level Five Learning Organisation. Proceedings of the 16th Annual National Advisory Committee on Computing Qualifications Conference, Palmerston North, New Zealand, 2003.
  • Marshall, S. (2005). Determination of New Zealand Tertiary Institution E-Learning Capability: An Application of an E-Learning Maturity Model: Report on the E-Learning Maturity Model Evaluation of the New Zealand Tertiary Sector. Report to the New Zealand Ministry of Education. 132pp. 
  • Marshall, S. (2006a). eMM Version Two Process Assessment Workbook. Wellington: Victoria University of Wellington. 
  • Marshall, S. (2006b). eMM Version Two Process Guide. Wellington: Victoria University of Wellington. 
  • Marshall, S. and Mitchell, G. (2002). An e-learning maturity model? Proceedings of the 19th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education, Auckland, New Zealand. 
  • Marshall, S. and Mitchell, G. (2003): Potential Indicators of e-Learning Process Capability. Proceedings of EDUCAUSE in Australasia 2003, Adelaide, Australia.
  • Marshall, S.J. and Mitchell, G. (2004), "Applying SPICE to e-Learning: An E-Learning Maturity Model?" in Proceedings of the Sixth Australasian Computing Education Conference (ACE2004), Dunedin. Conferences in Research and Practice in Information Technology, Vol. 30. R. Lister and A. Young, Eds. 
  • Marshall, S. J., & Mitchell, G. (2005). E-learning Process Maturity in the New Zealand Tertiary Sector. In proceedings of EDUCAUSE in Australasia 2005 Auckland, New Zealand. 
  • Marshall. S. and Mitchell, G. (2006) Assessing sector e-learning capability with an e-learning maturity model. Paper accepted for presentation and publication in the proceedings of ALT-C 2006, Edinburgh, UK. 
  • Milne, J. and Suddaby, G. (2005). E-Learning guidelines: Final report for the New Zealand e-learning quality standards, framework guidelines project. Massey University Training and Development Unit, Palmerston North, New Zealand. 
  • Mitchel, P. (2000). The impact of educational technology: a radical reappraisal of research methods. In Squires, D., Conole, G. & Jacobs, G. (Eds), The Changing Face of Learning Technology. University of Wales Press, Cardiff, UK. 
  • Neuhauser, C. (2004). A Maturity Model: Does it Provide a Path for Online Course Design? The Journal of Interactive Online Learning 3(1). 
  • NHSU (2003). E-Learning for Health: A Discussion Paper. UK National Health Service University Discussion Paper, NHSU, UK. Oblinger, D. G., & Maruyama, M. K. (1996). Distributed Learning. CAUSE Professional Paper Series, No. 14. 
  • Olivier, B. and Liber, O. (2003) Learning content interoperability standards. In Littlejohn, A. (Ed) Reusing online resources. Kogan Page, London, UK. 
  • Oppenheimer, T. (2003). The flickering mind. RandomHouse, New York, NY, USA. Parkes, R. (2004). Background Paper Supporting a Proposal to Implement E-Learning in a Large Health Care Organisation. Flexible Learning Leaders 2004 paper. http://www.flexiblelearning.net.au/leaders/ fl_leaders/fll04/report_interim/r_parkes.pdf Accessed 16 January 2006
  • Paulk, M. C., Curtis, B., Chrissis, M. B., & Weber, C. V. (1993a). Capability Maturity Model, Version 1.1. IEEE Software, 10 (4), 18-27.
  • Paulk, M., Weber, C., Garcia, S., Chrissis, M.B., and Bush, M. (1993b) Key Practices of the Capability Maturity Model, Technical Report CMU/SEI-93-TR-025, Software Engineering Institute
  • Peters, G. (1999). A systems failures view of the UK National Commission into Higher Education report. Systems Research and Behavioural Science, 16, 123-131.
  • Petrova, K. and Sinclair, R. (2005). Business Undergraduates Learning Online: A One Semester Snapshot. International Journal of Education and Development Using ICT 1(4). 
  • Phipps, R. (1999) What's the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education. The Institute for Higher Education Policy, Washington, DC, USA. 
  • Radloff, A. (2001). Getting Online: The challenges for academic staff and institutional leaders. In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.), Meeting at the Crossroads: Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education, (pp 11-13), Melbourne, Biomedical Multimedia Unit, The University of Melbourne. 26 
  • Reid, I. C. (1999). Beyond models: developing a university strategy for online instruction. Journal of Asynchronous Learning Networks, 3 (1), 19-31. 
  • Ryan, Y., B. Scott, Freeman, H. & Patel, D. (2000). The Virtual University: The Internet and resourcebased learning. London, Kogan Page. 
  • Ryan, Y. (2002) Emerging indicators of success and failure in borderless higher education. The Observatory on borderless higher education.
  • SECAT (1998). Why Would You Want to Use a Capability Maturity Model, Systems Engineering Capability Assessment & Training. 
  • SEI (2004). Process Maturity Profile: Software CMM 2004 Mid-Year Update, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, USA. 
  • Sherry, A. C. (2003). Quality and its measurement in distance education. In M. G. Moore & W. G. Anderson (Eds.), Handbook of Distance Education (pp. 435-459). Mahwah, NJ: Lawrence Erlbaum Associates. 
  • SPICE (1995) Software Process Assessment version 1.00. 
  • Spendolini, M. (1992). The benchmarking book. New York: Amacom. 
  • Strauss, H. (2002). The right train at the right station. Educause Review, (May/June), 30-36. Taylor, J. (2001). 29 Steps to Heaven? Strategies for transforming University teaching and learning using multimedia and educational technology, The Open University: 22. 
  • Twigg, C.A. (2001). Quality Assurance for Whom? Providers and Consumers in Today's Distributed Learning Environment. The Pew Learning and Technology Program, Rensselaer Polytechnic Institute. 
  • UMass Information Technology Council Subcommittee on Academic Technology (2005). Academic Technology For Teaching and Learning: Vision and Plan.
  • Vollmer, J. (2003). The Enterprise LMS Market: Where Are We Now? Chief Learning Officer. 
  • Wilcox, P. (2005a). Benchmarking. Briefing paper, The e-Learning Research Centre, The University of Manchester. 
  • Wilcox, P. (2005b). Maturity Models. Briefing paper, The e-Learning Research Centre, The University of Manchester. 
  • Young, B. A. (2002). The paperless campus. Educause Quarterly, 25 (2), 4-5. 
  • Zemsky, R. and Massy, W.F. (2004) Thwarted innovation: What happened to e-learning and why. The Learning Alliance, University of Pennsylvania

Where to find out more

Contact Us

Education Data Requests
If you have any questions about education data then please contact us at:
Email:      Requests EDK
Phone:    +64 4 463 8065