Determination of New Zealand Tertiary Institution e-Learning Capability: An Application of an e-Learning Maturity Model (eMM)

Executive Summary

Glossary

ANTA Australian National Training Authority
ADEC American Distance Education Consortium
ADL Advanced Distributed Learning
ALA American Library Association
CanREG Canadian Recommended E-learning Guidelines
Capability Capability, in the context of this model, refers to the ability of an institution to ensure that e-learning design, development and deployment is meeting the needs of the students, staff and institution. As well, capability includes the ability of an institution to sustain
e-learning support of teaching as demand grows and staff change
CMM Capability Maturity Model
eMM e-Learning Maturity Model
IHEP The Institute for Higher Education Policy
LMS Learning Management System
Practice Activities undertaken by institutions, that contribute to capability in individual
processes
Process A high-level activity that has been found through research and evaluation of e-learning
to positively contribute to institutional e-learning capability
Process area A collection of individual processes that share related institutional capability outcomes
SCORM Sharable Content Object Reference Model
SPICE Software Process Improvement and Capability dEtermination
W3C World Wide Web Consortium
WAI W3C Web Accessibility Initiative
WCET Western Cooperative for Educational Telecommunications
WWW World Wide Web

Introduction 

This report presents an overview of e-learning performance framed in a methodology designed to assessprocess capability. The approach used is designed to be independent of technology and pedagogy decisions,focusing rather on the ability of an institution to deliver e-learning in a high-quality and sustainable way.The methods used are based on the theoretical work of Marshall and Mitchell (2002; 2003; 2004) and are outlined in detail in the Methodology section below. This research is supported by the New Zealand Ministry of Education Tertiary E-Learning Research Fund.

This is an executive summary of the research reported in the complete project document available from http://www.utdc.vuw.ac.nz/research/emm/. The information presented in the body of the complete report includes a comparison across the New Zealand publicly funded tertiary sector. Using the e-LearningMaturity Model (eMM) framework, the report provides a high level overview of e-learning capabilityacross the sector. It is important to emphasize that the analysis presented is dependent on the materials provided by each institution. This material was supplemented by publicly available material such as websites, policies and enrolment packs, but the self-selecting nature of the projects selected for detailed examination (see the Methodology section, page 12, for details) means that some bias may be present.

It should be noted that this research does not publicly identify any participating institution. The reportsprovided to participants are confidential to them. This work is not an attempt to rank New Zealand institutions but rather to provide guidance both at an institutional and sector-wide level for improvingthe quality and sustainability of e-learning. To that end, no attempt has been made to either order theresults or assign numerical values to the overall performance. Instead, each section will note in generalterms the characteristics observed both in practice and at a policy level that might be said to be exemplars nationally.

This study uses the practices and processes outlined in Marshall and Mitchell (2004). These were formulated from the well-regarded ʻSeven Principlesʼ (Chickering and Gamson, 1987; Chickering and Ehrmann, 1996) and the ʻQuality on the Lineʼ benchmarks developed for and promulgated by the Institutefor Higher Education Policy (IHEP 2000). These are widely cited and regarded as useful for assessing theeducational effectiveness of e-learning.

This report is divided up into three main sections. The first section, starting on page 5, provides an overview of the results for the New Zealand tertiary sector that are based on the detailed analysis conducted. This includes observations of relative strengths and weaknesses as well as suggestions for improvement.

The second section, starting page 12 of the report, covers the methodology used in detail, including the underlying research and background to the project.The final section, available in the complete report, contains detailed, process by process, results for the institutions obtained by this research.

This includes the background justification for each process, indicators of capability, exemplars of best practice as well as detailed comments on the New Zealandtertiary sector capability in each process. 

Acknowledgements

A large body of research such as this is dependent on the support and assistance of a number of people. Most importantly are the staff of the various participating institutions who generously gave of their time in the completion of the questionnaires. While you cannot be named, your assistance was vital for the success of this project and is much appreciated; it is hoped that the outcomes of the analysis are of value to your institution.

The model owes much to the work of Dr Geoff Mitchell and his contribution and ongoing friendship remain key to the ongoing research. Also important was the contribution made by the two research assistants, Charlotte Clements and Darren Hoshek.

The support of the New Zealand Ministry of Educationʼs Tertiary E-Learning Research Fund and staff in enabling this research is acknowledged with gratitude.

1. Overview of results for the New Zealand Tertiary Sector

This report presents assessments of e-learning capability made of a total of nine New Zealand tertiary education organisations using the methodology of Marshall and Mitchell (2002; 2003; 2004). These represented six of New Zealandʼs eight universities and three polytechnics. The institutions range from very large to small, including distance and rural institutions as well as urban providers.

Capability, in the context of this report, refers to the ability of an institution to ensure that e-learning design, development and deployment is meeting the needs of the students, staff and institution. Capability includes the ability of an institution to sustain e-learning support of teaching as demand grows and staff change.

Performance of each institution was assessed by the examination of up to three typical projects supplied by that institution against measures from the e-learning and pedagogical research literature. Typically, the assessment of capability is made by identifying evidence of individual processes actually taking place within courses (more information on the methodology is provided in Section 2, page 12). The full list of processes assessed is given in Table 11, page 16.

One of the objectives of this report and the underlying research was to validate the methodology used. It is important to emphasise that, while the model is based on widely accepted indicators of e-learning capability, in many cases there is a lack of empirical evidence supporting their use. Assessments of individual instutional capability in particular areas should thus be used as a guide to further investigation and planning rather than absolute indicators of performance. It is hoped that ongoing work will provide evidence of the usefulness of the methodology as further assessments are made of other institutions nationally and internationally as well as changes in assessments over time for particular institutions.

Examination of the summary view of results for the sector in Table 6 (page 11, also repeated in the back cover of this report) illustrates that a range of capabilities have been assessed, with no institution uniformly stronger compared with the others. It is noteworthy that while the weakest overall capability was seen in one of the smallest institutions, much larger institutions are also assessed as lacking capability in many areas.

A number of observations that appear to apply across the sector can be made. Principally, there is a need for a greater self-awareness within the sector. In a number of cases very strong performance was seen in isolated projects and this is not being recognised by the institutions concerned and used as a basis for improving performance across the whole institution. Particularly within universities, it appears that many decisions within individual courses are made without an awareness of the work of other teachers in the same institution or of the wider scholarship surrounding teaching and learning.

Following on from this is the absence of a planned intentionality in the way many institutions are engaging in the use of e-learning. While all institutions are making use of learning management systems, many are not placing the use of these systems within a framework of strategy and guidance to teaching staff that will transform learning. There is a definite sense that existing approaches for teaching and learning are being carried over to technology without reflection and planning. A clear example of this is in the absence of linkages provided to students between the learning objectives of courses and the technologies and pedagogies they encounter. Similarly, there is little information provided to students across the sector that prepares thems for the use of e-learning within their courses. Only one institution assessed told students prior to enrolment what technology they would encounter in their learning. The ability to improve the use of e-learning technologies and pedagogies is also compromised by the weaknesses in the evaluation and review of existing practice seen throughout the sector.

Some institutions are engaging in a formal process of improving the use of e-learning technology and pedagogy within their courses. It is perhaps not unexpected that this has resulted in a stronger overall assessment than for institutions which have not done so. However, it is clear from the general absence of results in the higher levels of the model that much work yet remains to be done.

This section provides a brief summary of the performance of the sector in each process area. Readers may find the material provided on page 15 helpful when interpreting the tables. Detailed comments on each of these process areas and individual processes along with recommendations for improving capability are in the main body of the complete report. Readers are strongly encouraged to review the detailed comments for each process area and may also find the fold-out table at the end of helpful.

Learning

This process area is concerned with the pedagogical aspects of e-learning, particularly those which communicate the underlying structure and logic of a course to students. The goal is ensuring the attainment of the highest quality learning outcomes possible for students in an e-learning context. The individual processes are directed at preserving and extending the essential aspects of an effective learning environment that apply regardless of the particular technology, pedagogy and discipline.

Across the sector there is clear evidence that pedagogical practice is dominated by the independence of the individual departments and teaching staff working on courses. The common thread is that good policies are provided without guidance or examples as to how to actually comply with them in practice. Similarly, good practice is not being codified as guidelines and templates for reuse so as to encourage the building of capability. This information typically remains the province of a limited number of specialists or is buried within a single group.

Learning objectives are poorly used by most institutions, with only one providing course objectives in a clear, structured, statement prior to enrolment (process L7). When objectives are provided in a clear statement as part of the course materials, they are not linked throughout the course outline and used to assist students in their understanding the logic of the course. As well, the objectives stated are dominated by recall and comprehension rather than by analysis, synthesis and evaluation (process L1).

A major problem identified in the sector is the poor support of accessibility (process L10). Ensuring that course materials and activities are able to be accessed by students with a range of disabilities is a legislative requirement. Many of the courses and projects examined had little or no formal accessibility strategy incorporated into their design and development.

All of the institutions assessed need to consider better ways of sharing and promulgating solid solutions to standard problems as well as innovative and effective teaching practice. A number of examples of exceptional performance by individual teaching staff were observed in a number of processes in this and other areas. Sadly, comparison with the other courses from the same institutions makes it clear that this excellence is not being recognised and used to stimulate improvements in other courses. The sustainability of the results once the individuals involved leave must also be called into question. Institutions need to establish formal processes for sharing excellence and using it to support and training teaching staff.

Table 1 illustrates the overall findings of the research for this process area. A more detailed discussion of this process area can be found on page 22 of the complete report.

Table 1: Sector wide Learning process area capability

Click here to view this image.

Development

This process area is concerned with the management and technical aspects of e-learning design and development. The goal is efficient and effective use of resources in the creation and maintenance of elearning resources. The individual processes are directed at informing the development or resources and ensuring that this is done in a way that sustainably builds capability based on experience and success of e-learning deployment in the institution.

Capability in this process area across the sector is particularly dependent on the existence of formal procedures and e-learning support within an institution. Institutions with dedicated staff working within a documented policy and management framework have stronger capability (Universities A, B and E, Polytechnics Y and Z).

Institutions that have no dedicated e-learning support staff or limited resources, operating in the absence of formal frameworks have been assessed as significantly weaker in this area (Universities C, D and F, Polytechnic X). It is important to note that capability in this area is not conditional on the use of a centralised e-learning support facility. A largely devolved model of support, such as that used by University A, can work very well provided it is combined with comparatively strong policies and management oversight. Irrespective of how the resources are allocated, centralised or decentralised e-learning support facilities need to have some way of identifying and promulgating standards, guidelines and templates out to the wider university population, along with examples and case studies illustrating the benefits. The decentralised approach has the risk of pockets of excellence developing, while the centralised approach has the risk of building dependence on a limited number of specialists rather than building capability on a broader front.

A weakness prevalent in the sector is the lack of linkage between the educational outcomes desired and the technologies deployed (process D3). Technology use, such as the facilities of the LMSs, is dominated by administrative and peripheral requirements rather than educational activities. Across the sector there is very little evidence that teaching staff are being provided with training and support in how technology can enable more effective learning.

Table 2 illustrates the overall findings of the research for this process area. A more detailed discussion of this process area can be found on page 44 of the complete report.

Table 2: Sector wide Development process area capability

Click here to view this image.

Coordination and Support

This process area covers the day-to-day management and support of e-learning delivery, particularly as they impact on the ability of students to engage effectively with e-learning. The goal of these processes is ensuring the efficient and effective day to day management of e-learning delivery. This means students and teaching staff can focus on the educational aspects of the course rather than peripheral issues. The individual processes are aimed at ensuring students are placed in the best possible way to succeed in their studies using e-learning and are not hindered by lack of information, support or technology.

The variety of processes included in this area is potentially responsible for the observation that no one institution has dominated performance. All have areas of strength and weakness that need addressing. A consistent finding in this process area is that student support in e-learning courses is not as well developed or comprehensive as it needs to be if students are to move away from a face to face mode of learning (processes C5, C8 and C9). As with teaching staff, students need support tailored specifically to the e-learning approaches adopted by institutions if they are to learn efficiently and effectively. Some institutions have been able to extend student IT helpdesk services online, but there is room for significant improvement. Much of what was observed in the sector is built on a presumption that students already have the skills and background necessary to take advantage of e-learning. What support is provided appears to be a consequence of existing face to face support mechanisms having sufficient flexibility to cope with e-learning, but this is by no means a given.

The libraries of the institutions reviewed appear to be addressing the needs of students somewhat more effectively than the rest of the student support services, increasingly providing a full range of services online with help and support information (process C2). What appears to be missing is resources aimed at helping teaching staff support students in acquiring information literacy and research skills effectively. The use of of customised library support pages for all courses at University B appears very useful. Students are provided with a mix of resources pertinent to the course along with support information, and this encourages them to go beyond the material of the course by enaging in self-directed learning and research.

A particular weakness in the sector lies in ensuring that students are able to familiarise themselves with new technology and can practice using it before it affects their grades (processes C6 and C7). As noted in the Organisation process area, institutions are not effectively communicating expectations and opportunities to students before courses commence. Little use is being made of the opportunities provided by LMS software to offer guest courses that students can explore while deciding whether and what to study.

Table 3 illustrates the overall findings of the research for this process area. A more detailed discussion of this process area can be found on page 58 of the complete report.

Table 3: Sector wide Coordination and Support process area capability

Click here to view this image.

Evaluation

This process area is focused on quality assurance and evaluation processes throughout the entire lifecycle of e-learning design, development and deployment. The goal is encouraging reflective practice informed by evidence from previous success and failure. The individual processes are directed at ensuring the evidence collected is robust and able to provide a reliable base of knowledge for future strategy and sustainable development both of infrastructure and staff skills.

Evaluation of courses is a requirement imposed on all institutions by the Government monitoring agencies and these formal processes have carried across into e-learning courses (processes E1, E2 and E3). However, these are simply the same processes that are applied to traditional teaching. There is a need to develop evaluations that assess particular issues relating to the technology and pedagogies adopted for e-learning (process E4). This is apparent in the observation that the sector shows little capability in assessing the impact on student learning and staff workloads of technologies already in use.

A particular weakness of the sector is the absence of any attempt to formally assess teaching staff skills in e-learning delivery (process E7). The positive impact of assessment, particularly of a formative nature, on student learning is well established. The absence of it as a tool to support the development of teaching staff suggests that training and support is informal and not regarded seriously by the staff or institutions. All of the institutions offer workshops and support to teaching staff but performance by individual staff in these is not assessed and there appear to be few objective assessments of teaching staff skills in this area. The teaching qualifications offered by some institutions offer a potential opportunity for assessing performance and improving staff skills but these tend not to focus on e-learning and are in any case not required for all teaching staff.

Table 4 illustrates the overall findings of the research for this process area. A more detailed discussion of this process area can be found on page 82 of the complete report.

Table 4: Sector wide Evaluation process area capability 

Click here to view this image.

Organisation

This process area is concerned with the institutional planning and management of e-learning. The goal is ensuring that e-learning usage is well managed and planned to deliver the strategic and operational outcomes required by the institution. The individual processes are directed at ensuring the administrative and organisational aspects of e-learning are high quality, efficient and effective, particularly as institutions transition from face-to-face delivery.

Aspects of this process area are quite strong across the sector as they build on pre-existing processes that apply for all courses irrespective of the use of e-learning technologies and pedagogies. In particular, the processes that relate to communicating essential course information are generally adequate, although inconsistently applied at times (processes O5, O6, O7 and O8).

A notable weakness across the sector is a systematic lack of information provided to students in advance regarding the use of technology in courses (processes O4 and O9). Only one of the institutions (Polytechnic Y) provided specific information in their enrolment packs regarding the use of technology in particular courses, even when this went well beyond standard use of the LMS facilities. As noted in the Learning process area, much of the information that is provided to students in course outlines should be freely available before enrolment as it is vital for students. By hiding this information from students they are losing the opportunity to prepare for the courses and to plan for their own particular circumstances.

Also apparent is vulnerability in the sector as growth in LMS use continues. Much of the information in these systems is essential to business continuity and vulnerable to both intentional and unintentional loss or corruption. Formal consideration of the management of student information created and supplied during e-learning coursework appears generally lacking in the sector (process O3). Similarly, few institutions could identify a formal technology plan that guided the choices made in the design and development of e-learning courses (process O2).

Table 5 illustrates the overall findings of the research for this process area. A more detailed discussion of this process area can be found on page 98 of the complete report.

Table 5: Sector wide Organisation process area capability

Click here to view this image.

Table 6: Sector Comparison of eMM Institutional Capabilities

(See PDF)

2. Methodology

The assessment of capability in a complex area such as e-learning is difficult and necessarily involves reducing large amounts of detail into a broader overview that supports management decision making and strategic planning. It is inevitable that this approach will fail to single out the subtle nuances and innovative work of individuals that motivate teaching staff to work on individual projects. Institutions and individuals will always have the ability to choose to invest time and other resources in innovative, unique opportunities. The focus of this study, and thus the form in which the data is presented, is aimed at a less lofty goal, that of changing organisational conditions so that e-learning is delivered in a sustainable and high quality fashion to as many students as possible. As noted by Fullan:

“The answer to large-scale reform is not to try to emulate the characteristics of the minority who are getting somewhere under present conditions … Rather, we must change existing conditions so that it is normal and possible for a majority of people to move forward” (Fullan, 2001, page 268)

The framework used in this analysis is based on the Capability Maturity Model (CMM, Paulk et al., 1993) and SPICE (Software Process Improvement and Capability dEtermination, El Emam et al., 1998; SPICE, 2002). The underlying idea is that the ability of an institution to be effective in a particular area of work is dependent on their capability to engage in high quality processes that are reproducible and able to be sustained and built upon. The characteristics of an institution that enable high quality processes are to some extent able to be separated from the details of the actual work undertaken that will vary depending on particular circumstances. This separation means that the analysis can be done independently of the technologies selected and pedagogies applied, thus allowing for a meaningful comparison across the sector.

Capability, in the context of this model, refers to the ability of an institution to ensure that e-learning design, development and deployment is meeting the needs of the students, staff and institution. Capability includes the ability of an institution to sustain e-learning support of teaching as demand grows and staff change.

Building on the SPICE model, the e-Learning Maturity Model (eMM) divides the capability of institutions to sustain and deliver e-learning up into five major categories or process areas (Table 7). The key difference from the original SPICE model is the introduction of the Learning area, which replaces the Customer/ Supplier area used in software engineering.

Table 7: eMM process categories (Marshall and Mitchell, 2003)

Learning	Processes that directly impact on pedagogical aspects of e-Learning
Development	Processes surrounding the creation and maintenance of e-Learning resources
Co-ordination	Processes surrounding the oversight and management of e-Learning
Evaluation Organisation	Processes surrounding the evaluation and quality control of e-Learning through itsentire lifecycle. Processes associated with institutional planning and management

Within each of these areas are a number of processes, derived from the research literature on e-learning quality, which contribute to the overall ability of institutions to perform well in the given process area, and thus in e-learning overall. The advantage of this approach is that it breaks down a complex area of institutional work into related sections that can be assessed independently and presented in a comparatively simple overview without losing the underlying detail.

An obvious requirement of this model is that the processes chosen are based on empirical evidence and represent ʻcommon truthsʼ about e-learning capability:

“are there common practices or ways of creating e-learning resources and learning environments that are accepted, useful and able to be described in a way that others can adopt them and improve their own e-learning capability?” (Marshall and Mitchell, 2003, page 4)

The processes used in this research were developed from the ʻSeven Principlesʼ of Chickering and Gamson (1987) and ʻQuality on the Lineʼ benchmarks (IHEP 2000) as outlined in Marshall and Mitchell (2004). These are unlikely to be the best possible set of processes for ensuring e-learning capability development but they have the advantage of being widely accepted as guidelines or benchmarks for e-learning delivery (Sherry, 2003).

For each process in the model, evidence was sought as to the institutional capability as described below, and a ranking made at each of six levels (Table 8) generating a matrix, such as those used above, that summarises capability.

Table 8: eMM capability levels (Marshall and Mitchell, 2003)1

Level	Focus
5: Optimised	Continual improvement in all aspects of the e-Learning process
4: Managed	Ensuring the quality of both the e-learning resources and student learning outcomes
3: Defined	Defined process for development and support of e-Learning
2: Planned	Clear and measurable objectives for e-learning projects
1: Initial	Ad-hoc processes
0: Not performed	Not done at all

Level 0 (Not performed) means that the process is not performed at all, which can include a deliberate decision not to engage in a particular process. This level is a special case and a rating at this level means that no evaluation at the other levels of the process capability is undertaken.

Level 1 (Initial) is the default, ad-hoc level characterised by individual initiative and rankings at this level relate to how well the process is performed at all. It is important to emphasize that institutions can have extremely effective processes operating at this level, but in the absence of capability at higher levels there is the risk of failure or unsustainable delivery and the likely wasting of resources through needless duplication.

Level 2 (Planned) relates to the use of formal objectives and plans in conducting the work of the process. The use of formal plans makes projects more able to be managed effectively and reproduced if successful, but does not automatically result in more effective performance of the process outcomes measured in level 1. Nor does the term ʻformalʼ imply the use of institutionally defined process standards as measured in level 3.

Level 3 (Defined) is the use of institutionally defined and documented processes including formal standards and documented guidelines. An institution operating effectively at level 3 has clearly defined how a given process should be performed. This does not mean, however, that the staff of the institution follow those guidelines or standards.

Level 4 (Managed) examines the use of metrics and quality assurance processes such as formal evaluations. Performance at level 4 is somewhat dependent on defined standards (level 3) but capability at this level reflects an ability to measure and control the outcomes of the process and the way in which the practices of the process are performed by the staff of the institution.

Level 5 (Optimised) captures the extent to which an institution is using formal approaches to improve capability measured at the lower levels. Capability at this level reflects a culture of continuous improvement and the explicit consideration of past process performance when designing and promulgating new or updated practices, guidelines or standards for the process area.

At each of these levels, the individual processes are rated for performance from ʻnot adequateʼ to ʻfully adequateʼ (Table 9). The ratings at each level are done on the basis of the evidence collected from the institution and reflect whether the practices that underlie the process area are performed. The ratings are made on the basis of particular characteristics of the practices within the individual process at each level and thus should not be seen as strictly progressive. It is not uncommon, for example, to have a better rating at level 3 than at levels 1 and 2. Level 0 is a special case and is indicated by a completely empty rating in the charts used in this report. In some cases it was not possible to make an assessment of capability for individual processes at particular levels and in this case a ʻnot assessedʼ indicator is used.

Table 9: eMM capability level ratings (based on Marshall and Mitchell, 2003)

 

In this manner, capability of the institution in performing each of the processes that make up the eMM (Table 11) was assessed by the collection of information on up to three e-learning projects or courses from the participating institutions. This project information was combined with material from institutional web sites and enrolment packs to ensure a comprehensive understanding of how e-learning was being undertaken. The ratings provided for each process were checked and the quality of evidence compared between each participating institution to ensure a consistent rating scale had been used. A proportion of the final results were checked with institutions to confirm the interpretation of the material was correct and complete. A limitation of the current research is that only a single rater has been used and there is no ability to provide external measures of validity. This is an inevitable consequence of the stage of development of the model and the intention is to encourage wider adoption and replication of the current work in order to address these limitations. The original SPICE research was supported by extensive trials over a period of some years (El Emam et al. 1998) and the intention is to support a similar undertaking in the area of tertiary e-learning.

Each process was reviewed a second time and best practice elements of performance, policy and guidelines were identified. This material was aggregated into an institutional report that combined a sector comparison with a detailed review of the institutional capability determined. A draft of the institutional report was then provided to five of the nine institutions (three university and two polytechnic) for comment and feedback on any errors or misinterpretations.

It should be noted that experience of applying this type of assessment in the field of software engineering suggests that most, if not all, institutions initially assessed will show a low level of capability for the processes selected (SEI, 2004). This is not surprising as one of the drivers for the model in the first place is the widely held perception that e-learning could be implemented more effectively and efficiently in most institutions.

Project data collection and analysis

Full human ethics approval to conduct this research was obtained from the VUW Human Ethics Committee (Approval #73/2004).

Invitations to participate in the research were sent to twenty-one New Zealand tertiary organisations including universities, polytechnics, waananga and private providers. This included an information and consent form (Appendix A of the complete report) and a detailed questionnaire that allowed responses on up to three projects (Appendix B of the complete report).

Responses provided answers to specific questions and also a wide variety of associated documentation. The initial material was further supplemented by material such as policy and support information provided to students via institutional websites, enrolment packs for new students, and documents supplied by the institutional contact.

Interpretation of results The results of the assessment of each process are presented in tables similar to that below (Table 10). In this example there are five institutions that have different capabilities for the particular process.

Table 10: Example practice result comparing five institutions

 

Institution A is not performing the process well, with only evidence of some ad-hoc attempts shown by the partially adequate rating supplied for level 1 and the absence of any capability in the other levels.

Institution B is significantly more capable in the process than either A or C with evidence that the process is mostly performed well (the largely adequate rating of level 1) and in a planned fashion (the largely adequate rating of level 2). Note that despite there being evidence at level 2 of planning, this appears to be done without any attempt for consistency within the institution as no capability is shown at level 3.

Institution C on the other hand, while not as capable as B, shows evidence of having defined standards or guidelines for performing the process (level 3). However, these appear not to be having an impact on actual e-learning projects as shown by the lower ratings at level 1 and level 2.

Institution D shows a pattern of very good performance of the process (fully adequate rating at level 1), supported by largely adequate planning (level 2) and an initial set of standards or guidelines (partially adequate rating at level 3). This is perhaps the expected pattern of capability development, building from a base of ad-hoc behaviours that are becoming more standardised as the institution has more experience in e-learning.

Finally, institution E performs the process very well (fully adequate rating at level 1) supported by effective planning (fully adequate rating at level 2), largely adequate standards and guidelines (level 3) and an initial programme of evaluation and measurement of process performance (level 4).

Further analysis of the results in this example suggests that institution C and E will provide potential examples of useful standards, guidelines and policies, while institutions D and E (and to some extent B) will provide individual examples of how to perform the process well. This type of analysis is used to identify potentially useful approaches that are successful in the shared New Zealand context and which can be adopted by all institutions seeking to build e-learning capability. Systemic weaknesses, where no good practice can be identified in the sector, present opportunities for potential Government investment or policy direction, as well as collaborative work within the sector.

Finally, comparison across groups of processes provides an institution with the ability to identify areas of related weakness that can be addressed strategically. Priorities can be easily identified by either comparison with the wider sector, or by comparing process ratings within an institution.

Table 11: E-Learning Maturity Model Processes (Marshall and Mitchell, 2004)

Learning: Processes that directly impact on pedagogical aspects of e-learning
L1.	Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and programme requirements
L2.	Student interaction with teaching staff and other students is an essential characteristic and is facilitated through a variety of ways
L3.	Teaching staff clearly communicate how communication channels should be used during a course or programme
L4.	Teaching staff manage student expectations over the type and timeliness of responses to student communications
L5.	Feedback to student assignments and questions is constructive and provided in a timely manner
L6.	Students are instructed in the proper methods of effective research, including assessment of the validity of resources
L7.	Learning outcomes for each course are summarised in a clearly written, straightforward statement
L8.	Assessment of students communicates high expectations
L9.	Student work is subject to clearly communicated timetables and deadlines
L10.	Courses are designed to support a diversity of learning styles and to ensure accessibility
Development: Processes surrounding the creation and maintenance of e-learning resources
D1.	Guidelines regarding minimum standards are used for course development, design and delivery
D2.	The reliability of the technology delivery system is as failsafe as possible
D3.	Learning outcomes, not the availability of existing technology, determine the technology being used to deliver content
D4.	Technical assistance in course development is available to teaching staff
D5.	Teaching staff are encouraged to use technical assistance when (re)developing courses
D6.	Teaching staff members are assisted in the transition from classroom teaching to online instruction
Coordination & Support: Processes around the day-to-day management and support of e-learning delivery
C1.	A centralised system provides support for building and maintaining the e-learning infrastructure
C2.	Students have access to sufficient library resources that may include a ‘virtual library’ acessible through the WWW
C3.	Teaching staff and students agree upon expectations regarding times for student assignment completion and staff response
C4.	Students are provided with hands-on training and information to aid them in securing material from a range of sources consistent with the discipline or subject.
C5.	Students have convenient access to technical assistance throughout the duration of the course/programme
C6.	Students are provided with detailed instructions regarding the electronic media used in a course prior to commencing it
C7.	Students are able to practice with any technologies prior to commencing a course
C8.	Questions directed to student service personnel are answered accurately and quickly
C9.	A structured system is in place to address student complaints
C10.	Instructor training and assistance continues through the progression of the online course
C11.	Teaching staff are provided support resources to deal with issues arising from student use of electronically-accessed data
Evaluation: Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle.
E1.	The programme’s educational effectiveness is formatively and summatively assessed with multiple, standards based, and independent evaluations
E2.	The programme’s teaching/learning process is formatively and summatively assessed with multiple, standards based, and independent evaluations
E3.	Summative data such as enrolment numbers, completion rates, and costing is used as a measure of effectiveness within course/ programmes
E4.	Success of technology/innovation used as a measure of effectiveness within course/programmes
E5.	Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness
E6.	Instructional materials are reviewed periodically to ensure they meet programme standards
E7.	Teaching staff capability in making the transition from classroom to online teaching is formally assessed during training
Organisation: Processes associated with institutional planning and management
O1.	A documented set of formal criteria are used to determine access to funding and other resources which support course and programme (re)development
O2.	A documented technology plan is in place and operational to ensure quality of delivery standards
O3.	A documented technology plan is in place and operational to ensure the integrity and validity of information delivered, collected and stored
O4.	Before starting a programme, students are advised of any particular requirements of that programme to ensure they possess the personal and technical skills needed for that programme
O5.	Students are provided with supplemental course information that outlines course objectives, concepts and ideas
O6.	Students are provided with supplemental course information that outlines admission requirements, tuition and fees and other relevant administration information
O7.	Students are provided with supplemental course information that outlines requirements for additional resources such as books or other materials
O8.	Students are provided with supplemental course information that outlines student support services.
O9.	Before starting a programme, students are advised of any particular technological requirements of that programme to ensure they have access to the minimal technology required by the course design

Future development of the methodology and underlying model

A particular challenge with the approach used in the development of the eMM lies in the identification of the processes used to determine e-learning capability. The processes used here have been derived from two major sets of heuristics (Marshall and Mitchell, 2004) but clearly there are a number of other potential sets which could be used (Lezberg, 2003; Sherry 2003). The problem is distinguishing processes that contribute to effective and sustainable e-learning from other indicators that are more properly practices within the individual processes or descriptions of their outcomes. In the context of the SPICE methodology, El Emam et al. (1998) identified questions that should be considered when assessing individual processes in capability maturity models. These can be adapted to the current context:

• Are the processes that were chosen the characteristics that tend to make e-learning more capable?
• Do the processes cover all the relevant capabilities needed for effective e-learning?
• Are the processes defined so as to be independent of each other?
• Is this set of processes sufficient to characterise a range of capabilities?
• Do the processes represent the ʻuniversal truthsʼ of e-learning capability?
• Are the processes genuinely applicable to any institutional context?

Applying these tests and examining the results presented in this report, it is clear that the initial set of processes used can be improved. This is not unexpected; the SPICE process set was developed through a number of ʻtrialsʼ conducted over several years and the eMM processes are likely to require considerable refinement. Based on the current work, the following observations can be made:

1. Processes L1, L7 and E5 should be combined into a single process describing the use and maintenance of learning objectives as guides to student learning.
2. Processes C6, O4 and O9 should be combined into a single process describing the need to support student use of technology through advance warning of its use and opportunities to practice and develop confidence prior to engagment in course work.
3. Processes E1, E2, E3 and E4 should be combined into a single process describing effective evaluation of courses in an e-learning context.
4. Processes C5 and C8 should be combined into a single process describing the support available to students in the use of technology while engaged in course work.
5. Process C1 should be moved into the Development process area.
6. Processes O6 and O7 should be combined into a single process describing the provision of administrative information to students.

This reduces the current set of forty three processes to thirty four. However, review of the literature suggests that as many as one hundred additional processes could potentially be incorporated. Merging this wider set and evaluating their utility is a focus of ongoing work.

Data collection during the current research also illustrated that the collection of information for analysis is challenging for the institutional staff involved in the individual courses. The use of a detailed questionnaire, while efficient for the researchers, was less useful for the participants. The amount of time needed to complete the questionnaire was commented upon by some and has potentially reduced participation. Future assessments will require simpler initial questionnaires followed with visits in person. The use of a web-based system that would allow for immediate feedback and partial evaluations is also under consideration.

It is also clear that detailed analysis requires the use of technology be examined in the context of actual courses rather than at the more abstract institutional level. In this research, institutions were invited to report on overall projects as well as individual courses and it is clear that responses provided within a course context were easier to interpret and determine capability from. Future assessments will require technology use be evaluated within the context of courses.

References

Chickering, A. & Ehrmann, S. (1996). Implementing the seven principles: Technology as lever. AAHE Bulletin, Oct:3-6.

Chickering, A. & Gamson, Z.F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7.

El Emam, K. Drouin, J-N. & Melo, W. (1998). SPICE: The Theory and Practice of Software Process Improvement and Capability Determination, California: IEEE Computer Society.

Fullan, M. (2001) The New Meaning of Educational Change (third edition). Teachers College Press, NY, USA.

IHEP (2000). QUALITY ON THE LINE: Benchmarks for Success in Internet-Based Distance Education. The Institute For Higher Education Policy http://www.ihep.com/Pubs/PDF/Quality.pdf Accessed 20 Aug 2003

Lezberg, A.K. (2003). Accreditation: Quality control in higher distance education. In Moore, M.G. and Anderson, W.G. (Eds) Handbook of Distance Education. Lawrence Erlbaum Associates, London, UK: 427-434.

Marshall, S. & Mitchell, G., (2002). Potential Indicators of e-Learning Process Capability. in Proceedings of EDUCAUSE in Australasia 2003 (Adelaide, EDUCAUSE, 2003)

Marshall, S., & Mitchell, G., (2003). An E-Learning Maturity Model? in A. Williamson, K. Gunn, A. Young, and T. Clear (eds), Proceedings of the 19th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education (Auckland, Australian Society for Computers in Learning in Tertiary Education, 2002) http://docushare.unitec.ac.nz/dscgi/ds.py/Get/File-231/173. pdf Accessed 19 Nov 2004

Marshall, S.J. & Mitchell, G. (2004). Applying SPICE to e-Learning: An E-Learning Maturity Model? in Proceedings of the Sixth Australasian Computing Education Conference (ACE2004), Dunedin. onferences in Research and Practice in Information Technology, Vol. 30. R. Lister and A. Young, Eds.

Paulk, M., Curtis, B. et al. (1993). Capability Maturity Model, Version 1.1. IEEE Software 10(4): 18-27. SEI (2004). Process Maturity Profile: Software CMM 2004 Mid-Year Update, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, USA. http://www.sei.cmu.edu/sema/pdf/SWCMM/ 2004aug.pdf Accessed 19 Nov 2004

Sherry, A.C. (2003). Quality and its Measurement in Distance Education. In Moore, M.G. and Anderson, W.G. (Eds) Handbook of Distance Education. Lawrence Erlbaum Associates, London, UK: 435- 459.

SPICE (2002). Software Process Assessment version 1.00: http://www-sqi.cit.gu.edu/spice/ Accessed 18 Dec 2002.  ISBN 0-478-13291-3