Evaluation of Student Facing Web-Based Services: Final Integrated Report (CORE Education)

Impacts of these services on learning

How do these services align with young people’s preferences for their learning?

Young students preferences and dispositions to use online environments are different from adults and therefore different from what adults, teachers, parents and providers anticipate they may want or may be suitable. The main characteristics of the new era in Internet, known as web2.0, are the constant development of infrastructures that allow for the delivery of services expressly tailored to the needs of each user. These include options involving more dynamic user participation, social interaction and collaboration through access to social networking sites, wikis, and communication tools. In his review of the literature for this suite of reports Coogan (2006) identifies a series of factors that are likely to ensure sustained student patronage of educational web services, and these factors are considered in this section in relation to the services reviewed.

• "Web services need to meet young peoples immediate or medium term goals. These may range from a need to be entertained, for socialisation, for information or for help with homework and study" (Coogan 2006, p 24). Collectively all three services attempt to cover these needs. WickED provides young people with entertainment and opportunities for engaging  and fun learning. Both AnyQuestions and Studyit provide opportunities to meet students’ immediate needs for information through both static information pages and interaction with an expert and help with homework where there is a focus on content, and the option to ‘talk’ with an expert about students’ immediate and particular needs.
  
• "To engage young people in the internet educationally there needs to be a strong link between school and teacher driven demands (including self directed inquiry) and what is available on line to help students meet those demands" (Coogan 2006, p 24). Both AnyQuestions and Studyit, in particular, provide opportunities whereby young people can seek assistance for help with issues they are encountering with class homework activities. AnyQuestions operators noted the direct connection between homework sheets and many of the questions students asked for help with. AnyQuestions operators frequently provided help in interpreting or clarifying assignment questions and helping students construct an inquiry where students did not understand quite what was required or in which direction to pursue study (CORE 2007a p. 30). High stakes assessment drives much of the relevancy for students involved in Studyit. The service focusses completely on NCEA requirements within particular subjects as well as study advice and support. The service provides relevancy for older teens who, as a group, Coogan (2006) noted can be quite hard-headed about such demands. Both AnyQuestions and Studyit provides such a clear and obvious connection to school that students will use the sites without direct teacher intervention.
  

WickED links to school and teacher driven demands are more tenuous for students unless their teachers had encouraged the use of WickED during class lessons (NNR 2006c, p. 29). While student motivations for independently visiting the site were to learn and for classroom use (NNR 2006c, pp 10 &15) students were slightly more likely to value the site for their ability to use the site independently, to enjoy the amount to do on the site and to have fun than for improving learning or helping with schoolwork (NNR 2006c, pp. 12 & 13). Sustaining frequency of use and ongoing student engagement in WickED has been difficult to achieve. Indicative of the low frequency of use of this site is the finding from the NNR report (2006c, p. 29) that just over one quarter of student respondents (n= 146) visited once a week or more frequently. Indicative of the low sustained patronage of the site is the finding from the NNR report (2006c, p. 30) that of the 58 repeat users 19% indicated they had patronised the site for longer than 6 months.

• "There is a growing demand and expectation from young people to be able to create, shape, share, and critique content and those websites which enable this are those which flourish." (Coogan 2006, p 24). Both AnyQuestions and Studyit manage the demand for students to control and to a certain extent shape the content by allowing for student centred problems and questions to drive much of the interactive content. In Studyit the level of student ownership is enhanced further by the ability to interact with other students through the forums, allowing for the sharing and critiquing of comment by peers. Student comments about how to improve WickED indicated a desire to increase interactivity both in some of the more static areas of the site, and increase awareness of the ability to communicate on the site. There was also a desire to include more customisable options  (CORE 2007c, p.8 &10).
• "Those providing web content and services for young people need to be highly attuned to the design, navigation and linguistic attractants and repellents for young people" (Coogan 2006, p 24). Studyit provides a good example of attuning site design to suit the targeted age group. Preparatory work with focus groups of students relating to features of design, navigation and website wording contributed to the final design and development of the site. Students indicated they wanted a plain site that was easy to navigate to the spaces required. Considerable time was spent in the design of the words and names of features to retain a close link with notions of school sand teachers but also to be distinctly separate from them as well. The NNR (2006a) report indicated the need for the AnyQuestions introductory pages to clarify wording in such a way as to adequately explain the process of developing information literacy skills, that students will be taken through. WickED students also indicated a no nonsense desire to circumvent introductory sign in features so they could get straight to the interactives. Many students and teachers also found navigation of the WickED site more complex than necessary (CORE 2006b, p8).
  
• The literature suggests that the uptake of these services by students, is influenced to a large degree by how the design and operation of these environments meets and addresses young peoples preferences. Coogan (2006) suggests in his list of factors encouraging of young peoples’ patronage of educational websites that,
  
  
  • "Young people, especially teenagers, value highly interaction and collaboration with peers. Those websites that wish to engage students educationally need to provide for such interaction with young people taking increasing responsibility for those interactions as they grow older (p 24)."
    

Some evidence to support the argument that teens prefer interaction and collaboration tentatively exists within the older-teen (16 – 19 year old), user group of Studyit, a core group of whom are regular contributors to the forums (40.5% of this group said they had ‘used’ to the forums though in what way ie read or contribute was not clear). Younger users, those in the 10 - 15 age group were much less likely to ask a question or engage in the Studyit forums ( only 10.7% of this group had ‘used the forums’ this according to NNR 2006b, p.36) but specific reasons for why this was the case were not clarified in either the NNR report or the CORE report on this service. Data on the size of the login group in Studyit or server stats to ascertain the number of hits on this site was not available. The majority of teen Studyit users, enter the site as guest users, preferring to read the forum contents without contributing, and to use the information pages. The NNR report (2006b p.35) indicates that the forums were used by one in three student respondents and that the greatest use of the Studyit site is for information about ‘Subjects’, and ‘Study and Exam Advice’ (which are used by almost twice as many students as the discussions) as well as checking out ‘NCEA Requirements’. (NNR, 2006b p.34).

The importance of interaction and collaboration with peers is much less clear when AnyQuestions or WickED are considered. Peer interaction has not been designed to be a part of the AnyQuestions service and it is hard to see how it could be, when this has not previously been a part of the traditional service provided by reference librarians. While the opportunity for peer interaction has been included in the design of WickED (in the form of forums) this function has been little used. Neither the NNR report nor the CORE report were able to clarify, with any degree of specificity, why students were not using the WickED forums.  Feedback from students suggested they would prefer to see WickED meeting their learning needs through other forms of interaction that they could personalise and that they had encountered on commercial sites like Bebo.

It is unclear from the data in these six service reports, just exactly how important interaction and collaboration with peers is to the process of ensuring sustained student patronage of educational web services. It is difficult to assess whether this functionality is preferred by students or not or whether other factors such as links to schoolwork are equally or more important. Data on the sustained use of the Studyit site indicates a long-term commitment to the site by 16 – 19 year olds. 76% of this older user group had been visiting the Studyit website for 6 months or longer. Data on the sustained use of the WickED service over time indicated that 19% of students had been using the service for 6 months or longer. There was no comparable data available for the AnyQuestions service.

In what ways does it appear that these web services are changing how young people access and manage their learning?

While there is considerable evidence that the internet in general is beginning to change some of the ways in which young people access and manage their learning, there is little direct evidence from these reports as to the ways and extent these particular sites are changing the learning behaviour of young people. The reasons include,

• The reach of these services and the frequency and sustained use of the services remains confined to a small student group in each case. NNR data was collected from 293 AnyQuestions users; 375 Studyit users; and 204 WickED users. NNR netted a considerable number of first time users in each of these surveys further reducing the numbers who are using the services in a sustained way: 30% in the case of AnyQuestions, 29% in the case of Studyit and 60% in the case of WickED.

For a select and small group of students there is clear evidence that some learning behaviours have been altered. For the Studyit student who stated that she would ask a teacher if she had a problem and that would most likely be a Studyit teacher there has been a considerable change. For this student, Studyit provides the option of leaving the problem, until she has more time at home. For other Studyit students there is value in writing down and presenting the problem to others, itself a valuable activity encouraging greater thought and clarity around the problem. For other students there are advantages in taking a problem to an asynchronous environment where interested peers and expert teachers can assist the student asking the question to reflect on the problem by providing multiple perspectives. Asking for help with problems is done under the umbrella of anonymity and can be more comfortable for students who feel more vulnerable in the open classroom. For most of the students who make use of these sites these are resources that are now included in a repertoire of resources and supports such as internet search tools, family members, public libraries, study guides and others.

These services are not individually responsible for changing the way students learn but may be making a valuable contribution towards changing patterns of behaviour that enhance independence in young New Zealand learners. These services are appropriately designed and are meeting (with various levels of success), the target group preferences for online participation and interaction for those growing up immersed in these online environments.

In what ways does it appear that these services are impacting on children and young peoples’ expectations of learning and schools?

There is some evidence that the internet in general is having an impact on young people’s expectations of learning and schools. Both younger and older students are spending more time on the internet each week and much of that is for school related work. Overall, however, there is little evidence currently, that these sites are having an obvious impact on children’s expectation of learning or schools for the reasons listed in the previous section.

• Students are becoming aware that where learning is happening independently from school, this can be supported via interactive web tools. Students appear to want guides.
• Students are also aware that these services are not replacing existing ways of doing things but are supplementing, adding to, and expanding upon existing strategies and resources.
  
• For a certain group of well-disposed students ,learning continues beyond the physical bounds and timetable limitations of schools
  

What is required for these services to be successful for learners and sustainable in terms of ownership, management, pedagogical development, operation, resourcing, and alignment with the wider learning network of young people?

What is required for these services to be successful for learners and sustainable in terms of elements that encourage sustained student patronage and ownership:

Learner success with online services is more likely when,

• Student trust in the quality of feedback from teachers and operators is a very important feature of both Studyit and AnyQuestions. Maintenance of a high quality of service provision is essential.
• Young people can depend on access, through the technology, to ask questions at times which suit them. Studyit provides a good example of flexible dependable access. AnyQuestions ideally needs to be able provide greater flexibility of access to students.
  
•   There is better student and teacher understanding of inquiry based services and the processes that are supportive of learning in these environments.
• Students can feel safe and where teachers and parents can feel it is safe to recommend the environment for young students. Student safety is worked hard at in all three learning environments.
• There are no breaks in communication that disturb or interrupt access.
  
• The information students obtain is well aligned with their studies at school.
  

Elements that appear to contribute to sustaining student patronage and ownership include,

Strong link between school and teacher driven demands: For students to patronise these services, there needs to be a strong link between school and teacher driven demands (including self directed inquiry) and what is available from the service to help students meet those demands. For these services the links with schooling need to be clear and obvious for users and not be dependent on teachers specifically directing students to the site to undertake particular activities.

Guided services that offer relevance, focus, timeliness, and swift response times: It is the student need to have questions answered and problems solved that drives all activity within AnyQuestions and Studyit. Such services need to offer relevance, focus, timeliness, and swift response times. It is important for many students to feel there will be someone available to listen and respond and they are likely to have resolution to their inquiries. For many students who do not wish to engage directly with an expert these sites provide other avenues whereby students can independently seek answers to questions. The AnyQuestions databases and FAQ sections are built from previous student inquiries and the Studyit information areas including ‘Subjects’ and ‘Study and Exam’ advice areas and the ability for all registered users to read the forum postings also provide ways for students to independently seek answers.

Online services are attuned to the design, navigation and linguistic attractants and repellents for young people (Coogan 2006 p.24).

• Ease of access with limited need to sign in or login, plain and ‘no frills’ introductory pages, and easily found and located materials were what students required most. These attractants were obvious in AnyQuestions and Studyit.
• Students commenting on WickED made it clear that unnecessary and confusing graphics and unnecessary sign-ins were aspects that needed improvement (CORE 2007c p.8 &9)
• Younger students talking about WickED made it clear that their expectations about this site was being raised by what could be achieved on some commercial sites. WickED may be more valuable for students if it had the capacity for users to “favourite” and “theme” resources and build personal collections that they could point students to.
• Interpersonal relationships are features of the interactions occurring on the AnyQuestions and Studyit services. Collaboration with peers is a feature of the interactions of Studyit and it is probable that this peer interaction, contributes to the high level of sustained involvement in and long term use of Studyit by a core group of users. Coogan suggests that, ‘those websites that wish to engage students educationally need to provide for such interaction with young people taking increasing responsibility for those interactions as they grow older’ (p 24). This would appear to be the case with Studyit where contributions to the site increased with age and length of time the individual had been visiting the site.
• The role that anonymity plays in the interpersonal relationships and social dynamics of the AnyQuestions and Studyit services was not followed up in the data gathering from students for these reports and is an area that requires further understanding.
• The feature of the interactions in WickED is that students’ relationships are with the content and this is consistent with the literature relating to what younger children like. Younger children see the web as more a source of entertainment than homework help or communication with peers. They enjoy animation and sound effects and love interactivity (Coogan, 2006, p.35). Sites such as WickED may always hold a short-term interest for students who once they have visited the site on one or several occasions and have completed activities and interactive of interest, they may want to move on.
  

For these services to be successful for learners and sustainable in terms of management of these services they require,

• The maintenance of a strong connection between specific purpose, tasks and the targeted age group.
• To be operating within a highly relevant and contextualised external environment. AnyQuestions has taken a traditional educational front desk enquiry service for students into an online world. Operators are highly trained in winkling out resources and information and have added the world of the hidden internet, bibliographic and resource databases to their expertise at all levels of education. Studyit also operates within a highly contextualised external need for students – the desire to achieve well in high stakes assessments. Resources have always been supplied in this area to students needing more help by teachers, schools and publishing companies. The original WickED support provided a partnership focus and context with the four original study support centres.
• Effective partnership arrangements that can add strength and diversity to the operation of these services. In management the National Library provides a good sustainable model for AnyQuestions in the following ways,
• Strong project leadership
• Effective communication with stakeholders
• A trusted advisory group
• Stakeholder buy in is evident with the different libraries involved
• There is a transparent management process in place that includes ongoing analysis of the tools sets that being utilised and development of ongoing strategies for future development.
  
  • Broader stakeholder involvement does not appear to be part of either Studyit or WickED currently.The elements of good transparent project management could be applied to all three projects.
  • Who has control of what is learned and how the process is managed so that students feel they have some ownership of what is learned may be a factor which impacts on sustainability. Both AnyQuestions and Studyit have strong inquiry based pedagogical processes in place which allow for students to drive the content or specifically what is they need to learn. In WickED neither the students nor teachers are making decisions around control of content.
  • Clear lines of communication with schools are required.
  • It is important to maintain quality services. AnyQuestions operators and Studyit expert teachers are required to be content knowledgeable and capable in a range of skills. Selection of staff has in both AnyQuestions and Studyit followed a well-managed process ensuring equitable distribution of load across organisations or staff, and training and support programmes and policies to introduce staff to the online environment and to establish procedures for dealing with inquiries. Staff have developed new skills, and learned to deal with young students, in new ways and in a new environment. Skill development involves facilitation and guidance based on young peoples needs and interests. This can be challenging and requires considerable flexibility in approaches used.
  • Increasing the value of the educational experience for students is pivotal for support services. This can be achieved in online environments where the design of the services allows for a convergence of social, cognitive and teaching presence. These elements can be observed to be present to varying degrees in these services.
    

What is required for these services to be successful for learners and sustainable in terms of pedagogical approaches

It is difficult to assess, from the evidence collected in these reports, to what extent pedagogical processes alone are affecting the success (in terms of sustained patronage) of these services. Those elements that appear to the authors reviewing these reports to be important, can be identified as follows,

• Teaching presence: The presence of teachers provides structure and process and can take the form of a person or a resource. Teachers, operators in AnyQuestions, and mentors in Studyit, play an important part in facilitating discussion, setting the climate of individual exchanges, contributing to and maintaining the established climate across the service, and guiding and often selecting the content to be considered and negotiated. Teachers can respond flexibly to a great variety of student problems and circumstances. Involvement of teachers acting as guides and supporters of students learning allows for individualisation of student learning and greater control of that learning by students. Resources are an important feature of both services and supplement the information-dispensing role of teachers.
  
  The teaching presence, that is the structure and process underpinning WickED, occurs completely through the presentation of resources. Games and interactives can be pedagogically effective if they are include cognitive challenge and interaction between players.
  

• Cognitive presence: AnyQuestions and Studyit operate within particular and to some extent similar general cognitive contexts. They are ‘help services’ and generally operate around a problem-based approach to learning. In Studyit people and resources are geared towards helping NCEA students with high stakes assessments and deal with all the specific subject problems, misunderstandings, and angst that accrues around that context. In AnyQuestions people and site resources are geared to towards helping people into the information age, and to deal with all the specific problems around information literacy, and use of the tools and skills that are necessary to mange their way through a new and different environment. The cognitive context for WickED is not so clearly definable.
• Social presence: The social contexts within which these services operate vary greatly and are the defining difference pedagogically between these services.
  

The social processes operating in Studyit are intended to be group orientated. While the majority of interaction occurs between a mentor and inquirer the forum system allows for the opening up of involvement of other students and an encouragement of multiple perspectives on a problem.

The social processes operating in Studyit for increasing student contribution to their own, and others learning, are interesting. The service supports students in taking increasing responsibility to contribute and support the learning of others as can be seen in the increase in contributions to the forums by the 16 – 19 year old group compared to the younger 10 – 15 year old group (NNR 2006b) and the number of interactions that involve a number of contributors (CORE 2007b) This is achieved to a certain degree through open access to the archived forums where interactions are modelled.

The social processes operating in AnyQuestions are not group orientated. Interaction occurs between an operator and inquirer and is not open for the involvement of others or inclusive of other perspectives in the way that Studyit allows. The service does aggregate interesting questions and responses that may be asked by future inquirers into an FAQ database so that some sharing occurs and allows for independent searches. The type of social interaction occurring in AnyQuestions is completely adequate for the task and aligns well to the traditional tasks of reference librarians.

Social presence has not been developed in WickED.

• There is also alignment with wider learning networks of young people; which is cognizant of student learning occurring both within and beyond school and the concept of the 24/7-flexible-time learner who has greater ownership of learning.
• The convergence of pedagogical practices and an emerging web2 environment that allows for learner centred practices to proliferate online.
• Alignment with new wider curricula and consistency with current pedagogical practices that relate to inquiry, co-construction, participation and reflection.
• Aligning pedagogical practices and strategies with emerging web2.0 tools where students create content, collaborate with peers through mechanisms such as blogs, Wikis, threaded discussions, RSS feeds and others to form a learning network with distributed content creation and distribution of responsibilities. These tools when aggregated into learning environments within contexts relevant to learners can enhance inquiry, co-construction, participation and reflection.
  

What is required for these services to be successful for learners and sustainable in terms of operation of these services:

Ready accessibility to expert help: From student responses regarding the Studyit site which is based on asynchronous interaction, it is clear that they valued highly the ability of the service to have someone available to ‘listen’ and to respond within 24 hours to forum inquiries. Students could post inquiries at any time and responses were generally swiftly actioned. The synchronous nature of interaction in AnyQuestions meant that students could interact with an expert only during the hours of operation, which were limited from 2pm to 6pm on afternoons of schools days. Many students mentioned encountering ‘wait time’ for upwards of an hour in order to talk to an operator. Operators themselves recommended longer hours of operation especially into the evening when students were working from home and more operators to alleviate the issues for students.

Reliable and rapid connectivity: For AnyQuestions, addressing technical constraints emerged from the research studies as an issue for operators and users. The co browsing facility ideally requires broadband and drop off or inability to connect with students often occurred where students were on dial up connections. Lack of resolution for student inquiries often resulted from this reason. Also where operators could not connect with the student to use the co browsing facility the modelling of the information literacy trail that was central to the AnyQuestions approach was often not possible and operators would provide an answer or send a link to a URL where a student could find the answer. 

Older students in particular indicated a preference to have plain sites with ‘no frills’ and minimal visual effects. These were preferred only where they were purposefully used to further explain text. WickED users indicted their dislike of the hosts and the sign in feature of WickED.

It is interesting that the NNR reports noted that most students saw the greatest barriers to these services as being student access to computers both at home and at school.

What is required for these services to be successful for learners and sustainable in terms of resourcing of these services

The AnyQuestions and Studyit services need increased resources for publicity to further widen the student user group and perhaps target new students users and draw them into using the services. Further publicity is needed in schools to highlight the value of the services. Teachers have an important roles not only as introducers and recommenders  but in developing better understandings of appropriate use of inquiry and how students can learn by actively interacting and participating with other learners in these services.

Additional resourcing Is required to enhance the services offered by AnyQuestions . The service needs to

• Increase capability of the co browsing software to allow more operators to work at the same time.
• Increase operating hours to include more student homework time.
• Resources may need to be extended to allow operators to be able to work from home if operating hours are to be extended beyond the hours during which libraries are normally open for business.
  

The extent of additional resourcing suggested as required for WickED is very high and is contingent upon establishing clear agreement on the need, the audience and the service provided. Understandings of what is of educative value on the internet for younger teens has almost bypassed WickED. A comment made to the CORE research team was that WickED was based on what was now considered ‘old technology’. Further development is needed both with inclusions of various elements of web2.0 functionality and with deepening the educative value of the interactives for student learning.

The key thing to note in the discussion in this section is that as teens already have their own social networks, there is now little need to replicate these unless the context in which the service operates is completely relevant to students’ needs and well contextualised within existing educational services. For example Studyit, which is established as both highly relevant and well contextualised, could be enhanced with RSS feeds and aggregation capability.

What is required for these services to be successful for learners and sustainable in terms of alignment with the wider learning network of young people.

Young people make extensive use of the internet for homework and school projects. There is evidence that the internet helps to motivate and engage students in their school work. Young peoples’ preferences for particular activities, and the style and ways information is presented varies with the age. Coogan (2006), suggests that older teenagers are,

• goal orientated internet users who value clear organisation and discoverability of content along with interactivity and visual appeal. They dislike graphics for their own sake, poor discoverability and content that has a condescending tone and language…[they] are drawn to social networking sharing and content creation features afforded by a range of technologies which are central to the user driven iteration of the world wide web (p.3)

Studyit is a good fit with the preferences of the older teen and young adult age group that comprise its targeted users. The purpose of the site is orientated around student examination goals, and the site navigation is clear and well organised. There is a good mix of clearly presented webpage information and interactivity. Visual information is minimal but the site is attractive and with neutral colourings. Time in the design and development stages of the project had been spent in working on the tone, and language used on the site. Opportunities for content creation and sharing are available for the particular group of students who wish to engage with the content as well as socially and emotionally in this way. A positive and enriching discourse-culture is maintained by mentors and management. There is also a good fit evident between the goals of students and the services provided on the site. There is a strong connection between school expectations, students goals to achieve, and the people and static information resources available.

AnyQuestions is also a reasonable fit with the preferences of the age group targeted by the service, that is the younger 10 – 15 age group, although the goal orientated approach of this service also attracts a small group (12%) in the 16 plus age range (NNR 2006a, p.8). The targeted age group fits between the older teen group and a younger group who are identified as having different preferences. The purpose of the site is orientated towards providing help for student inquiries relating to schoolwork and school projects and utilises a chat room technology that should be familiar to a number of students within the age range although it was interesting that several students commented that the service provided them with the skills to use a chatroom. Interaction with an operator had been used by 74% of survey respondents (NNR 2006a, p. 18).

Clarity of information about how the service worked especially for new users was questioned in the NNR report (2006a, p 34) as there was some mismatch evident between what the service operators provide in the way of guidance towards locating appropriate resources and students who expected quick answers of the service.  Better information has been included in the pages providing information about the service to users but there is a role for schools and teachers to encourage more awareness of the process of inquiry in students, which would also change student expectations over time.

Opportunities for content creation and sharing as in web2.0 learning environments, are currently limited to interactions between inquirer and operator only, in the AnyQuestions service and wider sharing occurs only if the question and answer are made available in the searchable database.  This is a user driven service and given the importance placed on the internet and search engines in particular to assist with schoolwork, AnyQuestions provides an important and necessary service for students. There is, as in Studyit, strong connections between school expectations, students’ goals to achieve, and the AnyQuestions resources available.

According to Coogan (2006) ‘younger children view the internet more as a source of entertainment than do teenagers. They value multimedia content, games and interactivity, as well as clear navigation and simple restricted use of language’ (p.3).

The WickED service provides games and fun, multimedia, interactive, content should provide a good fit with the targeted age 7 – 12 age group. WickED, is however, is attracting a slightly older group and the service may need to adjust content and approach and as suggested by students themselves incorporate more user control, user driven and either social networking or immersive aspects of the world wide web into its resource structure.

The three services reviewed for this suite of reports are each uniquely different from each other, providing different models of engagement, and using different modes of delivery for different purposes and different groups of learners who have different preferences and different needs. Collectively these services represent and reflect some of the diversity of the online world that is available to young people.