Evaluation of Student Facing Web-Based Services: Final Integrated Report (CORE Education)

Overall impacts of the services and ways these impacts vary?

It is clear from a description of the sites and their functionalities that there is variation in educative purpose and in what these service sites offer students, teachers and schools. Each of the three service sites provides different types of support for different age groups for different purposes and each has a different focus. For example WickED provides opportunities for students to interact with existing highly visual content and resources while Studyit provides opportunities for students to create and work with their own text-based content in the forums. Studyit operates asynchronously and within a text-based environment while AnyQuestions operates synchronously where students can contact operators and communicate within the immediacy of a chat-room-type environment.  

What overall impacts are these emergent web-based learning services having on diverse young learners?

The following list of key findings have been identified from the research reports and from the literature review supporting the reports (Coogan 2006),

• students appear to benefit from using the sites, benefits are less obvious for  teachers and schools.
• The sites and services are accessed and used by niche groups of students within the targeted groups.
  
• Each of the services appear to be supportive of personal learning that meets immediate and medium term goals.
• Learners made fullest use of the site when there was evidence of strong links to school and, or teacher demands.
• The sites provided opportunities for patterns of learning that allow and encourage collaboration.
• These services encourage student involvement and responsibility for interaction
• Levels to which interaction is sustained over time, and frequency of revisiting within these services is variable
  
• Expanding access to knowledgeable experts in ways that better utilises the technology and broadens the reach of some traditional services for students
• Greater support for students’ home-based learning
  
• Opportunities for student learning  are bounded by the stated learning objectives of each site and the contexts within which they operate.
  

These benefits and who accesses them, varies considerably between the services, are more fully explained in the following section of the report and are considered in relation to the individual services.

The variability of these impacts

In this section the variability of the impacts across the service and for different users are discussed. Whether or not particular impacts are more apparent for certain users and whether impacts are more apparent for all users when certain conditions and practices exist is also discussed. An overview is provided in the summary section 2.4. 

Students appear to benefit from using the sites; benefits are less obvious for teachers and schools

As intended the major beneficiaries of each of the services were students, which is consistent with what each of the websites were designed to do. Teachers and schools had a role in introducing the sites and ways they may support learning to students but overall made little other use of the sites.

Although the Studyit learning environment was very closely aligned with school content and student incentives for using the site were focussed on NCEA assessment this site was not intended to be a resource for classroom teachers. Teachers were key promoters and recommenders of Studyit for students and there was some evidence of teachers recommending and introducing students to the site, and reminding to students to use the service.

There is some evidence that AnyQuestions is being used by some classroom-teachers especially as a resource for those developing inquiry approaches in the classroom. These were also often the children who appeared to learn the most from the site. Nevertheless little use could be discerned that integrated the service into classroom practice to any great extent. Primarily those using this site were students who were independently seeking answers to queries that were usually school related.

As with Studyit and AnyQuestions, the WickED service was primarily student-facing but was also more openly encouraging of teachers using some areas of the site, for example, the themes and interactive content in their classroom activity structures. This was supported by a teacher lounge, or forum area where teachers could interact and discuss ways of using the site in their classrooms with other teachers. Although evidence that the site was being used by teachers was found, the CORE research team found it difficult to locate many teachers using the site extensively.

One of the widely identified benefits of using the internet for learning is the extent to which the internet can support individual learners for both formal school activities and for informal and fun activities. Evidence was strong that that these sites were collectively doing just that although not all services were meeting all these attributes.

The sites and services are accessed by particular niche groups of students

Each of these services has been designed to appeal to, and attract the involvement of, a particular age group. Studyit, for example has been designed in consultation with student focus groups belonging to the target group. The design and style of the service, and the instructional approaches as well as the language used, have been carefully matched to the specific internet usage needs of an older teen group (Coogan, 2006) and the service attracts members from the designated age group. WickED has been designed with many more of the strongly visual, fun, lively and entertaining features that younger children like (Coogan, 2006) and is also attracting the intended group (NNR, 2006c).

It is interesting however, that each of the services appears to be attracting a specific niche user group within the broader target group, who are using the service wholly or partly.

AnyQuestions was promoted to the senior primary and junior secondary areas of schooling. The reasons for this choice included:

• the desire to involve students at both primary and secondary level
• need to target the pre-exam students( as it was considered that students in the exam years would be pre-occupied with their assessment activities
• need to operate within the constraints of limited resources and budget

The service appears to attract a wider audience than expected with about half the users being within the targeted age group and most of the remainder being older students or adults. The service appears to be used by a substantially greater number of girls than boys. In the targeted age rage 76% of users are girls and 24% of users are boys. Even given the propensity for girls to answer surveys compared to boys this represents a considerable difference. Interestingly while more girls may actually use the service a greater percentage of boys seemed to be happier with the service while girls appeared to be more ambivalent. Some Maori and Pasifika use has been identified at 14% and 15% of respondents respectively (NNR 2006a p.25). The service was intended to support individual learners, and was not specifically designed to support group or class research.

In Studyit, the greatest impact of the service appears to be on high achiever students. This evidence has surfaced in both qualitative (CORE, 2007b) and quantitative data reports (NNR, 2006b). The latter reported that only 6% of students who responded to the survey did not achieve NCEA Maths, Science and English. Student satisfaction with Studyit indicates that comparatively more males than females are satisfied with the service which may be due to the subject areas that are currently available and perhaps also the type of highly focussed content and interaction that is evident on the site. As with AnyQuestions, the Studyit  service was intended to support individual learners.

The impact of WickED on students learning is harder to define. Student satisfaction levels with the service indicate that students in lower decile schools have a higher appreciation of the site content than students at higher decile-schools who indicated slightly lower levels of satisfaction (NNR 2006c p 17) Appreciative niche groups also included students who speak Maori at home which is some 5% of the user base. (NNR 2006c p 3 & p.19) and girls appeared to enjoy using the site more than boys. 50% of boys as opposed to 26% of girls indicated they were very unhappy, unhappy or ‘in-between’ in their satisfaction with some of the interactives such as games quizzes and crosswords. (NNR 2006c p 17). The WickED service was initially designed to support a wider-range combinations for learning, that included individual, small group and whole class support.

All three services appear to be attracting particular niche or affinity groups within the larger targeted age group. All three sites are attracting a much larger user base of students who felt they are being successful or very successful at school (NNR, 2006a p 35; 2006b p. 15; 2006c, p.20) as is illustrated in Table 1.
 

Table 1:    Shows that these services were being used primarily by students, who felt confident they were doing-well in their studies.

 

The questions relating to confidence in schoolwork included in Table 1 above were not asked of those students responding to the Studyit survey. However in the case of the Studyit service 93% of student users were studying for NCEA and just under half were studying at NCEA level 3, which indicates a high level of school achievement. Indicative results also show that students from lower decile schools tend to be less confident about contributing and participating in the Studyit communication areas. (NNR, 2006b p.11)

There are some gender differences evident relating to subject specialisation (for example more boys are involved in maths and science subjects (NNR 2006a, p.38). The differing approaches to learning encouraged by each site appeared to offer varying results revealing the difficulties of generalising specific niche impacts across all services. For example: While girls appear to indicate a greater enjoyment of exploring and browsing for information on WickED (NNR 2006c p.17), boys appeared more likely to use the AnyQuestions site as a search resource, and are also more likely to be investigating the site for more ‘speculative’ reasons rather than use the chat function to contact an operator for help (NNR 2006a p.39).

The need for help and guidance in specific subjects could also be discerned from the data for each of these services. To some extent this was being determined by,

a)    the type of support being offered. In AnyQuestions, a first port of call for student inquiries relating to locating information, most students surveyed were turning to the site for support particularly in social studies (46%) and in science (40%) and also in English (28%), (NNR 2006a, p. 38). The inquiry approach ultised by the service is a most suitable one for these essential learning areas. The ‘Maths Stuff’ section containing a number of maths interactives was popular with WickED users.

b)    the extent of the expertise the site is able to offer. In Studyit subject areas support was limited to science and mathematics with the later addition of English. Over three quarters of students (76.8%) using the Studyit site required assistance within the mathematics area of the site. (NNR 2006b, p.39).

It is of interest, that the three traditional subject areas of science, mathematics and English and to a lesser extent the fourth area of social studies are those where students found they required most help and support from these services regardless of age or motivations for study. Even in WickED where the resources available span a range of content areas and where subject specialisation is largely blurred, some 38.4% of students visited the ‘Maths Stuff’ section. This was the second most visited area next to games/quizzes and crosswords, which attracted 40% of student users. (NNR, 2006c, p.36).

Each of the services appears to be supportive of students personal and individual learning that meets immediate and medium-term learning goals

Coogan (2006,) found that ‘first and foremost web content and web services need to meet young people’s immediate or medium term goals’ (p. 24) and these needs and goals may include entertainment, socialisation, information or help with homework and study.

These services are best at supporting individual learners and offer services that allow for the immediate and personal learning needs of students to be met in a number of ways that include help with homework and study, providing information, opportunities for socialisation and even entertainment.

All three services offer students help with homework and study within their particular areas. The synchronous aspect of the AnyQuestions service is highly personalised with the availability of an operator (in working hours) who, with the support of a text-based chat function and co-browsing facility, can work with students and provide guidance for their individual and highly targeted inquiries. Even where instances of school based use were uncovered by the Core research team, the service provided a highly individualised support service for individuals or small groups of children involved in an inquiry driven programme or project that were making an inquiry independently of the teacher. One of the interesting aspects of the AnyQuestions service, highlighted in both the NNR and CORE research reports, was the differing expectations between what students wanted and the process employed by the operators to assist students find the answers. Students frequently wanted an immediate answer to a narrowly focussed question, while the service operators offered much more help to students. Operators clarified what the student needed to know, assisted with setting up an inquiry and then provided guidance in how to find the answer. This role in helping students to a greater extent than was expected was one that engendered some frustration by some students but was appreciated by users returning to the site once they were aware of the learning process. This individualised support for the initial stages of the inquiry process is unique among the help site genre.

Specialist curriculum teachers in StudyIt provide individual support for students who are prepared to ask questions via the forum areas.

The strength of both AnyQuestions and Studyit is that the locus of control for learning rests with students. The services are well organised to meet student’s immediate learning goals. If students need an answer there is someone there who will either provide it or guide students towards a process of finding it themselves quite quickly. Both AnyQuestions and Studyit also met students’ medium and even longer term learning goals. As well as offering immediate answers, the AnyQuestions operators encouraged students to gain skills in independent search techniques and in planning appropriate approaches to beginning an inquiry. In the Studyit forums, students could be seen engaging in discussion about some quite long-term options that included future tertiary courses and study possibilities once they had finished secondary school.

Learners made fullest use of the site when there was evidence of strong links to school and, or teacher demands

Coogan suggests that in order to engage young people in the internet educationally there needs to be a strong, clear link between school and or teacher driven demands ~ and what is available online to help students meet those demands.

The findings of the AnyQuestions report (CORE, 2007) indicates that most students using the service do so because they have a homework activity or assignment to complete that has been set by the teacher. The service provides that strong, clear and conscious link between school and or teacher driven demand that is suggested by Coogan (2006) as an essential element to drive use.

Studyit also provided highly individualised and personalised support that illustrated a strong, clear link between school and or teacher driven demand. The fundamental purpose of the site, to support NCEA assessment is a direct link to schoolwork for the individuals who make the most of the service. Over half of the  261 respondents who said they were regular users of Studyit, said they mainly used Studyit when preparing for assessments and exams (NNR 2006, p.41). The instructional design of Studyit with its carefully developed focus on sharing personal and individualised interactions and making these interactions transparent to all users is a feature of the service that is not only highly suited to the targeted teen user group but is also conducive to a quite different pedagogical approach that is not possible in AnyQuestions or WickED.

Students using WickED reported the site helped them learn and they visited the site to get ideas for school related questions and projects. The goal of making learning fun seems to appeal to younger students who need more entertaining elements and visual content to support them with their learning (Coogan, 2006). Quite a large number of students (74%) indicated in the NNR survey (p.10) they had used WickED during a lesson.

However there appears to be less incentive for students to use WickED for immediate focused and independent help for schoolwork. The following reasons were apparent.

a)    Both students and teachers commented on the difficulty of finding what they were looking for on the WickED site. The need to spend time browsing resources and risk being sidetracked from the direct task is less appealing when a specific answer is required, or specific information needed that can more easily be obtained by asking a person directly as in AnyQuestions or Studyit.

b)    Clear links between school and or teacher driven demands are less obvious in WickED for students unless the teacher has used, or recommended specific resources available on the site or has set specific tasks for students relating to the resources available.

These services present different ways of seeking, sourcing and using information that is likely to appeal to different groups and different individuals within these groups as well as the particular learning situations in which students find themselves. The exploratory and browsing options provided by WickED offer a quite different learning context for the individual student than the much more targeted, directed question, and distinctly targeted approaches encouraged by AnyQuestions and Studyit.

The sites provided opportunities for patterns of learning that allow and encourage collaboration

Developing patterns of learning that allow and encourage collaboration between students and between students and experts or mentors is undoubtedly an area where educational use of the internet is innovative. Coogan suggests that “young people, especially teenagers, value interaction and collaboration with peers…websites that wish to engage students educationally need to provide for such interaction, with young people taking increasing responsibility for the nature of those interactions as they grow older “ (2006, p.24) encouraging such interaction also encouraged repeat visits to the site and sustaining ongoing interest for those involved.

These services encourage student involvement and responsibility for interaction

Of the three services, only Studyit, offers a range of means and opportunities for interaction and collaboration to occur between mentors and student contributors to the site that is visible and accessible to comment and reflection by all users. The service is highly encouraging of student responses to other student-contributors and there is a clear sense that these contributions that cover skills, knowledge and affective domains are valued.

The Studyit service is succeeding in engaging students educationally and it is clear that in many cases these older teens are taking considerable responsibility for the nature of the interactions that are occurring. It is interesting that two thirds of transcripts analysed involved 3 or more different people in a continuous interaction but what is even more interesting that more than one in three interactions involved 6 or more different people and one in seven interactions involved more than 11 people. A core group of students are quite deeply involved in Studyit and are able and willing to take responsibility for many of the interactions that are occurring. (CORE 2007b).

AnyQuestions provides for one to one interaction between students and a reference librarian or subject specialist teacher but these interactions are not visible to other students coming into the site unless they have been selected as typical of their type and added to the FAQ database. While many users of the service preferred searching the FAQ database to answer their inquires privately rather than contact an operator, there remains a limited sense that collective knowledge building is occurring on the site.

WickED offers forum spaces for contributors but these were not being utilised by students for purposeful learning or purposeful socialising during the period of research.

Levels to which interaction is sustained over time, and frequency of revisiting within these services is variable

The following tables provide an overview of a) The length of time students had been returning to the site (See Table 2 ) and b) The frequency with which students had accessed the sites prior to data gathering. (See Table 3 below)
 

Table 2:    Shows the length of time students responding to the NNR surveys had been using each of the sites.

 

Interactions over the longer term and frequent site revisiting were a feature of the Studyit service where 76% of the 16 – 19 year old users indicated in the NNR Survey (2006b p.30) they had been using the Studyit site for 6 months or longer. While only 2% of student users indicated they visited the site every day, just over a quarter of the 375 students who responded indicated they visited the site at least once a week or more often. A further quarter indicated they visited once or twice a month. First time users of the Studyit site tended to be younger than the targeted 16 – 19 age group.

Table 3:    Overviews the frequency with which students reponding ot the NNR surveys had been using each of the sites.(From NNR reports and presentations)

 

There is evidence that there is some sustained revisiting occurring with the AnyQuestions service. One in five users were visiting at least once a week or more often. Nearly one quarter of survey respondents indicated they used the site one or two times a month indicated that 44% of users are using the site regularly. Just how long some users had been using the site was not published in the NNR AnyQuestions Service Report presentation (2006a) although 29% of respondents indicated that the day they filled out the survey was also the day of their first visit.

In the case of WickED the Nielsen NetRatings WickED Service Report (2006c) indicates that there was a very high number of first time users entering the site and have ascertained this number at 60% of all students who responded to their online survey (NNR 2006c, p.29) There are very few students who are long-term users of WickED. Of the 58 respondents who indicated they were repeat users 36% indicated they had been visiting for less than a month and only 19% (ie approximately 5-6 students) had been visiting the site for 6 months or more. (NNR, p.30) For the qualitative CORE report longer-term users were unable to be located. There was very little evidence of social interaction in the WickED site which may have as much to do with the abilities and maturity of the users as the instructional design and the purpose and intention of the site. However these findings are important and do have implications for site design and for learning outcomes.

Both AnyQuestions and StudyIt transcripts were investigated and the kinds of inquiries students were making were examined for the ways in which the process of inquiry was being fostered and maintained. Four key stages can be discerned from the information processing inquiry models available and these stages are listed in Figure 1 below.

 
Figure 1:    Shows the four stages of inquiry identified in the CORE reports and the extent to  which each of these stages are fostered in StudyIt and AnyQuestions

 

The majority of users in both Studyit and AnyQuestions were clearly using the services at stage 2 of the inquiry process that is for investigating, finding out and researching. This is quite marked for the AnyQuestions service where 67% of students were asking for answers to preset questions providing a strong argument for locating the use of AnyQuestions in the context of the ongoing programme of learning within school. Frequently AnyQuestions operators could be observed taking students back to the initial phase of the inquiry process and helping students to clarify and redirect their inquiries. The majority of StudyIt students were also initiating the process at stage 2. However Stages 1 and 3 of the inquiry process could also be identified in close to 1/5th of the transcripts in each case. 

For these services to have an impact on learning, users must find them a reasonably essential tool for their inquiry or study, so much so that they will readily return to use the service as a first-port-of-call for help. Sustained and reasonably frequent interactions on a particular topic or problem also need to be observable in order to assume that some form of co-constructed and deeper level learning is taking place. While there is evidence that a distinct and sustained, although also undoubtedly a fairly fluid, learning community has formed around Studyit this is not possible to establish that this is the case for either of the other services where revisiting appears to be more infrequent in nature. In the case of Studyit 61% of interactions by students were return visits to the discussion. Although some evidence of co-constructing knowledge is evident in the AnyQuestions transcripts the probing, guiding, negotiating and redirecting of student thinking invariably happens within the space of a single event and is not normally a learning activity sustained over time and subject to reflection, alternative perspectives and changing ideas.

Expanding access to knowledgeable experts in ways that better utilises the technology and broadens the reach of some traditional services for students

The AnyQuestions and Studyit services makes it possible for students to ask for study assistance from a much wider field of expertise. There are advantages to students in being able to ask a reference librarian or subject specialist teachers for assistance in how to get started on an inquiry and is part of the traditional role of librarians which has not always been accessible to students. Coogan (2006 p.26) mentions the existence of research evidence indicating that ‘Ask a Librarian’ services generally are likely to be reaching audiences that would not normally engage with traditional libraries.

An important role of the AnyQuestions operators is not to simply provide a quick answer but also to assist students by providing guidance on constructing an inquiry and sourcing resources relevant to that inquiry. The NNR report (2006a) indicated good levels of user satisfaction with AnyQuestions, as have overseas evaluation reports on similar services, (Coogan, 2006, p 26), with just over one third of students (36%) finding AnyQuestions more valuable for school use than any other website and just under a third of students (30%) finding AnyQuestions equally as valuable as any other website they used. The NNR report also found that those students who reported having learned information literacy skills within the context of their inquiry were happier with the outcome of their visit than those who simply found answers to their questions (2006a, p. 45).

Studyit students could be observed exploring the nature of different perspectives provided by experts in the form of feedback on English essays, and alternative approaches to solving a problem in Mathematics or Physics. Students were also observed checking the coverage of topics within a subject with other students from different schools. Studyit was also of considerable support to students who did not have strong subject support within their school. For example, one of the teacher mentors in Studyit commented that the service was a godsend for students who have teachers newly arrived to New Zealand and who do not know NCEA. Many students could be observed checking their curriculum coverage with other students. Studyit offers other viewpoints and another alternatives.

Greater support for students’ home based learning

Each of the sites allows for an expansion of services that have traditionally been restricted to schools or libraries into different areas of young people’s lives. All of the services provided considerable support for students in crossing the divide between supporting students both at school and with independent study at home. An advantage area of internet support for students is that time when students are working on homework tasks and they release they don’t know where to begin, or don’t understand some aspect of the task set.

Any Questions operators are able via the AnyQuestions service to support students and teachers in classrooms, as well as students at home. Students without easy access to public libraries can also access this resource. Interaction with an expert on AnyQuestions is synchronous so the site is open for specific operating hours, 1 – 6pm Monday to Friday only, implying the service was designed to be used by students in the classroom as well as at home. The hours or operation were not generally seen as ideal, and were considered by operators to be a constraint on the effectiveness of the service. Operators in AnyQuestions felt the service could be extended to be available for longer period in the evenings to cover homework hours for students. Considering the value of the service for out of school support (73% of students used AnyQuestions for the express purpose of getting an answer to a homework question or a project (NNR 2006a, p.39), and 70% of students used the service from home compared to 30% for all other locations combined (NNR 2006a, p.29)) this seems to be a reasonable recommendation.

Table 4: Overviews the locations student  respondents indicated that had used computers to access each of the services.(From NNR reports and presentations)

 

Studyit has a particularly high level of home usage. Eighty-eight percent of students use Studyit from home although nearly a quarter of students had also used the site from their school library and just over 1 in 5 had used it from their classroom (Nielsen 2006c p.32). The instructional design of the site is orientated to serve the personal and independent study needs students and the site succeeds well in this respect. Studyit is available 24 hours a day, seven days a week, and there is a commitment by mentor teachers to respond within 24 hours, so that students can ask for just-in-time help at any time. Teacher-mentors indicated the highest-use-times for posting questions were from after school to 10.30pm and before school in the mornings. The site bridges the gap in supporting study from home with expert help well. When older teens, involved in their homework, and who as one student said ‘hit a rock,’ can and do receive a rapid response to their problem (CORE Report, 2007b). They often have it sorted before the next day’s school lesson. Many students commented that the service helped them ‘keep up’ with their schoolwork and not fall behind. (CORE Report, 2007b).

WickED, tended to have a higher level of in-school usage than did either Studyit or AnyQuestions. WickED was used in school classrooms, computer laboratories  or the school library by 70% and at home by 47% of students (NNR 2006c, p.31). It is not clear exactly how students were using WickED to help them at home with their studies as the majority of students interviewed and surveyed were first time users and few students using the site could be found to interview.

Opportunities for student learning are bounded by the stated learning objectives of each site and the contexts within which they operate

The potential impacts on student learning varies greatly from service to service, and from context to context in which it is used. Not all services provide for learning opportunities across all domains – that is cognitive, social and affective. Student use of the sites indicates considerable variability in practice and very different outcomes for very different groups of learners.

AnyQuestions: As already stated the primary purpose of the assisted approach in AnyQuestions is not to simply provide a quick answer but to provide guidance for students on constructing an inquiry and sourcing resources relevant to that inquiry. Conversely for 70% of students (NNR 2006a) their goal is precisely to get a quick answer for a homework question or a project. Only 5% of students indicated they had practiced using AnyQuestions as a class lesson.

The NNR report (2006a) mentions that few students (5%) appear to be being introduced to the concept of AnyQuestions as an ‘interactive’ environment so that many students appear to be missing the point of the site. It could be observed from the CORE analysis of transcripts that many students (57%), wanting a quick answer to an inquiry, were asking closed and quite tight questions which operators (in 34% of cases) often found difficult to open out and to guide the inquiry more extensively. Also the operators were experiencing both time pressures from students queuing for help, and pressures resulting from technical difficulties, that meant a straightforward answer to the question was often all that could be achieved.

There was limited evidence of complex thinking observed from the CORE analysis of the AnyQuestions transcripts when the attributes of the thinking key competency were applied to the students’ questions (2007a, p.12). Questions from homework sheets featured regularly and many of the questions were of a very basic, closed question type requiring only a simple factual answer and that students often revealed through the transcripts that they had little understanding of the context or the intent of the question and as a result appealed to operators to assist. The CORE research team considered that the level of ownership of student questions was a key factor that determined how extensively students were prepared to ‘dig-down’ with their thinking and learning.

The stated learning purpose of AnyQuestions, that is to provide guidance for students on constructing an inquiry and sourcing resources relevant to that inquiry, appears to be being achieved for a good proportion of the students involved. A range of reasons for using the site were indicated in the NNR report (2006a p. 40) by students and these included, getting stuck while searching the internet and needing help, wanting to know more about something, getting ideas for homework or projects, curiosity about the site, and because the teachers told them to go to the site to find the answer. Finding a resolution to queries was the most common learning outcome cited by 59% of survey respondents who said they found direct answers and by 31% of respondents who said they were given a link. While the use of information literacy skills was less commonly cited there were nevertheless a third of (33%) of respondents said they had found out about new sites and search engines, 22% said they learned how to search better by themselves and 17% who said they learned how to choose good search words. The CORE research team found that 67% or about two thirds of interactions fitted the ‘investigating, finding out and researching’ category of the inquiry process with only a small percentage of interactions fitting the other categories of the process (2007a, p.15. See also Figure 1 on page 17 of this report). Time may be needed to help establish the culture of site use where students entering the site can expect to be guided towards a suitable answer rather than given the answer.

Coogan (2006) profiles literature sourced from overseas and locally that suggests that many students seem to lack the skills to structure their searches so that they can find useful information quickly. He points readers to an Education Review Office report from 2005 that highlights significant weaknesses evident in New Zealand primary and secondary students (p.27) which are also highlighted in both the NNR and CORE service reports as issues that are also faced by the AnyQuestions operators. ERO (2005) highlight particularly an emphasis on collecting facts rather than the process of information retrieval and analysis and the lack of ability by students to ask rich, challenging questions which open up learning rather than constrain it.

Studyit: Three hundred and sixty–five students users (of whom 81% were in the targeted age range of 16 – 19) think highly of Studyit. This perception is critical to the growth of this service. Four in every five students (79.2%) indicated they were satisfied or very satisfied with the site. It is worth noting that according to NNR (2006b, p 8) this is a strong result relative to many websites evaluated by this group across a variety of industries and audiences. The attributes of the site that contributed to high levels of student satisfaction (all with a mean score of between 4 – 4.47 on a five point likert scale) are the 24 hour, 7day a week availability of the site, the study support value students get from the discussion forums, the ability to pose questions of the teachers, the currency of the site, the relevance to NCEA assessments, ease of use of the site, and how well Studyit fitted with students overall study efforts. It is interesting to note that the 16 – 19 year olds also rated the ability to offer support and advice to other students and the communication tools (e.g. the forums) in this group as well. Over two thirds of the targeted 16 – 19 year old group (70.9%) found Studyit to be as useful, or more useful than other websites used for schoolwork (NNR p. 16).

Perceived impact of the Studyit service on student learning appears to be positive. Over half the students indicated the service had helped with NCEA performance which is strong evidence foor a high level of alignment. Nine percent of users declared use of Studyit had made a big contribution to their exam results and 45% believed it had made some contribution. (NNR p.15) Interestingly postings to the forums tended to be made by ‘high achievers’ with 41% of those who posted frequently feeling that Studyit made a big contribution to their NCEA successes (NNR p.37).

Of the different areas of the site, more than two thirds of users (70.8%) access the subjects area which provides details about topics within a subject and clearly lists assessment criteria. Just slightly more than one in three Studyit-users visit and contribute to the forums with just over 14% of students being very active and posting questions and comments a lot (NNR p.37). The use of subject areas and forums offers quite different learning opportunities and different modes of learning. The structure of the forums allows students to take a variety of roles in social situations. Some of these peer interactions roles were examined in the CORE analysis of the Studyit transcripts (2007b) and most commonly included students providing other students with a direct answer but also redirecting students to other sources of information, probing through questioning and encouragement and affirmation - something that was more commonly expressed between students than by teachers of students (p.19).

The NNR survey provides evidence that only a small percentage of users make regular posts but that reading the posts (that is non active participation) can be just as satisfying and valuable (2006b, p.42).

The CORE service report (2007b) examined the depth and quality of learning occurring in the forums basing the analysis using the SOLO taxonomy. Two levels emerge from the analysis; the first identifies learning outcomes observed at a more surface level and the second identifies learning that could be observed at a deeper level. Some 58% of analysed transcripts had observed learning occurring at a surface level and 42% of learning observed could be identified as occurring at a deeper level. The overall quality of interactions on the Studyit website can be seen to be generally very high and is consistent with the initial expectations of many for a guided answer.

The site encourages students to explore at their own pace with no specific direction provided for ‘expected activities to be performed on the site. Students manage their own time and activities on the site and many obviously have considerable ownership of, and longer-term commitment to contributing and participating in site activity structures.

It is difficult to compare the value of the Studyit site in terms of learning for the target group with any other existing examples. Coogan (2006) acknowledges that external research about websites set up to support students in high stakes assessment appeared to be almost non existent. In fact it seems that far fewer of these services appear to exist than one would imagine .

WickED: According to the NNR service report (2006c), the impact of WickED on student learning was positive. While the site was predominantly reaching children who felt they are doing well in most things at school, those faring less well at school also felt they are gaining something from the site, especially with aspects of the site that were more entertaining or for amusement, help with school work, help working with other students and being able to work on the site independently (NNR 2006c p.21). Students’ mean-ratings of their satisfaction levels for a range of the learning aspects of the site listed above ranged between 3.8 and 3.3 on a five-point Likert scale. Because no archived interaction exists where learning outcomes can be observed and analysed and so few longer-term users of the WickED site could be located for interviews it is quite unclear from the qualitative data exactly what overall impact students felt WickED had on learning. It was clear that interaction was important to the targeted age group (who suggested higher levels of interaction in such areas as the science stuff area for example) as was the high level of visual material available rather than text material that required reading.

It is interesting that Coogan (2006) notes in his review of younger childrens’ educational sites that there are few equivalent websites to WickED that could be located or identified elsewhere. One that was reviewed, Scotland’s Gridclub, indicated that the service provided assets and interactive learning opportunities that might not otherwise be available for the age group. WickED is well positioned to establish and develop greater opportunity for peer interaction and more personalised student control of resources which may help to better position the service as an essential educational asset with a difference for young people. In the meantime interviewed teachers indicated they tended to bypass WickED and use the search engines to find classroom materials and students tended to visit once or a few times and not return.

What is the nature and extent of the comparative advantage of these services? i.e. in what respects are the impacts of these services unique to or more likely in these environments?

What is difficult to ascertain from the data gathered by the research reports into each service is the extent to which use is being made of these resources across the whole of the possible target group. Because these services are emergent in nature it is likely that in each case the user base is quite small when compared to the number of potential users of each service.

However, when taking a closer look into the user groups for each of the services it is evident that there are a number of advantages being offered that are unique to or more likely in these online worlds although this is clearly different for each of the services and variable across the services.

Characteristics that are unique to these particular online services include,

1. The most obviously unique of these characteristics is the extension of teaching and reference services through a form of ‘extended physical presence’ into the home or private-study realm of student’s lives. This is particularly evident in StudyIt and in AnyQuestions where these services indicated extensive use from home and provided help for when students faced some barrier ‘or rock’ that stalls progress for them. Studyit students commented that Studyit helped them keep up with their schoolwork so that they did not feel they were getting behind. Traditional boundaries to what is possible for learners are being explored and negotiated within these services.
2. Unique to these environments also is the extent and nature of the individualised and personalised problem-based interactions that are occurring. This is particularly evident in both the Studyit and in AnyQuestions services which both provide a social presence in the form of expert reference librarians or subject specialist teachers who are available to interact with students. In the interactions with an expert all inquiries are student initiated and represent the personal learning concerns of the student. Experts are facilitators and generally try to offer guidance for individuals to construct better inquiries or think alternatively or more deeply about the problem the student has presented. The majority of problems are being solved or resolved in some way. This has a 78% success rate in Studyit and a 65% success rate in AnyQuestions (The lower success rate for resolution AnyQuestions appears to be due to time constraints and technical difficulties). It is difficult to compare the extent of question asking online by students with that which may happen in the classroom. It is possible that many of the questions asked in these online environments may remain as ‘unasked wonderings’ by many students in busier, noisier and more generally complex face-to-face settings. One student in the Studyit context mentioned that when s/he had a problem s/he would ask a teacher but it was usually the Studyit mentor.
3. Unique, particularly to Studyit, and AnyQuestions is the anonymity maintained by the students approaching the expert for help. In Studyit anonymity is maintained by the use of nom de plumes, as are other teen help and support sites elsewhere. The site is open to public access (access requires only an automated registration) so the use of aliases is primarily for protection of individuals from disclosure and identification in the archived forums. Other effects of the use of aliases, and reduced importance of physical appearance and how they may impact on social identity, social interaction, relationship formation and learning outcomes within these environments, was not explored with students during the interview process. However it appears that the use of aliases may support higher levels of interaction where students feel they can ask that ‘silly’ question or open students up to critical analysis from peers when they post that full text version of their essay for critique and feedback. This aspect of both the AnyQuestions site and Studyit would be worth further investigation and is an aspect of the WickED site that needs further development.
4. A further unique impact on learning relating particularly to these services but was the generally greater control students could maintain over the time and pace of the interactions. In some AnyQuestions transcripts students could be seen to be abruptly terminating interactions when they could quite see where they were going or couldn’t keep up. In Studyit questions could terminate at any time in the process. Students often did this by a simple ‘thanks’ note. The need for students to have to write down their problem to put in front of an expert also allows for quite a unique process of thinking through and clarifying the problem that students do not have in face-to-face situations.
5. A unique feature of all these services is their very ‘New Zealandness’ and their distinct relationships with New Zealand ways, means and values. NCEA is a assessment system unique to this country. AnyQuestions involves trained librarians and subject specialist teachers in working with and recommending resources with strongly New Zealand themes and values to largely young New Zealand students. Although WickED relies on borrowed resources the service also provides some uniquely New Zealand interactives.
6. The nature and extent of the comparative advantages of WickED are not as evident as with the other services reviewed. Many of the features of WickED are replicated in good online interactive sites elsewhere.
   

Summary of impacts on diverse learners

The potential impacts of these services will vary according to the service and the user group making the most of the service. User group involvement appears to vary according to age, sex, ability and even to some extent decile grouping. Table 5 below provides an overview of the services and their varying impacts on learners

Table 5: Overview of the specific areas of Impact of these Services

Scope	AnyQuestions	Studyit	WickED
Purpose	Extend existing public work of reference librarians to help with inquiries to a wider section of the student community	Provide help for students in high stakes assessment - NCEA	To offer engaging, quality controlled, multicultural learning activities and interactives for students.
Age level	All age groups of students aged 7 – 20+	Secondary school students in particular senior secondary students aged 16 – 18 years	Primary school students aged 7 – 12 years
Availability of services	Site databases - 24 hours a day for 7 days a week.  Librarian services 4 hours a day for 5 days a week	24 hours a day for 7 days a week.	24 hours a day for 7 days a week.
Alignment with essential learning areas (ELAs).	All ELAs and cross curricula topics with % usage, social studies (46%) science (40%) English (28%) technology (15%) maths (15%) health & PE (12%) arts(11%)	NCEA level ELAs with % usage, mathematics (76.8%), English (53.1%)  biology (42.7%) chemistry (41.9%)  science (34.9%) physics (33.1%) other subjects (totalled = 6.8%)	All ELAs and cross curricula topics.(Blurring of subject boundaries makes usage difficult to determine with accuracy) mathematics,  science,  social studies,  language,  te reo Maori,  Pasifika,  cross-curricula topics.
Who and what students can interact with	Experts Site content	Experts Other students Site content	Site content
How communication is mediated	Synchronous interaction through text-based chat facility	Asynchronous communication through text-based forums	Asynchronous communication through text-based forums is available but very limited use.
Responses can be found from	From reference librarians Subject specialist teachers Search of FAQ database.	From interactions with teachers From interactions with other students Site content	Site content (Interactives including games and quizzes)
Sustaining activity on the site	Satisfaction aimed for in a single visit  Some evidence of repeat callers but for a different inquiry	Regular revisiting  Ongoing dialogue leading to sustained involvement	High level of first time visitors.  Some evidence of repeat visits to familiar sites
Domains where learning is evident	Cognitive Social	Cognitive Social Affective	Cognitive

Particular impacts are more apparent for certain users. These services appeared to be used largely by students who overall felt confident about their schoolwork and who are motivated to seek out and expand their personal knowledge. Inquiry would be a fundamental learning disposition of active users of these sites. These services can not be expected to provide for students who do not have a disposition to independent learning. Particular subject areas appear to draw more students and these tend to include mathematics, science and English or language although this did depend on the type of service being offered. For example the approach of AnyQuestions towards answering students inquiries using an information processing approach meant that mathematics questions were not so commonly asked of AnyQuestions operators.

Due to the individualised nature of learning in online environments it is difficult to identify specific practices or conditions that are likely to impact on all users. These services were catering for niche groups of learners rather than all learners. For example, the inclusion of te reo Maori and Pasifika resources in WickED were encouraging use by young Maori and Pasifika students.

It is important for services to provide a diversity of ways of accessing the information needed to cater for the different ways different students access their learning. Most services appear to cater for diversity within the targeted student group by offering a range of differing approaches to accessing information and resources. AnyQuestions offers a chat function as well as a searchable database for queries, and a FAQ database where students who prefer can undertake independent searches.  All services offered information resources (as opposed to interactive or communication options) where students could search or browse for information.

The NNR reports for both AnyQuestions and Studyit revealed that the information resources were widely and deeply accessed. For example in Studyit almost twice as many students accessed the onsite information about ‘Subjects’ than viewed the discussion groups or forums. While 74% of AnyQuestions users had used the online chat facility, 24% had used ‘Do It Yourself’ answers and a further 24% had used the Search FAQs function.

However just how diverse the user groups are for these services is a little uncertain. The NNR reports attempted to cover diversity of the service user groups by looking at school decile. Numbers of student-respondents from lower decile schools were lower than for mid and higher decile schools but satisfaction levels in Studyit and WickED were quite similar for students from diverse economic segments of society. Interestingly NNR survey respondents were asked about barriers to use of the services and a common barrier raised was limited access to both computers and internet connection speeds at home and at school (NNR 2006a, p72; NNR 2006b, p.69; NNR, 2006c, p.66). Services requiring broadband as an ideal connection speed such as AnyQuestions means that for many home users uninterrupted access via the co browsing function is difficult. Broadband access remains limited in New Zealand. (In 2005 only 33% of all people 10 and over with internet access at home were using broadband connections (NNR 2006a, p.29)

Close alignment to the aims and objectives of the New Zealand curriculum is evident in all three services especially with providing resources for the essential learning areas and in many of the essential skills areas. Strong linkages with school and teacher demands were evident particularly in AnyQuestions and Studyit where students could be observed asking quite targeted answers relating to the immediate need to complete set homework questions.

Examples of learning were evident across social, cognitive and affective domains although this varies considerably between services with evidence of all three domains being touched by the interaction observed in Studyit. There is also an impact on the type of learning that students can enjoy on these sites. Students can exert more control over the rate and pace of interactions with experts. They can experience just in time and more individualised and personalised learning. The extent of the reach of these services into aspects of student’s lives that have never been possible previously are features of the most used and most valued of these services.

Impacts are also evident on operators and teacher mentors who are moderating these sites. However it is clear these impacts are variable, are limited to specific services and cannot be readily generalised across services. Highly effective teaching practices can be observed in Studyit especially where co construction of knowledge is happening within the affinity group. While breadth and depth of knowledge building is not so evident in AnyQuestions as with Studyit there are still a wide range of effective teaching practices happening in this service.