Evaluation of Student Facing Web-Based Services: Studyit (Nielsen/NetRatings)

Immediate Learning For Young People

In this section, we explore the kinds of learning that are happening through the use of Studyit.co.nz – breaking down student learning into the five core areas: Thinking, Making Meaning, Relating to others, Managing Self and Participating and Contributing.

We also ascertain students’ motivations for using the site and their patterns of usage of the Studyit service.

Patterns Of Usage

Since launch, the number of users to the Studyit site has seen a great upward trend with 149% growth in unique visitor numbers for the period of May to December 2004 versus 2005 (there was little traffic between January and April 2004 as this was the pre-launch period of the site and comparisons against these periods would skew the results).  November, which was proven to be the heaviest usage month for the site in both 2004 and 2005 as students prepare in earnest for the completion of their NCEA studies, showed an increase of 57% in unique visitors in 2005 versus 2004.  The number of repeat visits also improved slightly in 2005 to 1.49 visits per unique visitor per month versus 1.42 in 2004.

These findings are consistent with those from the survey – over half of student users mainly visit the site to prepare for exams or assignments (further discussed later in this section).

Frequency Of Visitation

Just under one in three student users of the site are first time visitors (30.4%) and just over one in four (26.2%) visit Studyit once a week or more often.  This reveals a core group of loyal, frequent users. On further investigation, this group is more likely to:

• be in Year 13 (47% vs 32% of infrequent or first time users); and far less likely to be in Year 11 (14% vs 26%)
• be studying for NCEA Level 3 (51% vs 34%)
• achieve Maths, Science and English NCEA with merit or excellence
  
  • Maths: 74% vs 54%
  • Science: 78% vs 57%
  • English: 60% vs 51%
    

• feel that Studyit contributed to their performance (13% vs 5% ‘big contribution’ and 65% vs 23% ‘some contribution’
• use all areas of Studyit than infrequent users, particularly the forums (52% vs 16%)
• spend more time studying (42% spend more than 5 hours each week studying vs 26%)
• be only slightly more likely to spend more time online (65% spend more than 5 hours each week vs 59%)
• feel that student forums / chat groups are important, if not very important, to helping their study (77% vs30%⁷)
  

There is no significant difference in the frequency of use amongst male and female students.

Younger students are far more likely to be first time users than those aged 16-19 years (56% vs 24%) while the core target age group (16-19 years) are more regular users of the service – 55.4% visit at least once a month compared with 34.6% of 10-15 year olds. Just under one in three 16-19 year olds visit Studyit at least once a week though the site does not stimulate much daily use as do some online resources like instant messaging services, search engines and perhaps webmail.
 
Website tenure is another important measure that supports perceived usefulness and satisfaction with the site. Testament to Studyit’s appeal and usefulness among repeat visitors, close to two in five have been using the website for more than a year. This is predominantly those aged 16 to 19, as highlighted in the chart below.





It is interesting to note that, with the exception of those who have only been using Studyit for less than a month, patterns in frequency of visiting the site are very similar across long and medium term users (6 months to more than one year) and newcomers (1-6 months). This indicates that once the site has been deemed valuable enough to prompt repeat usage (which generally occurs after only a visit or two), frequency of visitation remains similar.

Those in high decile schools are slightly more likely to have been using the site for more than a year (40% vs 31% of those from low decile schools). Consequently, they are also far less likely to have been using the site for less than a month (6%) compared with low decile students (23.1%). This may be a ramification of less encouragement to use the site from teachers (refer section ‘Sources of Awareness’) but also potentially that lower decile schools have lagged in their widespread access to technology like PCs and Internet access.

With regards to the place of accessing the Studyit website, the largest proportion of students access from home and cite ‘home’ as the main place of access (85%). This correlates with their main Internet access point as found through this survey and NetRatings’ market study of 6 to 17 year olds.⁸

Over one in five students access the Studyit website via the school library (24.3%) or the classroom (22.4%), though fewer students name these locations as their ‘main’ place of accessing the site (8% and 4% respectively). While ‘home’ is the main location for all decile groups, only 71% of those in low decile schools are mainly accessing at home compared with 88% for mid decile schools and 86% for high decile schools. This is not surprising given the differing levels of home Internet access and speed afforded to low decile students - 87% vs 97% among high decile students. This finding reinforces the need to encourage school-based usage to ensure that all students have access to the benefits of online resources like Studyit.

Other access points such as public libraries see very little usage.

There is no significant difference between male and female students’ location for accessing the website, though older students (16-19) are three times more likely to connect via the school library than younger students. Similarly, almost one in four (24%) 16-19 year olds are accessing the Studyit website in the classroom compared to less than one in five (17.3%) students aged 10-15 years.

Interestingly, the pattern reverts for ‘main place of access’ as 16% of younger students mainly access in the classroom compared to only 6% of older students. Both groups predominantly name home as the main place of access though (87% of 16-19s and 79% of 101-15s). 

This may correlate with teacher-directed or encouraged use for younger students as opposed to independent use among older students – however, there was little difference between the extent to which older vs younger students were told about the site by a teacher.





Among the majority of students who access the site from home, 85% do so after school and 64% do so on the weekends. As found in the preparatory analysis, little pattern was found in the average visit duration during weekdays and weekends.

Among students making use of the site in the school library, over half access during a school study class and over one third at lunch time. A further 28% claim to access Studyit from the school library during a regular class and 13% do so after school.

The majority of those accessing in the classroom do so as part of a regular school class while a further 39% do so as part of a study class.

Studyit Content Consumed

Students are making use of a fairly good cross section of the site’s functions and content areas as seen in the chart below. The “Subjects” section has the highest usage overall with two thirds (67.2%) of students using the section. “Study and Exam advice” and “NCEA requirements” are also popular, which correlates with the 93% who are studying for NCEA.

These findings are largely consistent with those from the analysis of site measurement data, but based on page impressions only, the ‘Subject’ areas are the second most heavily used section behind the forums. However, this inconsistency between survey and site measurement findings is likely a ramification of the number of pages that each user ‘clicks into’ within the forums in contrast to fewer within ‘subjects’. Similarly, the measurement data sees around one fifth of the page impressions within each subject area attributed to the ‘Links’ and ‘Practice Questions’ pages while the survey found that only around 3% of students had visited this content.⁹ This suggests that the few who do visit these content areas consume a fairly large number of pages – though this finding is indicative only.





The forums are used by one in three students and very few students have ever emailed a teacher via Studyit (just 6%)¹⁰. As mentioned on the previous page, the site measurement analysis found that the forums were the most heavily used area of the site in terms of page impressions – the forums accounted for 39% of all page impressions (PIs) during August 2005 and 50% of PIs during November 2005.

The discussion boards are very popular amongst 16-19 years – they are four times more likely to use this section compared to students aged 10-15 years.

Older students aged 16-19 are using all sections of the site more than 10-15 year olds which is not surprising given they are more frequent users of the site overall and have been making use of the site for a longer time period. The “Subject” section is the most used section among 16-19 years (70.9%) while slightly more younger students visit ‘Study and Exam Advice’.





Interestingly, those at lower decile schools are more likely to use ‘Study and Exam advice’ and the ‘NCEA Requirements’ sections of the site than are their counterparts at higher decile schools, but less likely to use the forums. These findings are consistent with the satisfaction levels exhibited by these groups of students, with lower decile students being more satisfied with the site’s study assistance and relevance and slightly less satisfied with the support offered through the forums. These results are indicative only, based on a sample of only 24 students from lower decile schools.

Use Of The Forums

An important and unique element of the Studyit site and service are the online forums – moderated by teachers but largely facilitating support and communication from and with peers. This aspect of the site is consistent with the means by which young people now connect with one another, obtain social skills and social confidence and communicate with others across town and city, and even national, borders. A website launched last year in the US, myowncafe.org, was developed on this principle and one of the site managers stated “The site has to have an interactive component. Adults mainly use the Internet for research but teens use it mainly for communication.”¹¹

A healthy one in three (34.4%) students use the ‘Communicate” section of the Studyit website. The chart overleaf looks at students’ level of participation of this section. Only 14% of students are very active and post questions and comments ‘a lot’ while just over half have posted comments or questions but tend to read others’ messages or questions. One in three have never posted a comment / question – around half of this group feel they will post something in the future but the other do not intend to post comments or questions. It is promising to note that forum behaviour is not impacted by Internet connection speed – those with dial up at home behave the same way as those with broadband.





Male students are more likely to have posted in the past (72% vs 62%) but females have intentions to post (19%) and similar proportions intend to keep reading posts but not contributing (17%). 

While there may be some relationship between students’ confidence and the degree to which they post messages on the site, there is little difference in the levels of satisfaction with the site, between those who actively post messages vs those who simply read the comments on the forums. This is evidence that just as much value can come from reading the posts as making comments or asking questions. In some cases, however, the samples became too small for a thorough analysis.

It is interesting to note some subtle differences between participation in the forums across various subject areas. Those using the site for Science and Physics are slightly more likely to have posted comments and questions on the forums in the past (75% and 70% respectively) followed by English and Biology (67%), Maths (66%) and Chemistry (62%).

The postings are also largely made by ‘high achievers’ and those who are highly positive about the site – 41% of those who post comments / questions a lot feel that Studyit made a very big contribution to their NCEA performance vs less than one in ten of those who mainly read others’ posts. Interestingly, those who post a lot are far more likely to be high achievers in Maths and Science, while both lower and higher achievers in English reveal similar patterns in their use of the forums. This provides some indication that students of varying abilities in English are willing to post questions on the forums, while largely confident students are asking and responding to questions about Maths and Science.

Specific Subjects Used

Over three quarters of students are drawing on the Studyit service to support their learning of Maths and this is very similar among younger and older students; as is Science (37% younger vs 34% older students). English, biology, chemistry and physics is far more popular among 16 to 19 year olds.

On average, students have used the site to support their learning of 3.2 different subjects.

Likely correlating with the types of subjects being chosen by males vs females, female students are more likely to use the site for English, Biology and Science while males turn to it for Maths, Chemistry and Physics. Indications among lower decile students are more use for Maths and English but far less use for Biology, Chemistry, Science and Physics.

Levels Of Engagement

As noted in the preparatory analysis findings, one indication of the degree to which users are engaged with the site is to monitor the growth in unique visitors relative to growth in the number of pages consumed. The average number of pages viewed as a proportion of visits was 8.3 in 2005 versus 10.2 in 2004, indicating a slightly shallower visit depth and suggesting that visitors are exploring fewer pages on each visit overall. 

Furthermore, average length of site visitation should be tracked and measured against ‘ideal’ durations. The average time spent by a visitor to the site in 2005 fluctuated by a reasonably large margin throughout the year, and while the average was nine minutes and thirty-four seconds, the average visit length increased steadily from June and rose to a peak of 14 minutes and 49 seconds in November; probably coinciding with exam and assessment time. Just over half (54.2% of visits) were under 2 minutes.

When setting performance benchmarks for site duration the following elements should be considered:

• longer site duration is not always indicative of an improvement – it can indicate usability weaknesses which result in users taking longer to find the areas of interest or conduct ‘tasks’ on the site
• duration metrics should be reviewed at a specific content level – longer interaction in the forums section is likely a positive indication of engagement
  

Motivations For Use

The Studyit service attracts good levels of usage and repeat visitation across the student user base. The majority of students comment on the contribution that the site has on their NCEA performance.

The Studyit website is predominantly used for exam preparation, reinforced by the spike in visitation levels during November of both 2004 and 2005; though one in four students visit the site regularly throughout the year. These students were the most likely to cite frequent visitation – 79% of this group visit the site at least once a week, compared with around one quarter who do the same among those who ‘dip into’ the site at exam time or to answer a specific question.

There was no difference between older and younger students or any significant differences among students in lower vs higher decile schools. However, male students are more likely to visit for an answer to a specific question or issue (31% vs 19%) and females more likely to use at exam time (59% vs 44%).

Type Of Learning Encouraged By Studyit

Thinking

The site provides several areas where students can explore each of the subjects covered on the site (Biology, Chemistry, English, Maths, Physics and Science) in detail and, in fact, the ‘subjects’ category is the most used section of the site. 

The site includes subject areas which provide students with additional information in each subject area’s achievement criteria, a breakdown of content to be learned and tips and other options for students to explore the subject matter.  These site areas also provide students with resources to delve into subject matter in more detail (beyond achievement criteria level).  These subject areas also include links and practice areas, which students can visit for additional revision – however, these sections achieve little visitation (eg, only 3.1% of students had used the past exam papers).

Making Meaning

After the forums, site analysis indicates that the subject areas of the site tend to be the most heavily used parts of the site.  As a proportion of overall page impressions, the math subject area accounted for 19.3% of page impressions in August 2005 and 11.3% in November 2005.  Their relatively heavy use suggests that students are making use of the site’s content to dig deeper into different subject areas, though using the page impressions metric to determine this does not provide a complete picture of how the information is actually being used and how much time students are actually spending in each subject area.  The site statistics are consistent with the popularity of maths among site users (as found through the survey).

As found in the survey, more unique visitors use ‘subjects’ than the forums, though it is not surprising that the forums see a greater number of page impressions based on the volume of pages in this section.

Relating To Others

The site’s format, where students are provided with a framework (the forums) within which they communicate with others, provides them with a structure where they can interact with their peers on many levels. The forums are the most heavily used parts of the site, based on the number of page impressions.  As a proportion of overall page impressions, the forums accounted for 38.5% of page impressions in August 2005 and 49.9% of page impressions in November 2005 though the survey has found that just one in three students have used the forums. Among these users, however, the popularity of this type of content and interaction is clear.

The structure of the forums allows students to take a variety of roles in group situations, where students can comment on other student’s problems and provide support on queries.  While few users make regular posts, the survey provides evidence that reading the posts can be just as satisfying and valuable for students.

Managing Self

The site encourages students to explore at their own pace, with no specific direction provided for ‘expected’ activities to be performed on site.  Students are thus provided with ample opportunity to take ownership of managing their own time and activities on the site.

Feedback from students (provided directly to Studyit administers either directly via the site or via email and forwarded to Nielsen//NetRatings in hard copy) indicates an appreciation for the support the site provides to them in their studies, and an acknowledgement of the help provided on the site.  Suggestions were made for expansion of the site’s forums into other curriculum areas (including geography, economics), indicating students’ continuing interest in self study.  Comments also seem to indicate that participation in the forums on the site increased students’ own self confidence (“thank you ST {name of moderator} for being awesome!! I feel a lot more confident now ...”, “I've got the hang of it now!”).  Clearly, there exists a strong support base for the site, and these students could be harnessed as ambassadors to encourage other non-users to visit the site’s resources.

Overall, the site behaviour statistics reveal good growth in usage among students and the task is now to encourage greater repeat visitation outside the core group of regular users. The site itself is a great teacher of self management and this is likely the reason for overrepresentation among high achievers (as found through the survey).

Participating And Contributing

The forum structure of the site allows students to provide feedback and encouragement to other students as they prepare for the completion of their NCEA qualifications, giving them the opportunity to participate in the wider community of students beyond the classroom environment. 

Forums and the ‘chat’ environment online often make heavy use of abbreviations which can be hard for outsiders/new participants to understand.  While these same abbreviations are used in mobile phone SMS, there may be children whose lack of abilities in this area may put them at a disadvantage in the forums. 

The survey has found the need to encourage greater levels of usage of this section of the site – either for posting or reading – and particular encouragement among those from lower decile schools. Introduction to the forums could take place through additional promotion and education for teachers, highlighting the benefits of the forums and the safety of Studyit’s teacher-moderated forums. An interactive or animated forum intro / demo is also an idea for consideration for the site.

Footnotes

7. Please treat this finding as indicative only, based on a small sample of infrequent users who answered this question (only 10); the 77% among frequent users is accurate though, based on a sample of n=39
8. Nielsen//NetRatings New Zealand eGeneration Study 2005 - 2006
9. Note, all site measurement statistics quoted are from the month of November 2005; selected as a sample month for analysis
10. This may be an issue with the prominence of the function. CORE Education’s testing may shed some light.
11. “Libraries using Web to snare teens”, Christine Wallgren, The Boston Globe (www.globe.com), 15 December, 2005