Evaluation of Student Facing Web-Based Services: Studyit (Nielsen/NetRatings)

Quality of Service Provision

• In this section, we explore students’ perception and satisfaction with Studyit.co.nz, including that associated with specific areas and elements of the service.
• We also explore the impact of the service on students’ performance and their overall perceived usefulness of the site.
• Finally, we look at the reach of the service to students who are diverse by geographical location, ethnicity, language, social status, ability, access to technology, and age; and the means by which they become aware of the site.
  

Perceptions of Service Performance

Satisfaction

Students visiting the Studyit website reveal high levels of satisfaction and only 6.4% display dissatisfaction. Students who use the site more often and who have been using the site for a longer period of time are far more satisfied than first timers and infrequent users; and interestingly, those who visit regularly throughout the year are slightly more satisfied than students who visit mainly for exam preparation or with a specific question (94% vs 84%).

Overall, four in five students (79.2%) are satisfied or very satisfied with the site which is a strong result relative to many websites evaluated by Nielsen/NetRatings across a variety of industries and audiences.

 

Students within the key target age for the service, ie, 16 to 19 year olds, are marginally more satisfied than younger students aged 10-15, 79.1% compared to 72.0%; and males are more satisfied than females, 84.3% compared to 76.8%.

It is positive to note that students from low decile schools display similar levels of satisfaction with the service than their counterparts attending higher decile schools. The proportion who are ‘very satisfied’ is lower among those in 1-3 decile schools vs 8-10 decile schools (16.7% vs 27.1%) but combining ‘satisfied’ and ‘very satisfied’ measures reveals 75% for lower decile students vs 79.5% for higher decile students. There was no difference between mid decile and high decile students.

Contributing to students’ high levels of overall satisfaction are perceptions of specific elements of the service. As seen in the chart below, students are the most satisfied with the ability to access the website at all times – providing them with study assistance and the facilitation of communication with peers at all hours. The study support value of the discussion forums and the ability to post questions to teachers is also an area of high satisfaction for service users and one which differentiates the service from other tools.

 

Older students (16-19 years) exhibit higher satisfaction with all components of the site than do their younger counterparts. While the 24/7 availability is important to this group, aspects of the site relating to its relevance to NCEA assessments and the facilitation of support, communication and ‘connection’ with other students and teachers appears to be a valuable service for students in years 11 through 13.

 

	All Students N=365	Students 10-15 N=75	Students 16-19 N=296
The site is available all the time (24/7)	4.47	4.23	4.52
The study support value I get out of the discussion groups/forums	4.23	4.17	4.23
How up-to-date the site information is	4.13	4.06	4.13
How easy the site is to use	4.01	3.86	4.05
The relevance of subject information to what I am asked to do at school	3.99	3.84	4.02
The relevance of the information to NCEA assessments	4.09	3.84	4.15
The study advice provided on the site	3.97	3.82	4.00
How well Studyit fits in with my overall study efforts	4.00	3.81	4.04
The speed and responsiveness of the site	3.90	3.78	3.92
The ability to pose questions to teachers	4.15	3.70	4.25
The general NCEA information provided	3.96	3.64	4.04
Being able to offer support and advice to other students	3.96	3.56	4.05
Communication Tools (e.g. the forums)	3.96	3.55	4.05
The look and feel of the site	3.80	3.54	3.86
Being able to connect with other students	3.77	3.40	3.84

Among males and females, levels of satisfaction are fairly similar. The two exceptions relate to the study support value that the discussion groups/forums provide and the ability to connect with other students – females are more satisfied with these elements of the site, than are males. It is interesting to note that where satisfaction with specific components of the site is similar, males are more likely to give a higher rating for their overall satisfaction with the site.

Among students from low decile schools, the website availability (24/7) is favourable, as is its ease of use and the ability to pose questions to teachers. These students are also more satisfied than others, with areas such as the site’s study assistance and relevance, and the ability to connect with other students. They are only slightly less satisfied with support through the forums and the communications tools generally, than are their counterparts attending mid and high decile schools. Indications are that lower decile students make less use of the forums on Studyit (25% vs 35% of those from higher decile schools) which potentially stems from lower confidence in communicating with, and asking questions of peers¹. These students should be encouraged to make use of these communication tools to elicit more benefit from the service. 

 

	All Students N=365	Low decile  1-3 Students N=121	Mid Decile 4-6 Students N=254	High Decile 7-10 Students N=254
The site is available all the time (24/7)	4.47	4.62	4.55	4.41
The study support value I get out of the discussion groups/forums	4.23	3.70	3.75	3.87
How up-to-date the site information is	4.13	4.09	4.18	4.12
How easy the site is to use	4.01	4.42	3.98	4.01
The relevance of subject information to what I am asked to do at school	3.99	4.13	4.02	4.00
The relevance of the information to NCEA assessments	4.09	4.22	4.10	4.10
The study advice provided on the site	3.97	4.27	4.07	3.93
How well Studyit fits in with my overall study efforts	4.00	4.17	4.03	4.02
The speed and responsiveness of the site	3.90	3.91	3.81	3.99
The ability to pose questions to teachers	4.15	4.29	4.24	4.12
The general NCEA information provided	3.96	4.17	4.03	3.97
Being able to offer support and advice to other students	3.96	3.84	4.05	3.94
Communication Tools (e.g. the forums)	3.96	4.00	3.94	4.06
The look and feel of the site	3.80	3.96	3.78	3.80
Being able to connect with other students	3.77	4.14	3.73	3.83

 

Of great interest (and as mentioned earlier) is the finding that satisfaction levels increase for all aspects of the site the more frequently users visit the site. While this is not surprising given that higher satisfaction is likely to stimulate repeat visitation, it is also important to ensure that first time users are satisfied with the site from the outset to encourage further visits.

The chart below reveals the proportion of students who were either ‘satisfied’ or ‘very satisfied’ across each of the Studyit website sections and functions. “Emailing a teacher via Studyit” has the highest satisfaction rating of all the sections at 90.5%, while “Communicate” has the second highest satisfaction rating at 84.5%. Please note, however, that only 21 students responded to the ‘emailing a teacher’ rating question and this finding should be treated as indicative only, due to the small sample size.

The three sections with the highest usage “Subjects”, “NCEA Requirements” and “Study and Exam advice” achieve slightly lower measures of satisfaction while the little used “Encouragement” section (only 17% of students had visited this area) is slightly lower again, at 70% satisfaction.

The average score (out of a maximum 5) is also noted for each section.

Across the various student profiles some subtle differences are evident in their satisfaction with specific areas of site content; namely that:

• Older students are more satisfied with “Communicate” (49% are very satisfied and a further 36% satisfied) vs younger students of which 38% are very satisfied and the same proportion satisfied
• Interestingly, students aged 10 through 19 reveal similar levels of satisfaction toward the “Study and exam advice” section and 16-19 year olds are only slightly more satisfied with the “NCEA requirements” section
• Males are slightly more satisfied than females with the “Subjects” section (78% very satisfied or satisfied vs 71%) and females are more likely to feel neutral toward the category (33% vs 22%)
• Females reveal more satisfaction with the “Communicate” section (ie the forums) – 52% are very satisfied and 35% satisfied compared with males, 43% very satisfied and 38% satisfied
• Females are also slightly more likely to be satisfied with “NCEA requirements” than are males
• There is indication that females are more satisfied with the “Encouragement” section²
• Indications are that those in lower decile schools display similar levels of satisfaction with all of the main categories of content³ as their counterparts in mid and higher decile schools
  

Further charts relating to student satisfaction with the site and its individual areas can be found in the Appendices.

Perceived Contribution On Studyit On Students’ NCEA Performance

Among students using the site, 93% are studying for NCEA; just under half studying Level 3 NCEA.

Of NCEA students, over half (54%) feel that Studyit made a contribution to their performance.  9% feel the site made a very big contribution while 45% feel it had made some contribution.

Males and females felt very similar toward the service’s contribution to their achievements and as seen in the table below, it is positive to note that indications are that students in low decile schools are more positive toward the service having a big contribution to their NCEA performance than are mid and high decile students, though there was a much higher proportion who ‘didn’t know’ what contribution had been made.

High decile students are more likely to feel negative toward Studyit’s contribution to their performance; close to one in five (or more) feel it is not making much of a contribution or had not contributed to their NCEA performance. This correlates with some of the satisfaction ratings for various elements of the site – high decile students were less likely to be satisfied with the study advice and general NCEA information provided, and with the ability to pose questions to teachers, than were those in low and mid decile schools.

 

	Low decile students  N=24	Mid decile students  N=133	High decile students  N=161
Studyit made a very big contribution to my NCEA performance	16.7%	10.5%	6.2%
Studyit made some contribution to my NCEA performance	29.2%	48.9%	42.2%
Studyit didn't make very much of a contribution to my NCEA performance	16.7%	10.5%	21.1%
Studyit did not contribute to my NCEA performance at all	8.3%	16.5%	18.6%
Don't know	29.2%	13.5%	11.8%

 

With respect to achievement levels in specific subjects, a pattern emerges whereby those who achieved NCEA with excellence or merit are generally more positive toward Studyit’s contribution to their performance. This holds true for Maths and Science but is less pronounced for English, possibly due to the type of learning that takes place for English as opposed to subjects like Maths and Science. Aside from Maths though, English is the next popular subject and just over half of all students use the site as a learning tool for this subject (refer section ‘Specific subjects used’ for more detail).

Results for the three subjects can be seen below: 

 

MATHEMATICS	NCEA Students N=348	Not achieved N= 18	Achieved N= 107	Achievement with merit N= 134	Achievement with Excellence N= 89
Studyit made a very big contribution to my NCEA performance	9.2%	0.0%	6.5%	9.7%	13.5%
Studyit made some contribution to my NCEA performance	44.8%	33.3%	39.3%	48.5%	48.3%
Studyit didn't make very much of a contribution to my NCEA performance	15.8%	11.1%	15.9%	16.4%	15.7%
Studyit did not contribute to my NCEA performance at all	16.4%	22.2%	16.8%	14.9%	16.9%

 

SCIENCE	NCEA Students N=348	Not achieved N= 22	Achieved N= 90	Achievement with merit N= 130	Achievement with excellence N= 106
Studyit made a very big contribution to my NCEA performance	9.2%	0.00%	10.0%	7.7%	12.3%
Studyit made some contribution to my NCEA performance	44.8%	22.7%	37.8%	51.5%	47.2%
Studyit didn't make very much of a contribution to my NCEA performance	15.8%	9.1%	17.8%	14.6%	17.0%
Studyit did not contribute to my NCEA performance at all	16.4%	27.3%	14.4%	14.6%	17.9%

 

ENGLISH	NCEA Students N=348	Not achieved N= 22	Achieved N= 133	Achievement with merit N= 117	Achievement with excellence N= 76
Studyit made a very big contribution to my NCEA performance	9.2%	4.5%	9.8%	10.3%	7.9%
Studyit made some contribution to my NCEA performance	44.8%	31.8%	43.6%	46.2%	48.7%
Studyit didn't make very much of a contribution to my NCEA performance	15.8%	18.2%	15.0%	13.7%	19.7%
Studyit did not contribute to my NCEA performance at all	16.4%	13.6%	14.3%	19.7%	15.8%

Perceived Usefulness

Over 43% of students feel Studyit is a more useful website than others they have used. A further 32% believe it is ‘about the same’ which leaves just under a quarter who feel it is less useful or they are not sure. Please note, in-depth examination of the variety of resources used by students (both online and offline) can be found in the section, Alignment and Transfer of Learning, further in this report.

Not surprisingly, students within the site’s core target age (16-19 years) are slightly more likely to hold it in higher regard compared with other online resources they use for their studies. Over three quarters of 16-19 year old students (78.1%) find Studyit to be as useful as, or more useful than, other websites used for schoolwork.

 

Operational Characteristics Service Reach

Demographic Profile

Studyit.co.nz users are predominantly in School Years 11 to 13, with final year students (Year 13) making up the largest group of students at 39.7%. This representation is consistent with the site target audience and its objective of assisting students with their NCEA exams, assessments and preparation.

 

As noted in the Background section, two thirds of student users are female.

Students predominantly state that English is the language most often used at home (85.6%) but a total of 27 different languages were specified overall (as seen in the chart overleaf). Maori is mainly spoken by less than 1% of the user base which is below the New Zealand population rate of x%. As found in the Expert Usability Review activity, there is no option to receive the Studyit content in languages other than English.

It is positive to note though, that the 14% not speaking English as the main language at home are only slightly less likely to make use of the forums (30% vs 35%) and they are more likely to be ‘very satisfied’ with the site (33% vs 27%) and more satisfied with the ability to connect with other students (33% vs 22%). Satisfaction with ‘ease of use’ was very similar across the two groups and those speaking another language are slightly more satisfied.

This provides indication that there are no barriers to ease of use among those of a ‘non English speaking background’.
 

 

 

Level Of Education And Achievements

Correlating with students’ age and school year, over two in five students are currently working on level 3 NCEA, as seen in the chart below.

 

 

The table below shows students’ level of achievement for three core NCEA subjects. Studyit students are “Achievers” – with just 5.2% of students using the website not achieving NCEA Maths and 6.3% of users not achieving NCEA for Science and English.

 

	Not achieved	Achieved	Achieved with merit	Achieved  with excellence
NCEA – Maths	5.2%	30.7%	38.5%	25.6%
NCEA – Science	6.3%	25.9%	37.4%	30.5%
NCEA – English	6.3%	38.2%	33.6%	21.8%

 

Across gender, males are achieving higher levels of NCEA in both Mathematics and Science, whereas females are achieving higher levels of excellence in English. This follows males’ and females’ use of the site for specific subjects, though females were stronger users for Science. A table of these results can be found in the Appendices.

Representation Across School Decile

The chart below plots the school deciles of students visiting the Studyit website. The website does not seem to be reaching many students in the lower deciles one through three, and is over-represented in the top two deciles, 9 and 10.

As seen in the section, Sources of Awareness, there is far less encouragement and promotion of the site among teachers in lower decile schools and this is an important finding from the research.
 

 

Accessibility

Almost all student users (95.7%) have access to the Internet at home – this is higher than seen among the general population of 15 to 17 year olds in New Zealand of which 78% have Internet access at home. Almost three quarters of all student users of Studyit have access to the Internet via the school computer room (74.1%) and 73.6% have access through the school library (73.6%). Internet access is mirrored for both students of 10-15 and 16-19 age groups. The one exception is location “somewhere else” where 20% of students aged 10-15 years have access compared to half the proportion (10.8%) of students aged 16-19 years.

Students attending lower decile schools are less likely to have Internet access at home compared to students in mid to high decile schools, but they are more likely to have access to the Internet in the classroom and ‘somewhere else’ than other students. This may relate to relatives’ or friends’ homes.

 

 

These findings are consistent with those from Nielsen/NetRatings’ NetWatch Internet Audience Monitor in which Maori children have been found to have less access to the online medium than other children of the same age; particularly in relation to home access (49% vs 78% of 15-17 year olds).⁴

Two in three (65.7%) student users also have broadband Internet access at home. This compares to just 22% of the total New Zealand population⁵ who have home access to broadband and reinforces that the user group is largely comprised of a high access user base. Not surprising given the cost of broadband, high decile students are more likely to have broadband at home (70.8%) compared to just over half (52.4%) of lower decile students.

However, those with broadband are not more satisfied than those who access Studyit via dial up – which is an important finding considering the fairly low level of broadband penetration within New Zealand households and the access differential seen between higher and lower income homes.
 

 

Sources Of Awareness Of Studyit Service

Teachers are by far the greatest source of awareness of the site for student users, followed by search engines and then word of mouth from peers. Offline advertising plays an extremely minor role. This is a far different story to that seen generally for website awareness whereby word of mouth from friends is stronger at 46%, search engines are slightly less important (14%) and teachers are far less influential – just 5% of children and young adults mainly find out about websites this way.⁶ 

An important finding is that students from higher decile schools are almost twice as likely to find out about the Studyit website from a teacher compared to lower decile students, 59.0% compared to 33.3% for students from a decile 1-3 school. Lower decile students are more likely to find the website through an Internet search or from a friend. This highlights a significant difference in the encouragement given by teachers in differing types of schools. 

While there are few differences between younger and older students, recommendation from friends is far more important for older students’ awareness of Studyit (14% vs 4%) while search is slightly more prevalent for younger students (31% vs 24%).

Female students are more likely to find out about the site by using a search engine than males, 28.7% vs 18.2%.
 

 

 

Footnotes

1. These results are indicative only based on a low base of lower decile students (n=24)
2. Samples are fairly low for this section, n=41 females and n=21 males
   
3. Samples for lower decile school students are particularly low in some areas; n=19 Study and exam advice, n=5 Encouragement, Communicate and NCEA Requirements, n=14 Subjects
4. Nielsen//NetRatings NetWatch, Q3 2005
   
5. Nielsen//NetRatings NetWatch Market Report, Q2 2006
6. Nielsen//NetRatings eGeneration NZ study, 2005-2006