Evaluation of Student Facing Web-Based Services: WickED (Nielsen/NetRatings)

Findings And Observations

The findings and observations derived from this research and explored in this report cover four specific areas:

1. Quality of service provision
2. Immediate learning for young people
3. Alignment and transfer of learning
4. Learning for teachers and other adults

Each section of the findings and observations explores one of the four key areas outlined above and is intended to provide insight into how the usage patterns, behaviours and perceptions of site visitors interacts with the key goals for the WickED service and, more generally, those of the Ministry of Education.

Where possible, this report has incorporated findings, knowledge and general commentary based on Nielsen//NetRatings’ expertise in the field of Internet and technology. Particularly, learning’s from market reports and non-confidential research concerning youth have been incorporated in some instances to supplement the primary research findings of the WickED evaluation.

Quality of Service Provision

• In this section, we explore students’ perception and satisfaction with the WickED site, including that associated with specific areas and elements of the service.
• We also assess the reach of the service in terms of demographic profiles, student confidence and Internet accessibility
• We reiterate the findings relating to the safety of the WickED website for young users
• And provide an understanding of the means by which student users find out about WickED

Perceptions of Service Performance

Close to two in three student users of WickED (63%) are ‘happy’ or ‘very happy’ with the site and this is impacted by:

• Frequency of visitation: 79% of those who visit weekly or more often are happy or very happy vs 36% of less frequent users and 55% of first time users
• Students’ motivation for visiting the site: those who visit “to help me learn”, “used WickED within a class session” or “to get ideas for school questions or projects” are more likely to be happy or very happy with the site than those visiting for fun, to get answers for school questions, those who were just curious or had been told to use the site by a teacher/other adult. This is a positive sign that both independent and teacher-led usage can both result in satisfaction among young users.

Overall satisfaction level of WickED users

Interestingly, characteristics such as site tenure, Internet sophistication, home connection speed and primary vs secondary students had little to no impact on overall satisfaction with the site.

Depicted in the chart below, girls are happier with the WickED website than boys (69.1% happy or very happy compared with 51.1% of boys). There is little difference between the three decile groups when comparing happiness ratings, though a higher proportion of students in higher decile schools are very unhappy with the website compared to students in the lower and mid decile schools.

As seen in the chart on the previous page, more than one in ten users (13.7%) are unhappy to some degree with the site and close to a quarter are ‘sitting on the fence’. The 37% who are less than happy overall are only slightly more likely to be outside the target age ie, over 12 (39% of those in year 7 or above vs 32% in year 6 or below) but first time users are far more likely to be less than satisfied (46% vs 24% of repeat visitors). This can be improved with greater ‘hand holding’ and guidance through the website with dedicated links and content for ‘new users’ and welcome messages to introduce students to what is on offer on the site and how they should start using the site.

Evaluating specific elements and features of the site, students are the most satisfied with the ability to work by themselves, achieving an average score of 3.76 out of 5. The variety of content / activities, degree of ‘fun’ and ease of use also rate fairly well though all areas reveal room for improvement, to lift these scores closer to the maximum of 5.

As noted in the Interim Report, comments from the August 2005 student workshops indicated that navigation was a possible area of weakness for the site and the survey findings reinforce this to some degree – over half of users are satisfied with the site’s ease of use, but 30% are neither happy nor unhappy and 19% are less than happy. Continued monitoring of site usage patterns via the behavioural statistics as well as ongoing usability assessment can assist in refining navigation and overall usability practice for the site.

 

Please note, a detailed chart of this question can be found in the Appendices.

Consistent with the findings relating to overall satisfaction with WickED, girls are happier than boys in all aspects of the website, particularly “ease of use”, “fun” and “how much there is to do on the site”. Students from low and mid decile schools also rate all aspects higher than those at high decile schools, notably the way the site improves their learning and helps with school work. Detailed average scores can be found in the Appendices.

Not surprisingly, the more frequently students visit the site, the more likely they are to be satisfied across all aspects measured.  The table below reveals average scores out of a maximum of 5 – split between first time visitors and repeat visitors.

	All students N=146	First time users  N=88	Used two or more times  N=58
Being able to work by myself.	3.76	3.57	4.05
How much there is to do on the site	3.58	3.36	3.91
How much fun I have	3.53	3.32	3.84
How easy the WickED website is to use	3.5	3.26	3.86
How much it helps me improve my learning	3.46	3.22	3.83
How much it helps me with school work	3.41	3.24	3.67
How fast the WickED website is to use	3.36	3.16	3.66
How much it helps me to work with other students	3.29	3.13	3.55

Satisfaction with specific content areas of the WickED site

Many content areas achieve good levels of satisfaction though care should be taken when looking at these results as many have very small base sizes. Any element with a base below 30 should be treated as indicative only and those with less than 10 responses should be treated with caution.

‘Maths Stuff’ achieves the highest rating (now found within ‘Homework Help – Maths Help’), and fares particularly well among repeat users (who are also the most likely to make use of this section of the site). Those at lower or mid decile schools are more likely to be ‘very happy’ with this content though when including ‘happy, higher decile students are just as likely to be positive overall toward ‘Maths Stuff’. Similarly, girls are far more likely to be ‘very happy’ but boys are just as likely to be positive overall (80% of boys are very happy or happy as are 77% of girls). A detailed chart can be found in the Appendices alongside the average score achieved by each section, out of a maximum of 5.

There are some differences evident in student’s satisfaction with content areas like Maths and interactive ‘fun’ depending on their motivation for using the site. Those who visit WickED to learn exhibit greater satisfaction with both of these popular areas of the site while those who have simply been directed to use the site are less satisfied. Classroom-based use elicits similar levels of satisfaction as independent use for both fun and to learn.

Students’ suggestions to improve WickED

Within the online survey, students were asked to suggest improvements to the WickED site, ie, “How could WickED be made better?” Overwhelmingly, students are after fun and interactive learning, taking full opportunity of the functionality of the online medium.

‘More quizzes’ is the most cited means of improving WickED among both boys (53.1%) and girls (64.9%) while competitions and activities with other students are also popular. This is consistent with some of the comments from attendees of the WickED Workshop, conducted in Auckland in August 2005, as quoted in the initial interim report by Nielsen//NetRatings:

• “I think there should be like a game that the player is some one the walks around in levels and to get to do the next level they have to answer a question".
• “I think you should have some dress up games where you can choose different people to dress up with different clothes and some animal games. Then it would be the perfect site.”
• “More games.”
• “The WickED site could make more games.”

Boys are more likely than girls to request functional improvements such as “better looks” or “make it faster” while girls were more likely to suggest the interactive content enhancements (“more quizzes”, “competitions”, “games” and “more activities to do with other students”).

Differences are also evident across age groups:

• 11 and 12 year olds are less interested in quizzes (51%) than 7-10 year olds (68%) and those over 13 years (69%)
• They are also far less interested in competitions (32%) vs 58% of 7-10s and 67% of 13+
• Those over 13 are the most likely to suggest improvements to the appearance of the site (51% vs 32% of younger users; they also request more interesting themes and topics, changing themes and more links to other sites; and more activities with other students – consistent with the shift toward online socialising that has huge uptake among teenagers

The chart overleaf summarises the suggested improvements.

These suggestions are consistent with current levels of satisfaction with the fun and interactive content on the site, ie, playing games, doing a quiz or a crossword. Two in five students have made use of these elements of WickED and over two thirds of these users are happy with it, though one in five are indifferent and just over one in ten are unhappy. This reveals more dissatisfaction than with ‘Maths Stuff’.

As seen in the chart below, indications are that boys are far less satisfied than girls, which is interesting, given that girls were more likely to suggest improvements to this type of content on the site.

Section satisfaction ratings - 'Played game/quiz/crossword'

 

Operational Characteristics

Service Reach

The following section provides an indication of the profile of students visiting WickED. As the survey responses were primarily gathered via an opt-in method of participation, the research has not provided a true, random capture of visitor profiles so service reach must be regarded as indicative. It is likely skewed toward students who enjoy participating in surveys and who want to provide their opinion of the site.

Demographic Profile

Consistent with the age breakdowns found in the Sample Composition section of the report, WickED users are predominantly in Years 6 to 8 (63.7%), with Year 7 students making up the largest group at 26%. Those in Year 8 are likely above the key target age for the site (ie 12 years of age).

There is far less use below Year 6 which may be a ramification of teacher-led use and introduction in the classroom (found to be higher among 11 and 12 year olds) as overall, those at the younger end of the target age i.e. 7 – 9 year olds, are underrepresented.

Current School Year

Interestingly, there is a large proportion of boys aged 12 using the site (37% of all boys vs 16% of girls the same age) but more girls aged 11 and under using the site (51% vs 38% of boys). This is an interesting finding and examined in more detail in the section, Patterns of Usage.

Main Language Spoken

79.5% of the student respondents stated English as the language spoken most often at home, slightly below that of Studyit (85.6%). A total 12 different languages were specified overall with Maori mainly spoken at home by 4.8% of student users, higher than the 0.5% of Studyit users. The WickED site includes Maori resources which should assist in making Maori speakers feel welcome and comfortable using the site.

 

Language does not appear to have an impact on students’ satisfaction with the site - 53% of those mainly speaking another language at home are happy overall with the site’s ease of use vs. 51% of those in English speaking households; and they are more satisfied with the way WickED helps them with their school work – 60% of those in a non English speaking home vs. 47% in an English speaking home.

Language usually spoken at home

 

Level of Education and Achievements

WickED is predominantly reaching students who are confident with their school results and as seen in the chart below, girls are more likely to feel they doing well at school than boys, who are far more likely to admit having some difficulties at school. There is little difference across the various age groups.

How well students feel they are doing at school 

Indicatively[1], more confident students are more likely to be repeat users of WickED (43% vs 18% of less confident students), more satisfied with the site overall (65% happy overall vs 55%) but not more satisfied with WickED’s facilitation of fun, independent learning and as a helpful resource for school work. In fact, less confident students are more likely to feel positive toward WickED in many of these attributes. Results can be seen in the table below:

This is a positive finding as it indicates that just as much value, if not more, is being obtained by less academic students in the vital areas of learning such as managing self, thinking and relating to others. Further detail on these core competencies can be found further through the report.

Representation Across School Decile

The below chart plots the school decile rating of students taking part in the study. Close to two in five (39.2%) students are in the top three decile groups while one in four (24.9%) attend lower decile schools and one in three (32.8%) attend mid decile schools. Although there is a slight under representation of lower decile students, access is far more evenly distributed than on Studyit (just 7% of students using Studyit attend lower decile schools).

This is testament to the recommendation from teachers at lower decile schools (just under half of lower decile students found out about WickED via a teacher, though it is lower than found for higher decile schools) as well as only a small divide in home Internet access, as discussed overleaf.

Student Decile Rating 

 

Accessibility

A high proportion (84.9%) of all student users of WickED have access to the Internet at home and at school (79.5%). Seven to twelve year olds have slightly higher access to the Internet at home than do teenagers (87% vs 80%) and WickED users have better access to the online medium at home than does the general New Zealand population of roughly the same age: 78% of 10-12 year olds in NZ have a home connection vs 87% of 7-12 year olds who use WickED. Access among these youngsters is also significantly higher than for the general NZ population at 68%.[2]

Access to the internet by location 

As seen in the chart above, a reasonable proportion also have access ‘somewhere else’ and students mentioned friends’ / relatives homes or at mum or dad’s place of work.

Again in contrast to the Studyit user audience, students attending lower decile schools are on par with their mid decile counterparts in terms of Internet access and only slightly below high decile students in their access to the Internet at home. However, while most locations of access are similar across the three groups, lower decile students are less likely to cite access in a community study centre which is an interesting finding given that WickED was originally developed for use in study and after school centres.

As explored in the next section, Patterns of Usage, there is little use of WickED taking place in study centres (less than 10% of student users overall) and 6.5% of lower decile students actually state they have accessed WickED from a study centre, which contradicts the 0% who felt they did not have Internet access from this location. Regardless, the proportion is very low considering the original remit for WickED.

15.1% of students do not have access to the Internet at home. Students from lower decile schools have less access to the Internet at home (19%) than students attending high decile schools (12.2%) and mid decile schools (17.1%).

Those who feel they are faring worse with their school work are also more likely to have less access to the Internet – 22% don’t have home access vs 13% of students who feel they are doing well / really well; 30% don’t have Internet access at school vs 18%; and only slightly more have Internet access at a study centre – 15% vs 11%.

Interestingly, there is very little difference between Internet access for first time users of WickED compared with repeat users – though first timers are slightly more likely to have Internet access at a study centre (15% vs 7% of repeat users) but less access via a public library (46% vs 53%).

Home Access Speed

Among all student users of WickED (inclusive of those without home Internet access) 43.8% have Broadband Internet access at home, translating to 52% among those who do have a home connection. This is well below the 65.7% of Studyit users with fast access but consistent with Nielsen//NetRatings’ findings that households with teenage children are more likely to have upgraded to broadband based on the children’s influence.

Among 7 to 10 year olds with a home connection, 39% have broadband and this jumps to 58% among 11 and 12 year olds – significantly higher than the 38% among the general NZ population of 10-12 year olds, based on Nielsen//NetRatings’ NetWatch data Q3 2006.

So while broadband access is higher than average among the current WickED user base, there remain 30% of home users who connect to the site via dial up (25.3% of all student users) and this remains a consideration for future site developments and content delivery. On some sites, a broadband and narrowband option is provided to users such that rich media is only presented to those on a fast connection. Regardless, future developments should be tested thoroughly in both a dial up and broadband environment.

While lower decile students are almost as likely to enjoy Internet access at home, they are significantly less likely to have broadband (52.4% of those with a home connection) compared with their higher decile counterparts (70.8%). This is likely due to the costs associated with this technology.

However, indications are that those with broadband are not more satisfied with WickED overall, with very similar levels of satisfaction exhibited among home broadband and dial up users, nor are they more satisfied with the speed of the site – 51% of home dial up users are ‘happy’ or ‘very happy’ with “How fast the WickED website is to use” vs 45% of those on broadband at home. There are, however, slightly more dial up users who are less than satisfied with the speed – 27% vs 20% of broadband users.

Safety measures

As noted in the interim report, Nielsen//NetRatings’ eGeneration study of 2005 confirmed that young New Zealanders have integrated the Internet into mainstream activities which strengthen their connections to their real world communities and enrich their social interactions with peers.  Using web-based tools as a way to engage with students allows educators to communicate with students in an environment that they have embraced as their own and encourage students to develop their educational pursuits outside of the classroom.

The reality of the Internet, however, is that students can be exposed to offensive content and risky situations, either as a result of an innocent search which leads students to questionable sites or by their own volition.  Sites catering to children and young people can support safe Internet practices both within their own designs as well as by providing information and tools to encourage children and young people to wisely navigate the internet while guarding against privacy invasions and authenticating online information.

WickED was found to provide a safe environment for children, and for the most part does not demonstrate practices which would put children at risk.  There were no remarks relating to an unsafe environment among student users or adult users of WickED, within the survey responses.

In terms of communicating to children about safety issues, the website demonstrates very good procedures, such as the liberal inclusion of links to Internet safety tips and guidelines – suggested for inclusion on the home page to enhance their overall visibility and encourage readership.

One area of the site that may warrant additional vigilance is the online forum – though as will be seen in the next section, less that one in ten student users have made use of the forums and the ‘communication and interaction’ components of the site are more attractive to teenagers, than to the younger target audience. 

 

 

 

 

 

 

 

 

 

 

Regardless, online communication via instant messenger, chat and forums can be a high risk activity for children as ‘stranger danger’ prevails and continued moderation and editing of the forum contents is required, as currently performed by the TK Angels. If there is to be an aim of increasing use of the forums, particularly among the target user audience (7-12 year olds) the resources put toward editing and moderation may be stretched and these issues need high consideration before any promotional activity occurs. This is integral to the safety of children using WickED going forward.

Further detail relating to best practice in supporting child safety online for websites geared to children and young people can be found in the Interim Report which complements this document.

Sources of Awareness of the WickED Service

Teachers are a great source of promotion of the WickED site and more than half of all students (55.5%) had found out about the site through a teacher – more so among 11 and 12 year olds of which 60% had found out via a teacher, consistent with their greater use of WickED in a class lesson.

Search and word of mouth among students are also important forms of site awareness; search being more prevalent among the 13+ group and hearing about the site from a friend a more common source of awareness for 7 to 10 year olds.

In a similar pattern to Studyit, though to a lesser extent, teachers at high decile schools appear to be more avid promoters of WickED - 67.3% of students in high decile schools (8-10) first found out about the site through a teacher compared to 45.2% of students from low decile schools and 48.8% for mid decile students. This is a significant difference and highlights the need to improve awareness levels and encouraged use of the site in lower decile schools.

Boys are significantly more likely to find out about the site by a teacher compared to girls, 65.3% compared to 50.5%; while girls are slightly more likely to find out about the site by using a search engine (12.4% compared to 8.2%).

How did students find out about WickED?

 

Footnotes