NMSSA 2021: Technology - Key Findings Publications
Publication Details
The National Monitoring Study of Student Achievement (NMSSA) is a national sampling study designed to assess student achievement across the New Zealand Curriculum (NZC) at Year 4 and Year 8 in English-medium state and state integrated schools. The study is organised in five-year cycles, with all eight learning areas of the NZC monitored within this period. The first cycle ran from 2012 to 2016. The 2021 study represents the culmination of Cycle 2. As part of the 2021 study, NMSSA monitored achievement in the technology learning area. The study assessed achievement and also collected contextual data using questionnaires for students, teachers and principals. The last NMSSA study involving the technology learning area was in 2016.
This report is designed to provide a succinct overview of key findings related to technology from the 2021 study. The report is supplemented by a report focused on curriculum insights for teachers, a technical report, and an online interactive statistical application. All reports and the interactive application can be found on the NMSSA website.
Author(s): Educational Assessment Research Unit and New Zealand Council for Educational Research.
Date Published: November 2022
Executive Summary
Technology was introduced as a learning area in the NZC in 1995, and updated alongside all other learning areas in 2007. In 2017, the learning area of technology was revised to strengthen the positioning of digital technologies within the NZC. This change was substantial. It signalled the need for a greater focus on “students building their skills so they can be innovative creators of digital solutions, moving beyond solely being users and consumers of digital technologies”.
Interruption to the 2021 study
The 2021 NMSSA assessment programme was interrupted by a nationwide lockdown associated with COVID-19 that occurred midway through data collection in Term 3. This resulted in the entire programme being suspended for two and a half weeks. When the lockdown was over, NMSSA implemented a shortened programme in the schools that had not yet been visited and were still able to be involved.
This did not include schools in Auckland where the lockdown continued. The interruption meant that the national sample for 2021 is made up of fewer students from a smaller number of schools than originally intended. In total, about 1,200 students were involved in the study at each year level. The students represented 61 schools at Year 4 and 64 schools at Year 8. This compares with the original intention to sample about 2,200 students from 100 schools at Year 4 and 100 schools at Year 8.
The interruption to the programme and the impact to the sample means that care should be taken when interpreting results, especially for smaller groups in the study. Given this, NMSSA has chosen to limit some aspects of the reporting, especially for smaller groups in the sample such as Pacific students. While the reporting does provide results for Pacific students, these are not used to draw inferences at a national level or to make comparisons with other groups.
General impact of COVID 19
It is also important to consider the more general impact of the COVID 19 pandemic when interpreting the results from the 2021 NMSSA study. In the 18 months leading up to the study, schools, students and whānau had to cope with a considerable amount of disruption, including extended periods of remote learning. Over this time, schools had to prioritise how they used time and put more effort into maintaining student wellbeing and providing pastoral care. Although NMSSA cannot directly quantify any learning losses associated with disruptions caused by COVID-19, it is likely that they have had at least some negative impacts.
The assessment programme for the technology learning area
In 2021, NMSSA assessed achievement in technology using a revised version of the Technological Literacy (TELI) assessment that was administered in the 2016 study. The revisions included material focused on the new digital technology areas. This material was both integrated into existing tasks, and incorporated within new tasks.
The changes to the TELI assessment mean that results from 2016 and 2021 cannot be compared using the same scale. To provide a sense of change over time, this report looks at how raw scores have changed on items that were used at both time points. Reporting is not provided against curriculum levels because of the relative newness of the digital technologies content, and the fact that the structure of the NZC is currently under review.
Information about the contextual factors associated with learning in technology was collected via questionnaires. In total, 803 students at Year 4 and 795 students at Year 8 completed a computer-based questionnaire related to learning in technology; up to 18 students in each school. Up to four teachers in each school completed a teacher questionnaire, with both classroom teachers and specialist technology teachers invited to participate. In total, 125 Year 4 teachers and 166 Year 8 teachers responded. Principals from 54 schools participating at Year 4 and 50 from schools participating at Year 8 completed a principal questionnaire.
Key findings
Achievement in technology
The difference between the average scores for Year 4 and Year 8 students on the 2021 TELI assessment was 32 TELI scale score units. This indicates that, on average, students make about 8 scale score units of ‘progress’ per year, between Year 4 and Year 8 and represents an annualised effect size of 0.4.
Girls scored higher, on average, than boys at both year levels. The difference between the average scores for boys and girls was 5 scale score units at Year 4 and 7 units at Year 8.
Achievement varied within and across ethnic groups. Each ethnic group was made up of high and low achievers and, on average, the learners in each group made similar levels of ‘progress’ between Year 4 and Year 8. On average, students who identified as ākonga Māori scored lower than those who did not—by 12 scale score units at Year 4 and 14 units at Year 8. It is important to note that greater proportions of ākonga Māori, compared with non-Māori learners, attended lower decile schools and that achievement also varied with decile.
At both Year 4 and Year 8, students attending high decile schools scored higher, on average than students who attended mid decile schools. Students in mid decile schools, in turn, achieved more highly than those attending low decile schools. The difference between students attending high and low deciles schools at Year 4 was 21 scale score units. At Year 8, it was 20 units.
An indication of how achievement has changed overtime can be provided by considering how scores have changed on assessment items that were used in both 2016 and 2021. At both year levels students, on average, scored lower on most of the common items in 2021 compared with students in 2016. The size of the changes varied from item to item, with the median change being a 4 percentage point decrease at Year 4 and a 6 percentage point decrease at Year 8.
Contextual factors associated with learning in technology
Use of specialist teachers in technology
The majority of Year 4 classroom teachers indicated that they had sole responsibility for teaching their students technology (81 percent). In comparison, 10 percent of Year 8 classroom teachers indicated that they had sole responsibility, with the majority indicating either shared responsibility (e.g. with a specialist teacher or an external provider) or no responsibility.
Consistent with this finding, a higher proportion of Year 8 teachers than Year 4 teachers indicated that they were employed as specialist teachers of technology (24 percent of Year 8 teachers and 3 percent of Year 4 teachers).
Additionally, a higher proportion of principals from Year 8 schools indicated that their school provided technology teaching by technology specialists for students from other schools (38 percent of principals from Year 8 schools compared with 9 percent of principals from Year 4 schools). Results suggest that the two technological areas most often taught by specialist teachers at Year 8 are designing and developing materials outcomes (delivered by a specialist in 94 percent of schools) and designing and developing processed outcomes (delivered by a specialist in 88 percent of schools).
Specialist teachers of technology more frequently reported teaching the more established technological areas of designing and developing materials outcomes and designing and developing processed outcomes than classroom teachers. In contrast, classroom teachers more frequently reported teaching the two new digital areas than specialist teachers.
Attitude and confidence towards technology
Students at both year levels were generally positive about technology, with the majority of students selecting that they ‘agree quite a lot’ or ‘totally agree’ with statements such as ‘I like learning about technology at school’ and ‘I think learning about technology is important’. On average, Year 4 students were more positive than Year 8 students. Overall, boys were more positive than girls.
Students were generally confident about their ability to use technological ideas and processes. Overall, boys reported a higher level of confidence.
Almost all teachers who responded to the questionnaire agreed that teaching technology was important. Results suggest that teachers felt more confident about teaching technology than about assessing technology. This was particularly pronounced at Year 4.
Teaching and learning in technology
Responses from Year 8 students indicate that in 2021, learning in technology involved working with a variety of technologies. About three-quarters of students noted opportunities in hard/resistant materials (75 percent) and food technology/biotechnology (74 percent), while smaller proportions indicated they had been involved in electronics (38 percent) or media/graphic design (37 percent).
More students in 2021 reported involvement in each of a list of technologies than was the case in 2016. Most notably, the proportions of students who indicated they had ‘done’ computer programming/coding/robotics rose from 28 percent in 2016 to 54 percent in 2021, and the proportions of students who indicated they had ‘done’ media/graphic design rose from 19 percent in 2016 to 37 percent in 2021. Students at mid decile schools reported notably less frequent involvement in three of the technologies listed than students at high decile or low decile schools. These technologies were computer programming/coding/robotics, electronics, and media/graphic design.
Overall, Year 8 students reported more frequent opportunities to learn technology at school than Year 4 students. The opportunity to learn technology most frequently reported by Year 4 students was talking about and making models of their design ideas. At Year 8, the most frequently reported learning opportunities included exploring and working with different materials (e.g. textiles, food, word, or metal). Talking about their own and others’ work in technology was reported as happening frequently by students at both year levels. Consistent with the reports from students, Year 8 teachers reported that their students had more frequent opportunities to learn technology than Year 4 teachers. In general, teachers indicated that the students in their classes experienced each of the learning opportunities more often than the students themselves indicated they did.
A greater proportion of Year 8 teachers than Year 4 students reported that their students spent more than 40 hours a year learning technology at school. Students spending more than 40 hours a year was noted by 66 percent of Year 8 teachers and 21 percent of Year 4 teachers. No differences by decile were found in teachers’ reports of time spent learning technology.
Almost all principals were positive about teaching and learning in technology at their school. Principals were less positive about the quality of reporting to parents and whānau than they were about other elements of teaching and learning in technology.
Digital technologies
Most teachers at both Year 4 and Year 8 indicated that they had received professional learning and development (PLD) support focused on the digital technologies curriculum content. School-led PLD was the most common form of support for teachers at both Year 4 (75 percent of teachers) and Year 8 (66 percent of teachers). Approximately one-third of teachers also reported receiving support from the Ministry of Education’s Kia Takatū ā-Matahiko Digital Readiness Programme.
Teachers’ responses indicated a reasonable level of confidence with the new digital technologies curriculum content. At least 70 percent of teachers at both Years 4 and 8 indicated that they ‘agree’ or ‘strongly agree’ with statements related to confidence, such as: “I am able to teach the digital technologies curriculum content” and “I understand the revisions to the digital technologies curriculum”. It should also be noted that a non-negligible proportion of teachers indicated that they lacked confidence with digital technologies. Up to 26 percent of Year 4 teachers and 13 percent of Year 8 teachers reported that they ‘disagree’ or ‘strongly disagree’ with the confidence statements.
Most teachers reported that they had experienced some level of challenge in understanding the digital technologies curriculum content. The two factors identified by teachers as most challenging both involved using equipment to deliver aspects of digital technologies.
Almost all of the principals indicated that their school had, to some extent, updated their planning processes to incorporate digital technologies within the technology learning area (93 percent of principals in Year 4 schools and 92 percent of principals in Year 8 school). Overall, however, most principals reported that their schools had experienced some level of challenge in implementing the digital technologies curriculum content. The principals of schools participating at Year 8 were more positive about their school’s provision for student learning in the new digital technologies curriculum content than principals of schools participating at Year 4. Seventy-eight percent of Year 8 principals rated their school’s overall provision as ‘good’ or ‘very good’, while 56 percent of Year 4 principals gave the same rating.
Half of the principals indicated that teachers in their school were either ‘enthusiastic’ or ‘very enthusiastic’ about implementing the digital technologies curriculum content. Slightly higher levels of enthusiasm were noted at Year 8 than at Year 4.
Navigation
Contact Us
Education Data Requests
If you have any questions about education data then please contact us at:
Email:
Requests Data and Insights
Phone:
+64 4 463 8065