Analysis and Use of Student Achievement Data
This group of five studies reports on aspects of the AUSAD initiative in Mangere and Otara that is designed to improve the capacity of the schools to analyse, share and learn from their student achievement information.
Author(s): Helen Timperley in association with Linda Smith, Judy Parr, Jennifer Portway, Sarah Mirams, Suzanne Clark, Mali Allen and Jill Page. Report prepared for the Ministry of Education.
Date Published: 2004
This report is available as a download (please refer to the 'Downloads' inset box). For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
This group of five studies reports on aspects of the AUSAD initiative in Mangere and Otara that is designed to improve the capacity of the schools to analyse, share and learn from their student achievement information. The Mangere and Otara schools have a predominantly Pasifika school population. In Mangere the schools comprise 64.6% Pasifika students, 23.9% Māori, 7.1% Asian, 2.7% are European, and 1.6% other nationalities. In Otara, 71.5% of the students are Pasifika, 23.8% are Māori, 3.1% are Asian, and 1.3% are European. The studies have been designed to focus on different aspects of the initiative. The first three studies have a teaching and leadership focus and report on the following:
- The professional learning processes and resulting student achievement associated with analyzing achievement information;
- The classroom time involved in collecting some assessment data and the uses to which they are put;
- The impact of assessment information on teaching practice in writing.
The final two studies focus on the community and school interface and examine issues related to parents' and boards of trustees' relationships with the teaching professionals and include:
- Reporting to parents in Māori bilingual units;
- Governance understandings of their accountability role in relation to student achievement
Study One: Follow-up to Report on Sustainability of Professional Development in Literacy: Shifting the focus to Professional Learning.
This first study is a follow-up to the report on the sustainability of professional development in literacy (Timperley & Wiseman, 2002). The earlier report examined the in-school processes and student achievement over an eighteen-month period in seven schools following their participation in the ECPL professional development in literacy contract with Dr. Gwenneth Phillips. A key finding of that study was that significantly higher student achievement was evident after children had been at school for one year in two of the schools that regularly analysed their student achievement information and discussed it with the teachers as a group. The five schools with lower achievement focused their efforts on improving the quality of their programmes, independently of the analysis of the achievement information. The analysis procedures in the two schools with high student achievement involved plotting the children's text level on a benchmarked graph (known as the "wedge graph") according to the number of weeks they had been at school. The benchmarks indicated whether the children were achieving above, at or below expectations of average progress. The children's class and teacher were identified and teaching strategies were discussed either at the meeting or individually with the teachers for children who were falling below expectations. This report has been summarised in a booklet entitled, "Shifting the Focus: Using Student Achievement Information for Professional Learning" (Timperley, 2003).
During the year following the writing of that research report, the five schools with lower student achievement adopted the same analysis procedures as those with higher student achievement. The schools had varying involvement with the ECPL professional development over this year, but in most cases, the continued focus involved the literacy leaders who attended meetings once per term, rather than the Year One teachers directly. Issues of how to analyse the achievement data were introduced to the literacy leaders at these meetings.
The study reported here examined whether the changed processes in these schools with regard to the analysis of student achievement were associated with improved student achievement and how the teachers perceived the process. The second part of this study addresses the parallel issues of what might be involved in extending an evidence-based approach beyond the school so that schools share and learn from their achievement data and from one another.
It was found that all five schools regularly examined the text levels reached by the students in their class and syndicate and spent much of their meeting time targeting individual students and discussing the teaching implications. In all five schools, average student achievement scores rose, with this improvement significant in three of the schools. In four of the five schools, the achievement levels were not significantly different from those in the two higher achieving schools involved in the previous study.
A questionnaire survey of 33 teachers and follow-up interviews with 10 of the teachers and the five literacy leaders was undertaken to understand the extent to which they believed they learned from the exercise of examining the data and how it led to different teaching practices and higher student achievement. Nearly half the teachers rated their learning from the exercise very highly, with most of the rest giving a moderate rating. Only three teachers did not consider it helped them to learn much about teaching more effectively. The main reasons associated with high levels of professional learning were the way in which underachieving students were targeted and the support the teachers received to think about and adopt different teaching strategies.
Teachers were also asked if the process of examining the achievement data in a group where they and their students could be readily identified had any negative emotional impact on them. Only one teacher, who was new to the area and the school, indicated that this was the case. The main reasons given by others for the absence of negativity on their part related to feeling supported and being helped to teach better.
The impact of the process of shifting the focus from talking about programmes in the absence of achievement information to analyzing the data together at team meetings was different for different teachers. Some described it as a developmental progression and had difficulty remembering the transition, others remembered if very well. All, apart from the one teacher above, had since become unconcerned about the process because the professional learning benefits outweighed concerns. Conditions that were identified as important to achieve these learning benefits and improved student achievement were a high sense of teacher self-efficacy, an understanding that inflating achievement scores was self-defeating, a shared understanding within the group of the process and pedagogical approach, having an agreed benchmark clearly marked on the graph, and holding regular review meetings when student progress was plotted on the graph and discussed.
Ten principals, deputies and assistant principals were also asked about the conditions needed for these within-school processes to be extended beyond the school so that schools shared achievement data among themselves. The conditions identified included, professional and personal trust, professional self-confidence, a belief that the potential learning benefits justified the risks involved, and a perception that the benefits would be worth the effort.
A condition that few specifically identified, but was evident throughout their responses, was the willingness to become a professional learner. Many of the senior managers involved in these studies actively sought and constructed highly challenging situations from which they could learn. Their student achievement is improving as a result.
Study Two: Running Records and the Time to Teach
Running records provide rich diagnostic information on students' reading strategies and fluency. They were developed as an assessment strategy by Marie Clay (1993) to give teachers a tool to analyse young readers' decoding strategies. In many New Zealand schools, administering running records at regular intervals is an established practice throughout the school and their correct administration and use has become a key part of the Ministry of Education's recent literacy strategy (Ministry of Education, 2000). The diagnostic information is gained only through individual assessment of oral reading. This condition inevitably means that the assessment process takes considerable time. The potential exists for the administration of running records to become a routine practice, which takes up significant amounts of class time, but does not realise the full benefits. This study examined the time involved in administering the running records and the uses that were made of the results by teachers and management staff.
Eight Mangere and Otara schools that were participating in the AUSAD initiative were involved in this study. In each school, observations were made of a minimum of three teachers who, in most cases, administered two running records with the same child and followed each observation with an interview. The principal and the literacy leader responsible for the junior and senior schools were also interviewed.
Running records were administered at both senior and junior levels in all schools. With the exception of the junior department in one school, they were typically administered for all children in the class within a set period (approximately two weeks) of each school term because consistency was needed in those schools where the senior management collated the data. Considerable resources in terms of time were allocated to the administration of these individualized assessments. On average, over all the schools, approximately 20% of reading time was committed to this activity. This time commitment was difficult to justify in terms of the use made of the running record, which was primarily to establish a text level and to group students for instruction. Concerns about the efficient use of time were particularly applicable to fluent silent readers, for whom running records are unlikely to be an accurate indication of their reading ability. In many cases, it was apparent that the texts the children were reading were too easy, possibly reflecting low expectations of the students' ability.
There is considerable tension between obtaining diagnostic information for classroom teaching and the special requirements involved in collecting data for aggregation school-wide. When the data are to be used for the latter purpose, it is important that the teachers complete the running records with all students within a short time frame. This means that administering them is not motivated by a teacher's need to know about an individual child's reading strategies, but rather the need to complete the exercise within the time frame. Under these conditions, it is not surprising that establishing text level was the main use to which the running records were put.
One of the most powerful ways to promote both professional and student learning is to provide feedback on performance and how it might be improved. Few students received any feedback from their teachers and few teachers received feedback from management about their students' reading.
The implications of this study are that assessments need to be matched to the purpose, and that the value of the uses to which they are put must be carefully weighed against the time involved in their administration.
Study Three: Assessment and Its Impact on Written Language Teaching Practice
The purpose of this study was to understand the assessment information used by teachers of Year Two students to guide their writing programmes. Assessment was once thought of as a summative activity that was undertaken at the end of teaching a topic or unit of work and used to judge the achievement of a student in a specific topic at a specific time. Recent research into the nature of assessment as a more formative process that is designed to promote learning has reframed the notion for researchers, administrators, and practitioners. Teachers are now encouraged to evaluate student learning on an ongoing basis and to use the information to promote that learning.
Assessment policy in New Zealand reflects this shift in thinking. The Curriculum Update recently issued by the Ministry of Education in 2001 states that "The primary purpose of school-based assessment is to improve students' learning and the quality of learning programmes."
Written language provides many assessment challenges because of the complexity of the activity. Not surprisingly, there are many suggested frameworks for assessing student progress in writing that are aligned with different conceptions of effective written language instruction. Given this complexity, combined with the variety of demands of most classrooms, it is not surprising that the most significant assessment record has been described as a "log in the head", whereby the teachers gather assessment information anecdotally and informally throughout the school day and use it to plan their writing lessons.
A variety of frameworks were used to analyse the "logs-in-the-head" of nine teachers in three Mangere schools. All teachers had participated in the ECPL professional development with Dr. Gwenneth Phillips. Each teacher was interviewed prior to a lesson observation, was then observed teaching writing, then re-interviewed about the lesson. The teachers were asked about the assessments they used, their specific lesson and longer-term aims for three students, to describe the three students as writers and how they judged the success of their writing programme.
When asked what assessments were used in writing for the three students, the teachers were not particularly forthcoming. The most commonly nominated assessment (4 teachers) were independent writing samples but two of these teachers indicated that the samples were primarily for inclusion in the children's portfolio for the benefit of the parents and the two others were vague about how the samples were used. A very different picture emerged, however, when teachers were asked what led them to select particular aims for the children's writing lesson for that day, and how they would judge if children were on track to achieve identified aims at the end of the following term. Their responses to these questions revealed a much richer picture of observation, talking to children and looking at the children's work.
The most frequently nominated aim for the lesson was focused on the content or topic that the children were to write about and for the children to become independent writers. Another aim nominated by three teachers was related to production or quantity of writing. Identified skills were primarily at the lexical or word level, i.e. learning to write words. Only three teachers moved beyond the word level of skill complexity when describing aims. Two included text level comments but only one teacher contextualised these comments within a communicative / rhetorical purpose for the writing. The classroom observations confirmed that the lessons were consistent with the stated aims.
The concern we wish to raise is the knowledge the teachers had of the writing process on which the "log-in-the-head" template was based. For most teachers, their descriptions and their teaching practice were focused at a relatively low level of skill and decontextualised writing from its rhetorical or communicative purpose. Writing was about producing something independently with a focus on word-level skills. While some writing should be personal expressive largely for self, most writing in everyday situations is designed to communicate something to someone, so the communicative purpose and needs of an audience should be fore grounded.
It is increasingly recognised that reluctant writers are often reluctant because writing for them is a skill-based exercise devoid of communicative meaning. A parallel may be drawn with reading as a process of developing skills to decode words, rather than a set of skills designed to enable one to understand the author's intent (Phillips, McNaughton & MacDonald, 2001). The risks involved in this orientation to either writing or reading are the training of children who can decode and encode but do not understand the communicative purpose and its associated language resources.
When asked how they judged the most and least successful aspects of their writing programme, the majority of teachers nominated particular activities or how well they managed the programme. Student achievement as a criterion was rarely mentioned. The relative absence of student achievement as a judgement criterion means that it is unlikely that teachers will be focused on accelerating that achievement in the same way as many have in reading (see Study One in this report).
Study Four: Reporting to Parents in Māori Bilingual Units
This study focuses on reporting achievement data to parents in Māori Bilingual units in three South Auckland schools (they included two Year 0 - 6 schools, and one intermediate school). This research involves interviews with the teachers and parents. Bilingual units of the type studied in this research are part of a complex picture of initiatives in Māori language education. The Ministry of Education currently distinguishes five different forms of Māori language education: Kura Kaupapa Māori; Kura Māori (also referred to as Other Immersion Schools); Bilingual Schools; Schools with Immersion Classes; and Schools with Bilingual Classes. Added to these forms of schooling type are also classifications of the level of Māori language being used in the school or classroom. The bilingual units are classified according to the percentage of instruction in Māori. These include Level 1: 81 to 100 percent; Level 2: 51 to 80 percent; and Level 3: 31 to 50 percent (Ministry of Education, 2002). Schools are funded according to the level of Māori language being used.
In the context of addressing Māori educational under-achievement what do Māori language educational programmes such as the bilingual units offer? Answering this question is somewhat difficult for while there have been major initiatives to improve schooling outcomes, and significant research that suggests areas needing greater attention in school performance, there is in fact a great paucity of research about what happens in Māori language educational settings. It is difficult to apply research based on immersion programmes directly to bilingual units. The bilingual units fall betwixt and between full immersion in Māori and full immersion in English. Their purpose and individual histories would suggest that they are expected to meet parental expectations about offering Māori language instruction, provide more generalized support for Māori children in a mainstream school and demonstrate a school's responsiveness to its Māori parents and community. Some bilingual units may also describe themselves as whanau units with greater emphasis placed on the values and sense of belonging being provided for Māori learners. Some bilingual units probably represent a compromise between parental expectations for immersion or bilingual instruction, school responsiveness and actual availability of Māori language resources including a qualified teacher who is fluent in the Māori language.
In this study, bilingual units are characterized as units that aim for dual medium instruction. Recent New Zealand studies show that Māori parents choose bilingual education as a dual instruction process rather than total immersion because they also want their children to be taught English simultaneously. These parents believe that bilingual education will enskill their child in both English and Māori thus believing they are getting the best of both worlds for education (Doug, 1996; Keegan, 1996; McKinley, 2000).
The reporting process is a key part of involving parents in their child's education and is linked typically to informing parents about their child's academic achievement levels and strategies to continue the achievement. Reported achievement is dependent on quality assessment information. However, assessment processes in Māori education are seriously under-developed. (The assessment tool asTTle was not widely available at the time of this study). Assessment in a minority language such as Māori is more than just an educational assessment task as it involves complex issues of language, culture, intellectual development and translation. An additional difficulty experienced by the teachers in Māori bilingual units within a mainstream school setting is the pressure to administer both English and Māori assessments.
It is important that what is reported to parents is not a public relations exercise but serves to promote better student achievement. It has been shown that many schools report what they think the parents want to hear, and this tends to be the positive aspects of the child's learning. Māori parents, as for any other parents, need accurate information from the teachers on their child if they are to make informed decisions about their child's education and how to improve learning.
In this study, answers were sought to three questions, "What was reported and how was it reported?" "What conditions determined the reporting process?" and "What conditions influenced parental involvement?" Teacher-parent reporting interviews were observed, followed by separate research interviews of the teachers and parents. A summary of the findings in relation to each of the research questions follows. Firstly, the reporting process consisted of a brief (5-15 minute) parent interview in two of the schools accompanied by written information focusing primarily on achievement in English. Nationally benchmarked information was provided, although this was sometimes difficult to understand and in one school parents had access to this written information only during the interview itself. Portfolios of student work were used in all schools. Reporting on the child's achievement in Te Reo was rarely mentioned during the 21 interviews and when it was, the quality of the information depended primarily on the teachers' personal judgments.
Secondly, a mix of conditions determined the reporting process. The type of written report and the structure of the teacher-parent interviews were determined by the mainstream school to which the bilingual unit was attached. Unit staff did not always see these methods of reporting as appropriate.
The limited reporting of Te Reo Māori was related to the level of use of Te Reo in the bilingual units. This use depended on the child's competence in the language rather than depending on the unit's classification of the percentage of language instruction. The teachers did not believe it to be in the children's best interests to teach curriculum knowledge in a language in which the children were insufficiently competent to understand the content.
Formal assessments in Te Reo Māori were rare because few protocols were available and those that were available were considered inappropriate because of the differing amount of time devoted to Te Reo in different Māori medium settings. The bilingual unit teachers believed that their children would be disadvantaged by comparisons with students who participated in Māori immersion programmes. The units or individual teachers generally designed the assessment tools used to report Māori-related aspects of the instruction. Neither the participating parents nor their children were fluent in Te Reo Māori and there was some confusion about expectations of the students' attaining fluency in Te Reo Māori. Some parents thought that by sending their children to a bilingual unit, they would become fluent in Te Reo. Students whose parents were involved in this study did not actually become fluent in Te Reo. It appears that a spiral effect of not being fluent in Te Reo Māori prior to attending the unit led to little Te Reo Māori being used because the child could not understand the concepts in Māori. As a result, the concepts were taught in English, with the curriculum emphasis being more closely related to Tikanga Māori than to Te Reo Māori.
Thirdly, in relation to the question about the conditions influencing parental involvement in the reporting process, nearly all the parents reported that they were very satisfied with the reporting process, but half indicated that they would have liked more information on their child's competence in Te Reo Māori. All units held additional whanau hui during which parents were encouraged to find out about the programme and participate in their child's education in various ways. In many cases, the teachers placed greater value on this process than did the parents. The interviewed parents all expressed a strong commitment to being involved in their children's education irrespective of whether their own educational experiences were positive or negative, or whether their own parents had been involved in their education. Sometimes negative experiences and lack of parental involvement in the parents' own schooling provided the incentive for these parents to ensure the pattern was not repeated.
Study Five: Accountability and Governance Understandings of Student achievement
Recent legislative changes relating to governance responsibilities have demonstrated an increasing emphasis on understanding the impact of educational practice. It has become the board's responsibility to monitor student achievement against expectations for that achievement, and to review the effectiveness of the teaching and learning programmes in relation to the expectations. Given that the board does not have direct involvement with students and programmes at the classroom level, these responsibilities are inevitably exercised through an accountability relationship with the professionals.
The concept of accountability has many facets to take into consideration. Explicit understanding of the issue under consideration is necessary if being accountable is to have validity of purpose and to be an authentic activity for all stakeholders. This would mean boards of trustees had some understanding of what is involved in assessing students, what the data mean and how they can be used for accountability purposes. A potential complicating factor in the practice translation in the educational governance context is the substantial international literature that documents the limited participatory role typically played by lay governors in educational matters. In a situation of limited participation, it is unlikely that boards would be able to exercise their accountability role.
As part of the AUSAD initiative the Ministry of Education provided financial assistance and ongoing support to enskill board members in this role by arranging a number of learning meetings for board members with the broad purpose of promoting better understandings of professional reports that contain student achievement data. External contractors provided the training.
In this study, we sought to determine the understandings of the boards of trustees' accountability role with regard to monitoring student achievement and what actions might follow. In addition, we sought to find out how this accountability role was enacted in practice during the board meetings.
The intended study involved interviewing board chairs from all participating schools and attending their board meetings when the AUSAD initiative and / or student achievement information was discussed prior to the training, then following it. Competing priorities led to variable attendance of board members and a much slower start to the training than anticipated, resulting in only three boards taking part with no post-training interviews taking place. The study involved six interviews with board chairpersons and parent representatives, three teachers responsible for the AUSAD initiative and one Ministry of Education official. Nine briefing meetings and training sessions were also observed, together with three observations of board of trustees meetings.
The results indicated that the link between accountability and governance with regard to student achievement is a tenuous one for parent board members. The board responsibilities outlined in the legislation and nominated by our respondents implied that boards should have knowledge of student achievement, involvement in setting expectations for that achievement and monitoring likely contributions to it, such as the effectiveness of programmes. The reality of what happened in the board meetings, however, seemed a long way from this description. In one of the three schools, a description of achievement accompanied by a description of AUSAD activities at a very general level formed the basis of the accountability process. In the other two schools, AUSAD activities were described in the absence of the achievement information. Board members rarely participated in any discussion, but rather, adopted a listening role. They did not perceive the accountability role to be difficult or that the professionals were dominating. Rather, it was just the accepted way things were done.
Traditionally, silence on the part of Board members has been attributed to patterns of professional power play that exclude lay members of boards. In this study, the teachers indicated that they were very aware of this possibility and took steps to try to avoid it. Board members themselves confirmed that they felt free to ask questions of the professionals.
The problem probably lies more within the complexities of the task, rather than with any "failings" on either side. The professionals themselves were undertaking intensive training as part of the AUSAD initiative to describe and analyse achievement information, set appropriate expectations, and work out the implications for teaching and learning programmes. These are demanding tasks in themselves. It is challenging to expect boards of trustees to have adequate knowledge of the highly contentious issues surrounding achievement, target setting and monitoring of programmes to the extent that they could hold the professional to account.
These findings need to be set in context in the sense that the board training was in its early stages. The list of training topics itself provides an indication of the complexity of the task being undertaken. A different picture might be evident after more training has been completed. In the meantime, the accountability aspects of governance with regard to student achievement appeared to be enacted by the professionals going to considerable lengths to make their reports understandable to their boards who were then responsible for holding the professionals to account. The activity was primarily a professional one, while their boards listened to their descriptions and explanations in relative silence.
Where to find out more
For more information about this publication please email the: Research Mailbox