Evaluation of Teacher Professional Development Languages (TPDL):
For teachers of languages in years 7-10 and the impact on language learning opportunities and outcomes for students
This study was carried out during 2008 and aims to inform the Ministry about the TPDL and the impact on language learning opportunities and outcomes for students.
Author(s): Sharon Harvey, Clare Conway, Heather Richards & Annelies Roskvist, AUT University
Date Published: December 2009
This report is available as a download (please refer to the 'Downloads' inset box). To view the individual chapters please refer to the 'Sections' inset box. For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
Chapter 2: Design of study
The design for the research is a mixed methods approach which balances a broad brush picture of the gains and issues for teachers and students with some 'up-close' examinations of what happened in classrooms when language teachers undertook TPDL.
The research questions were as follows:
- What is the impact of the TPDL programme Years 7-10 on the development of teachers' fluency in the teaching language?
- What is the impact of the TPDL programme Years 7-10 on teacher second language teaching knowledge?
- What is the impact of the TPDL programme Years 7-10 on teachers' knowledge of the Learning Languages strand of The New Zealand Curriculum (2007) and specific curriculum guidelines?
- What is the impact of the TPDL programme Years 7-10 on student learning and outcomes?
- How sustainable and replicable is the TPDL programme Years 7-10?
As noted in the previous chapter, the concept of effective teaching will be addressed throughout the report.
Design of study
The study was divided into three phases coinciding roughly with the beginning, middle and completion of the TPDL course. The aim was to gauge the effects of TPDL on practices through the year. Unfortunately the late signing of the contract meant that teachers had already started TPDL by the time the first phase of research was underway. Each phase involved the deployment of three research instruments: a survey, a case study observation and a case study interview.
Phase one: Between April and early June 2008.
Phase two: Between July and September 2008.
Phase three: November 2008.
Data was gathered in the following ways:
- Three questionnaires were administered to all 2008 participants on the TPDL programme - one in late April/May, one in July and one in November (see Appendix Four, p.147). The questionnaires examined changing teacher language proficiency, practices, perceptions of the course and perceptions of outcomes for students in the 2008 TPDL cohort. They were constructed using the detailed questions in the RFP (Ministry of Education, 2007b) as a guideline as well as issues arising from key second language acquisition literature. These issues were: motivation, language input and output, language pedagogy, knowledge of language and knowledge of culture (Ellis, 2005a & 2005b; Erlam, 2005; Gibbs & Holt, 2003; Crozet & Liddicoat, 1997; Dornyei & Czizer, 1998; Harmer, 2002; Byram, 2007) The first survey collected baseline and biographical data while the second and third surveys included issues arising from the emerging case study data as well. Each questionnaire was assigned a code and was tracked through the code rather than through individual teachers' names. While this did not provide anonymity for respondents (the research team could, if required, identify individual responses) it did increase the level of participant confidentiality.
- Seven case study participants were recruited through the initial surveys and were interviewed and observed in their teaching three times over the course of the year. The interview questions were semi-structured, and generated from the survey framework, essentially interrogating the key research questions in more depth. The interviews enabled the researchers to probe teacher understandings in order to gather their feelings, views and attitudes towards their professional development (Kervin, Vialle, Herrington & Okely, 2006). Handwritten interview notes were taken.
- Tolich and Davidson (1999) suggest observation guides can be drawn up from a range of sources. The researchers developed the observation prompts from key literature (including: Krashen, 1981; Erlam, 2005; Gibbs & Holt, 2003; Ellis, 1993; Crozet & Liddicoat, 1997) and TPDL documents (e.g. milestone reports). Data was recorded through note taking. Because the presence of a researcher in the room with teacher and students may in itself be intrusive (Labov, 1972) the team did not digitally record teaching sessions as this may have introduced further distractions.
- TPDL milestone reports were another source of data. They were analysed and integrated into the writing of this evaluation. The milestone reports were all written by Wendy Thompson, Project Director of TPDL as feedback on the course to the Ministry of Education. There were nine milestone reports available for 2007 and 2008.
Participants and response rates
The participants were Years 7-10 language teachers who were enrolled in the 2008 TPDL programme. We asked all of them to respond to the surveys (n=58) and to volunteer for the case studies. There were 34 responses to the first survey out of a possible 58. There were 29 responses for the second survey and 25 for the third. Some questions had invalid responses and so total answers did not always tally with total response numbers. The researchers have only reported valid responses. The team provided a book voucher as a small incentive for teachers who completed all three surveys.
Case study participants were purposively selected from volunteers to provide as wide a mix as possible in terms of the following variables:
- geographical area
- type of school (i.e. intermediate, full-primary, district high, secondary)
- decile rating
- varying school communities (rural, small town, large city)
- level of students (Years 5 - 10)
- languages taught (Chinese, French, German, Japanese, Spanish)
- types of language learning opportunities for TPDL participants (e.g. polytechnic, distance, community provider)
- level of teaching experience (we have two case study participants who are first year teachers)
As mentioned previously, all seven case study participants participated in the whole study. The research team would like to acknowledge the teachers who were willing to participate. The case study teachers in particular were generous with their time and information.
In this section we outline some of the specific issues that arose in relation to ethical review for this project. Firstly, in relation to consent, all case study participants individually volunteered their participation through the initial survey and their individual consent for participation in the case studies was obtained through them (rather than through their schools). Principals were asked for research access to schools for the case study research only after teachers had volunteered to participate. All survey participants responded voluntarily and this was reflected in the decreasing number of responses: 34 participants for the first survey, 29 for the second and 25 for the third.
As part of the ethical approval process the research team were required to consider the relevance of the Treaty of Waitangi to the research. The research team noted that some teachers who took part in the case study research or those filling out the surveys could be Māori. The team explained that all participants would be given adequate time to consider the invitation to participate and that Māori may want to discuss their involvement with whanau. In the event no participants identified themselves as Māori.
Further, it was envisaged that the results of the research could be of interest to Te Reo teachers working in Kura Kaupapa Māori and mainstream classes with children acquiring Māori as a second language. The research might also be of interest to educators designing professional development for teachers of Te Reo. Because this research was carried out for the Ministry of Education the research team will discuss with them ways of sharing findings with Te Reo programmes and other relevant stakeholders.
In regard to issues of a conflict of interest, the researchers judged that they were unlikely to have any existing relationships with the 2008 TPDL cohort. The only perceived coercive influence might be that the research was funded by the Ministry of Education who were also sponsoring the TPDL. The researchers were therefore careful in the documentation and their interactions with teachers to point out their independence from Ministry and the fact that Ministry would not know who participated and who did not. Ministry have not had access to any raw data.
The research team felt that the ethical risks to participants were minimal. However, we acknowledged that the case study participants may have felt uncomfortable being interviewed and observed especially if they were new to language teaching, and if they felt self conscious about teaching and speaking the teaching language. Some teachers may have felt uncomfortable about being observed if they did not have a well-behaved or engaged class. The researchers therefore explained to case study teachers that their particular data would form part of a much larger picture and the Ministry would be interested in this rather than any specific detail relating to individual teachers. The team also reiterated the privacy and confidentiality protections designed into the research, which have safeguarded individual identity. Furthermore the researchers explained that they themselves were language teachers and language teacher educators who were used to observing teachers with different levels of experience and proficiency in the teaching language. Finally, teachers were able to withdraw from the research with no adverse consequences for their TPDL or any future professional development in which they chose to take part in. While there was a decline in respondent rate for the surveys, the seven case study teachers remained in the study throughout.
The research team analysed data in the following ways,
- The surveys were quantitative and qualitative in design although the first survey, ascertaining base line data, was predominantly quantitative. Because of the relatively low numbers, quantitative data was simply compiled on Excel spreadsheets and transferred into graphs. Qualitative survey data was entered into digital files and manually coded for significant themes related to the key research questions.
- The interview and observation data was recorded in note form and transferred into electronic transcripts by the researchers. Data was analysed in terms of the key research themes. Further sub themes were identified on an ongoing basis. Analysis was gradual, incremental and initially tentative so that premature explanation and conclusions were avoided (Tolich & Davidson, 1999). Non identifying quotes from the interview data were extracted to highlight issues or findings and enrich the final research narrative.
- Data from the 21 observations was analysed through mapping key components of observed teaching on to a grid. The grid called for responses to key components of effective language teaching: language input and output, focus on the form of the TL, instruction related to culture, student motivation, and learning strategies (see Appendix Four). The grid was a way of systematising the interpretation of the observational analysis and it enabled comparison between each of the three observations in a particular case study. Furthermore, the researchers kept a running record of the lesson, including direct quotes where possible.
- Milestone reports from the TPDL programme were available from the Ministry of Education. These were coded for themes and data incorporated where relevant.
Limitations and clarifications
The research has several limitations. The first is associated with the signing of the contract which occurred on 2 May, 2008. By this time TPDL participants had already embarked on their course. For those who were studying their teaching language (TL) for just one semester, that component was already half way through. The other components of TPDL were year long and so teachers had only experienced one weekend session of EDPROFST360 and one observation by the TPDL facilitator when they first met the researchers. Understanding that the validity of the research depended on reaching teachers before too much of the TPDL had passed, the research team secured initial ethical approval for the project prior to the contract being signed and the first survey was sent out in late April. Nevertheless, because of the time taken to identify and secure the participation of case study participants the first round of case study interviews and observations did not occur until May.
Another limitation of the study is the relatively low representation of Asian languages. Of the initial 34 responses only eleven teachers taught an Asian language. Few of these offered to participate in the case studies and in the event, the team were only able to secure one teacher of an Asian language for the case studies.
Among other things, the Ministry of Education wanted to find out through this evaluation, the outcomes for students learning languages in Years 7-10 as a result of their teachers undertaking TPDL. While Chapter six does address issues that we have been able to assemble through teacher perceptions and milestone data, the research was not designed to ascertain student outcomes through primary data collection. Data on student progression for language students in these years is not easily available on a national level and schools themselves have variable (and generally minimal) approaches to assessing achievement and reporting progression in learning languages. Moreover it was not within the scope of this research to seek ethical approval to engage students themselves as research participants. This would be possible however in a further study.
Finally, because of the discrepancy in respondent numbers to the three surveys, the researchers have generally only compared the stable cohort (core) of teachers who responded to all three surveys. Where this is of interest we have also included information from the full survey one cohort. Survey two data has only been included where the results were particularly pertinent to the question being discussed.
An explanatory note is warranted concerning the major assessment teachers undertook for TPDL. The milestone reports for the 2007 cohort referred to an 'action research project'. Later milestone reports, however, referred to the assignment as an 'inquiry learning project'. The researchers have used the term 'action research project' throughout this report. TPDL participants also referred to the project as the 'action research project'.
Finally, in an effort to ensure that confidentiality of participants was preserved, especially that of case study participants, we have mostly had to leave out references to particular languages and countries. Had we included references to languages, and particularly Asian languages, participants would have been easily identifiable. As language teachers ourselves we regret this because including more detail about the actual utterances used and related contextual information would have provided more 'colour' and specificity to the report.