Executive Summary

The Ministry of Education is committed to aligning policy and practice to improve students' achievement.  It must be informed by and respond to information about what works, under what conditions, for whom, why and how and what makes a bigger positive difference.  However, not all knowledge will inform better decision making, and not all good decision-making will lead to better outcomes – the path from research to policy is not straightforward.

The Iterative BES Evidence-based Policy Project (EBPP) has been developed as a change-management pilot project that seeks to influence the balance of adaptive and generative learning in the Ministry of Education through an examination of factors influencing the relationship between research and policy practices.

The Iterative BES Programme is a key source of learning outcomes-linked evidence that could inform generative learning within the Ministry.

To provide a framework for appreciating the contexts of policy advice, the literature review for this project focused initially on the effectiveness and impact of the use of outcomes-linked research evidence in policy development practices. Where possible literature originating in New Zealand takes priority and international findings are applied with caution.

Literature Review

The changing context of educational policy advice

50 case studies in developing countries found that political context was the most important factor affecting the degree to which research had an impact on policy. (Court and Young, 2003)

The critical factor influencing uptake of research was found to be the nature of the evidence and whether the research was credible and relevant in terms of operational usefulness and problem solution. The other major factor focused on the social context linking researchers and policy makers.

Claudia Scott documents a weakening of the relationship between research and policy following the introduction of Mixed Member Proportional Parliamentary system in New Zealand.

The State Services Commission (SSC) study (1999a) showed that "… policy makers react to major problems, formulate quick solutions to them, take decisions, implement these and then move on to the next set of problems. (p9). SSC sees short time frames as inhibiting in-depth research, commenting however that "if organisations do not have robust information and research capabilities to anticipate demands, they will never be ahead of the game and the quality of their advice will suffer"

The SSC report (1999a) notes that whether or not advice is backed by quality information, the brevity required in the presentation of advice, and the fact that advice is generally not referenced with information sources, means that there is no mechanism to assure Ministers that the assertions in advice are more than informed guesswork.

Alcorn et al. (2004) conclude that "there is clear evidence of a critical mass of nationally and internationally excellent researchers in education in New Zealand" . However they do note that a large proportion of education academics were not highly rated in the 2003 Performance Based Research Funding (PBRF) evaluation of research excellence (TEC, 2004) (p10)
Davies, Craig and Robertson (2004) suggest that the PBRF incentives for research dissemination, crucial to informing policy, emphasise international publication for small specialised audiences over publication in New Zealand journals and the mass media.

The research context

Luke and Hogan (2005) indicate that definition of terms, underlying ideologies and methodologies are a rich ground for mis-understanding between researchers, policy advisers, practitioners and other stakeholders in the wider community.

From the policy viewpoint, conceptualisation of 'useful research evidence' varies with national context in ways that facilitate or constrain application of certain methodologies. (p12)
Court and Young (2003) found that across these case studies, policy uptakes were greatest where influencing and communication strategies were in place from the beginning of research programme.

Iterative Best Evidence Syntheses

The Iterative Best Evidence Synthesis Programme embodies factors contributing to a positive relationship between research and policy in accord with Court and Young's (2003) framework of political context and purpose, quality of evidence and links between researchers and policy advisers.

Evidence-based or evidence informed policy?

The distinction between evidence-based and evidence-informed policy, according to Edwards (2000), centres on the degree to which research is expected to provide direct guidance about effective practice in teaching and learning in schools, or whether it 'informs' through increasing understanding of practices, processes and institutions.

Claudia Scott indicates that what she terms "high performance advising' will be fostered by a transformational approach to policy development and enhancing opportunities for strategic thinking and conversation.  However, Behm et al concluded that public servants are quite proficient in transactional policy skills, but less so in transformational policy skills.

Factors in the research-policy-practice gap

Research evidence is perceived as "softly spoken" because, "empirical inquiry simply cannot make its voice heard amidst the clatter of other, political imperatives on policy making" (Pawson, 2002a).

The gaps and misunderstandings between researchers and users do not arise from people's faults, but from the realities of their contexts (Levin 2005). For example  policy advisers see research as heavily qualified by conditional constraints and irrelevant to their concerns, but researchers suspect that the knowledge they generate is subject to the predispositions and agendas of policy advisers, with the result that it is not read, understood or used, or may even used in a manner that distorts findings.

Factors affecting policy advice include:

• tensions between planned policy platforms, external political pressures, changing circumstances, unexpected events and crises
• there is never as much time as politicians and policy advisers would like to think deeply about issues
• the nature of democracy in that those on opposite sides of parliament have different purposes for considering research.

Good policy is likely to be the result of policy advisers and researchers working together, since researchers can change perceptions of problems and the values associated with policy, but policy advisers have better insights into influencing bureaucracy and constituents.  However there are risks on both sides concerning allegiance and independence.

Research can influence policy

The Numeracy Development Project is an example of research influencing operational policy and practice.

There is some evidence that research, particularly by teachers undertaking action research, can affect teaching practice more than it affects policy.

Translating research into policy and practice

There is agreement in the literature that many sources of information influence policy decisions and that research will often be one of the less important factors.  However, Reid (2003) could find no significant international or national body of academic research on the actual process of research integration with policy as seen from the policy advisers' viewpoint.

What is meant by 'using research'?

Assumptions around effective use of research are rarely discussed in the literature. Edwards (2000) notes that there is a risk that a policy maker's view of really useful research is that it "takes the problems which the government brings to it, contributes to their solution and refrains from adding new problems or further complicating old ones"

Research use, problem solving and information literacy

The SSC study (1999) showed that policy analysis and design of delivery instrument, process co-ordination and the design and management of implementation have been the focus of most attention in the policy cycle. However, there were gaps around evaluation, issues identification (which also implies problem definition), the notion of long-term forward looking research-based policy analysis, public consultation and strategic analysis and management.

There is a challenge in distinguishing between genuinely research-based evidence and the plethora of ideas that are presented as if they were the result of reliable research. These ideas come from governments, think-tanks, inspection regimes, auditing agencies and even researchers themselves, who have sometimes made claims beyond the evidence and their personal competence.

Mechanisms enhancing the relationship between research, policy and practice

Walter et al. (2005) support Pawson's (2002a) view that improving the correspondence between research and policy is more likely to be the result of exploring mechanisms that explicitly or implicitly underpin different approaches to research-policy interactions than it is of any particular intervention.

Walter et al. identify five mechanisms that appear to underlie interventions to improve the effectiveness of research use: dissemination, interaction, social influence, facilitation and reinforcement.

Use of conferences and workshops to supplement dissemination of written materials has been found to have a small positive impact on changes in practice.  In addition, Walter et al note that seminars facilitating discussion of research findings, alongside provision of written material, have a positive impact on participants and may support changes in practice, but the evidence is less robust.

It is suggested that publication in academic journals is the most passive kind of dissemination, yet it is one for which researchers are rewarded by their institutions.
Walter et al. note that seminars facilitating discussion of research findings, alongside provision of written material, have a positive impact on participants and may support changes in practice.

There is evidence that interactive approaches (e.g. development of partnerships and collaborations between researchers, policy advisers and practitioners) also facilitate the adaptation of research to local contexts.  However success is constrained by "the time and energy required to establish effective working relationships, differences in culture, goals, information needs, timescales, power, regard systems and language issues of project control and direction."

In the course of day-to-day work, practitioners and policy advisers turn to colleagues as a key source of knowledge.  Interventions applying social influence strategies thus focus on interactions within policy and practice contexts rather than looking beyond them to interactions with researchers and their success depends at least partially on the actions of opinion leaders.

Facilitation focuses on the context within which research findings are to be utilised.  It is defined in terms of provision of technical, financial, organisational and/or emotional support enabling policy advisers and practitioners to use research findings. Provision of professional development to increase research use capability is one form of facilitation.

Reinforcement approaches reviewed were more coercive in nature than the other approaches.  Prompts and reminders (including manual and computer generated, separately and together) seem to be effective for a range of behaviours, but findings regarding audits and feedback are more varied other than for implementation of guidelines.

Creating the conditions to support initiatives across research, policy, and practice

Court and Young (2003) concluded that the context in which research is to influence policy development affects outcomes, and change management research emphasises the need to create conditions that support sustainability of change.

In one initiative, facilitation and interaction mechanisms were supported through employment of an 'implementation officer' who worked directly with practitioners and service planners on issues related to the adaptation and replication of research recommendations and findings. The role was found to be pivotal in the success of research implementation.

Conclusions across health, education, criminal justice and social care sectors consistently suggest that for policy use:

• research must be translated, adapted for or reconstructed within the context of local practice.
• those who are expected to use research must have a sense of ownership of the project, programme or tools that it generates.
• individual enthusiasts or 'product champions' are crucial to 'selling' innovations through personal contact.
• successful initiatives are those in which the demands of the context of implementation has been analysed to create strategies supporting change.
• the use of research is enhanced where evidence is credible to the potential users, there is endorsement from opinion leaders and high-level commitment to the process is apparent.
• Strong and visible leadership can provide motivation, authority and organisational integration.
• adequate financial, technical, organisational and emotional support is important.
• research use needs to be integrated into existing organisational systems and ways of working.

Learning conversations

The policy development process is convoluted, the information needs of policy advisers are complex and neither of these areas of concern are necessarily apparent to research producers or BES writers.

A series of ten Learning Conversations was conducted with policy advisors from all groups in the Ministry.

The role of research in general

For two people, consideration of research was seen as central to their roles and pervaded their work.  For example,

"…  it drives the way I think about conceptualising ideas and prioritising work, and making recommendations for interventions.  I see it as being absolutely fundamental to the way I think and work." (Participant 4)

The role of outcomes-linked evidence

Five participants indicated that they specifically seek out outcomes-linked research, although two indicated that this is not always easy. However, two participants expressed awareness that their attention to outcomes-linked evidence had increased only fairly recently, both attributing the change to BES activities.

The role of BES reports

While valuing the Iterative BES Programme reports as a prime source of relevant outcomes-linked research, participants saw a need to consider additional research sources.  Some participants spontaneously provided examples of instrumental or direct use of BES research in decision making. However, several cautions emerged regarding uneven awareness of BES in workgroups, potentially superficial understanding by those lacking awareness of the developmental context of BES, and dangers of over familiarity with reports potentially leading to dissemination of inaccurate messages.

Features of BES that make it a significant source of outcomes-linked evidence

The focus on student outcomes, the trustworthiness of evidence and the emphasis on research from New Zealand are key features of BES that make it a significant source of research evidence for policy advisers.

The influence of BES development processes on thinking

Five responses indicated a direct influence on personal action resulting from involvement with BES development, with three people indicating that they now read research more critically. … The evidence suggests that the BES development process reflects the interaction mechanisms and enabling climate conditions identified by Walter et al. (2005) as enhancing the relationship between research and policy.

BES as a change agent within the Ministry and in the sector

Within the Ministry, participants tended to agree that BES is having a considerable effect as a change agent, but while people are using the knowledge, they are not necessarily acknowledging that it came from BES.

As a change agent in the sector, the most concrete point to emerge from the Learning Conversations was that, participation in BES development by teachers began with personal interest but has moved to organisational commitment to the programme that provides continuity across representatives. However, some participants noted that across the sector understanding and utilisation of BES seem to be variable.

Conditions or contexts in which BES is particularly useful

The task contexts in which people turn to BES for information vary from broad direction setting, framing ideologies underlying action plans, motivating change and benchmarking international research against the national context are all important, but do not necessarily leave a verifiable trail. The basic condition indicating utility of BES was the high and obvious relevance of content to current tasks (mentioned by five participants).

Reducing barriers to the use of BES outcomes-linked evidence

The most prevalent personal action people took was simply to read.  However, eight of the ten participants quickly noted that one reading was insufficient to build confidence or complete understanding.  … "our organisation could do is to be a lot clearer about acknowledging that people have only ever got so much time to read, and be very clear that reading is a significant and important part of our work."

One set of dissemination strategies may be needed to establish the use of the current set of BES reports, but others will be required to keep interest in the programme alive over time, particularly in light of the way relevance of BES waxes and wanes in relation to the focus of policy tasks.

Resources and activities

Four people commented that the BES reports are densely written and a change in reading formats would be appreciated by some.  Retention of the quality was however, important – the issue was speedy, not necessarily easy intellectual access to relevant content.

The value of highly digestible case studies was alluded to on three other occasions in interviews, particularly in making connections with the education sector.  Scenarios produced for one BES and presented orally have been useful in engaging audience attention and publicising good practice. More of these together with two page summaries of research were requested.

The area of greatest agreement among participants, that oral presentations and discussion of BES content are useful as a first step to engagement and have a role in increasing utilisation of BES in policy development.

While oral transmission of information is an important factor, Participant 9 highlighted the importance of collaborative tasks in applying learning.  A key feature of working together is however having purposes that converge, as might be found in cross-Ministry interest groups.

There were four comments with implications for organisational conditions related to time management, commitment and human resources. For example, throughout the interviews, time management was a recurring theme.

Half of the participants had difficulty giving specific examples of the contribution of BES to policy. It does seem that at this point in its development, BES provides a general background noise that informs rather than explicitly guides policy.

Dissemination Management Factors

Kirst (2000) notes a discrepancy between the pervasive view that policy research either does not reach or is not used by educational policy advisers and the frequent citation or acknowledgement of policy research in the US.  He seeks resolution of this contradiction through an examination of the effectiveness of education research dissemination.

Kirst (2000) notes that decades of research point to five dimensions that influence the outcome and effectiveness of policy research dissemination efforts: the source of the communication, the dissemination channel, the communication format, the dissemination message and characteristics of the recipient.

Learning Conversation participants have drawn on first hand experience of the forums in which BES evidence features. They are now becoming sources of contextual and language translation themselves, although some clearly would not recognise themselves as reliable experts just yet.

The ideal dissemination strategy involves distinctive messages and multiple channels.  Furthermore, the most effective formal and informal dissemination channels are the natural networks comprised of leaders and practitioners with an interest in a particular issue.

Research findings on effective dissemination of written materials are consistent with regard to formats: effective dissemination requires that these materials be jargon-free, brief and provide concrete illustration. Costs of information consumption rise unacceptably as potential users face the task of extracting information relevant to their setting, presenting it succinctly and interpolating the significance for their institutional needs.

Information, in the form of messages from research, is more likely to be used when it is timely and meets needs associated with the current phase of the policy cycle and the policy problem. The complexity of BES messages and the relations between them   demands innovative approaches to dissemination formats e.g. 'research nuggets' directly influenced the focus of interventions in Liabo's (2005) study.

Kirst (2000) concludes that the salience of recipient characteristics underscores the importance of a two-way dissemination strategy in which information providers acquire information about user needs, preferences and problems and in which, where possible, users participate in framing research objectives.

The Iterative BES Programme seeks to influence policy and practice in the sector.  By far the largest proportion of actors in the education sector are teachers, whose characteristics as recipients of research documents from the Ministry need to be considered.  For examples, outcomes-linked evidence is likely to be interpreted in terms of the teacher's experience, style, values and implicit beliefs of theories about teaching and learning.

With regard to the characteristics of media reporters, Levin (2004) observes that the accuracy of messages reported depends to some extent on policy advisers' ability to couch explanations for decisions in terms that minimise the possibility of misinterpretation.

Resource Framework and Specifications

The Iterative BES Programme is a multi-level, collaborative knowledge and capability building project, yet, no matter how high the quality of the research, one cannot assume that it will automatically have consistent or coherent influence on future research, policy or practice across the multiple audiences for whom it is intended.

While factors influencing the impact of research on policy have been identified and various models of the policy process have been discussed, the actual process by which research is integrated into policy remains rather vague.

There is no evidence offered in the literature reviewed that researchers, policy advisers or education practitioners share any sense of how research in general will be used. Consequently, the notion of 'using research' needs to be clarified with respect to the Iterative BES Programme. A critical step is to use Learning Conversations across the Ministry, not only to discover the information needs of policy advisers, but to invite their input to development of understanding of the research-policy-practice nexus from their experiences and knowledge of literature that resides in other silos.

At present a number of 'BES champions' have been identified, but there is no co-ordinated effort to take advantage of their potential as opinion leaders and contributors to an enabling climate (Walter et al. 2005). Providing them with resources that meet the needs of their audiences would build their confidence in conveying the complex messages of BES reports in formal oral presentations and discussions.

Promotion of ownership could be achieved through opportunities for varying combinations of researchers, policy advisers and other sector representatives working collaboratively on tasks associated with BES, but not necessarily part of the BES development process.

Formal and informal networks have been found to be important in promoting research uptake and working collaboratively across the research-policy divide achieves this by contributing to shared experiences of reality.  It is therefore suggested that when issues groups are set up for a specific policy purpose, a research implementation/adviser role is included

Facilitation as defined by Walter et al. (2005) includes organisational support for the use of research yet throughout the Learning Conversations a recurring theme was lack of time and lack of recognition of the centrality of reading to the policy advice role.  Changes in ways of working that encourage reading, interaction and reflection may assist, but are beyond the boundaries of the Iterative BES Programme to implement.

Social interaction is particularly effective as a dissemination tool and the Iterative BES Programme appears to be highly unusual in the degree of technical and financial support offered to enable participants from outside the Ministry to attend e.g. quality assurance and development meetings. Whether similar levels of opportunity and support are available to all Ministry employees is less clear.  Holding collaborative dialogues with those in regional offices centred on issues they wish to address would address this.

Researchers and policy advisers should take responsibility for ensuring that research is used constructively.  For example, it may be that reading this analysis of audience needs may promote discussion of innovative ways in which writers themselves can contribute to a range of supplementary resources.

A key message from the literature has been that one needs to listen to the intended audiences for research to understand their information requirements. For policy advice, Ministry Staff need to:

• be continuously aware of BES reports so that relevance to current policy tasks is apparent,
• have easy physical access to existing reports,
• have speedy intellectual access to report contents in part and as a whole for a variety of purposes, and
• be alerted to highly significant or influential evidence as it emerges from BES reports under development

To achieve this, dissemination materials and activities should all, on the basis of the capability building principles underlying BES, the literature reviewed and the evidence from Learning Conversations:

• respond to specific purposes of  various audiences in and beyond the Ministry
• use the language of those audiences
• support translation of evidence and construction of understanding for local and wider issues
• be catalysts for further learning and exploration of BES reports
• provide speedy, not necessarily easy intellectual access to concepts and evidence, and
• provide succinct coverage of specific research evidence, yet make connections to the theories emerging from syntheses and across the various BES reports.

A resources framework is provided in terms of the needs of potential BES users, inductees, BES independent, BES occasional, BES dependent, Project coordinators and BES Champions.

The wide range of suggested two-page communications would facilitate introducing BES to users in manageable chunks but result in fragmentation of the evidence.  This is a challenge in that the unifying theories resulting from the syntheses could disappear from view. To counter that, a graphic device could be developed that illustrates the key concepts of the particular BES, highlighting the concept of the Gem or Nugget in question. References to the whole report, the original research items and to related two-pagers would be included.

Attention to the visual format (text, graphics, overall visual complexity) is needed and departures from traditional presentations need to be piloted with readers as there are indications that innovation does not always enhance conceptual engagement.

Within the Ministry, the Library could play an important part in bolstering confidence in evaluating research materials and ensuring policy advisers are aware of BES as they begin seeking information on a new issue.  A two-page guide to assessing evidence that prompts comparison with the outcomes linked evidence from BES could be included with the first batch of papers/references sent from the library in response to search requests.

Making all BES related materials available on the Ministry website would have some advantages, but would need to be accompanied by regular reminders of the URL.  Putting information on a site does not equate with its use.

A check of the BES reports currently available on the Ministry website found that those in .pdf format make no use of 'bookmarks' that can be on-screen as aids to navigation in long documents.  Similarly, Learning Conversation participants requested what amount to conceptual maps for each BES but these too are already available through use of the Document Map function within Word.

The value added by digital availability is widely assumed to include:

• easy physical access and ability to print as needed, thus reducing print stocks and costs,
• provision of hyperlinks between and within documents allowing dynamic searching for information specific to particular issues across reports,
• provision of glossary items via drop down boxes or rollovers so that reading is not disrupted but sense-making is enhanced,
• provision of links to relevant issues groups or learning communities,
• links to professional development opportunities,
• access to the research behind BES through the database being constructed by NZCER, and
• options to provide feedback.

Looking beyond the Ministry, there would also be the potential to link practitioners with others beginning empirical investigation of particular issues in their schools.  Research collaborations could be supported and their findings reported through the same channel.  This has the potential to contribute strongly to a culture of inquiry and to support the implementation of a research and development strategy.

In terms of the Ministry itself, the most pressing issue is that of discussing expectations around BES – what are expectations for its use and how is that use to be demonstrated?  The answers would inform selection of the set and detailed format of the suggested resources most effective in supporting generative learning among policy advisers.