Evaluation of the Sustainability of ECE Services During the Implementation of Pathways to the Future — Ngā Huarahi Arataki

3 - Methodology

A multi-method evaluation was conducted, comprising three streams of data collection:

• Consultations with sector stakeholders from umbrella groups and national organisations;
• Existing data sets maintained by the Ministry of Education and Statistics New Zealand; and
  
• Financial and related data collected from locality-based case studies using a survey administered with the assistance of NZCER and TKRNT field researchers.
  

Integrative analysis of the data from these three sources was carried out to triangulate findings and identify themes in relation to each evaluation question. Further detail on the data sources and the analyses carried out is provided below.

Stakeholder Consultations

Umbrella groups and national organisations provided qualitative feedback on the impacts of the Strategic Plan, drawing upon the experiences of their broad membership bases and the advice/support and advocacy roles they fulfil on behalf of their members/centres.

Consultations were carried out during March-June 2005 (baseline) and December 2006-March 2007 (follow up). The consultations sought feedback from a range of stakeholders regarding changes in the financial sustainability of ECE services associated with the implementation of the Strategic Plan. The qualitative information gathered through this process subsequently informed the quantitative analyses by identifying specific issues to investigate through analysis of the available data.

Interviews were conducted with the following organisations, which were identified in consultation with the Ministry of Education to ensure the consultations captured an appropriate cross section of perspectives:

• Kindergarten Associations: Auckland Kindergarten Association, Central North Island Kindergarten Association, Kids First (Canterbury); and Northland Kindergarten Association (baseline only).
• Playcentre Associations: New Zealand Playcentre Federation, Auckland Playcentre Association, Canterbury Playcentre Association; Northland Playcentre Association.
• Kohanga Reo: Te Kohanga Reo National Trust (TKRNT).
• Home Based Education and Care: New Zealand Family Day Care Association (NZFDCA) and Barnardos (national and office and coordinators).
• Education and Care Services (ECSs): Early Childhood Council (ECC); New Zealand Childcare Association (NZCA); Pacific Islands Early Childhood Association (PIECCA). In addition, three of the Kindergarten Associations interviewed also operate ECSs and provided feedback from this perspective, as did Barnardos.
• Ministry of Education: Education Management Policy Division; Advice and Support Facilitators.
  

The interviews were supplemented by a review of relevant information and commentary published on these organisations’ websites.

Analysis of Secondary Data 

At the time the initial study design was developed, the Ministry did not hold longitudinal financial data on ECE services’ expenditures. Consequently, the initial evaluation design focused on the collection of baseline and follow up financial data from the locality-based case studies to explore shifts in patterns of expenditure and indicators of sustainability. The limitations of this approach (including the small sample size for this type of analysis and potential difficulties in obtaining accurate financial data from services) were recognised, and were one of the considerations that led to the inclusion of national stakeholder interviews within the study design.

During the three year period of the evaluation, new sources of data became available which provided opportunities to conduct a more comprehensive and nationally representative analysis of financial data. In particular, the Ministry conducted operating cost surveys in 2005 and 2006 to inform the setting and review of funding rates under the new funding system. These surveys collected cost data which was previously unavailable at a national level and which was of value in the evaluation. Both the raw data and the Ministry’s analyses were reviewed for the evaluation.

Another emergent source of data was a survey of ECE prices, which Statistics New Zealand conducts for the Consumer Price Index (CPI). The quarterly data set from this survey has been used in the evaluation as an indicator of changes in the average level of prices paid by families for ECE.

In consultation with the Ministry of Education, a range of further secondary data sources were identified that would support the analysis of additional indicators and the exploration of issues identified by umbrella groups during the baseline round of consultations. These included a range of data sources currently being used in the ECE Strategic Plan Monitoring Framework¹ (as well as draft findings from analyses conducted by the Ministry within the Monitoring Framework), including capacity and enrolments data collected from RS7 and RS61 returns, cost data reported to the Ministry by community-based ECE services, licensing data, and data on numbers of registered teachers. The relation between data sources and evaluation questions is summarised in Table 1 below. The data sources and their strengths, limitations and interpretation issues are discussed alongside findings in section 4.

Locality-Based Case Studies

Case studies were undertaken by NZCER and TKRNT in a sample of 46 ECE services across eight localities to provide a micro-level understanding of the initiatives. NZCER conducted the case studies in education and care centres, kindergartens, Pasifika centres, playcentres, home-based services and playgroups. TKRNT conducted the case studies in kohanga reo, Maori immersion centres and nga puna kohungahunga.

For the sustainability evaluation, the opportunity was taken to collect data from these case studies on costs, revenues and sustainability impacts. Data collection was undertaken via a treasurer survey which was included within the data collection tools administered by NZCER and TKRNT field researchers. HOI participated in field researcher briefings prior to the baseline and follow-up data collection periods to ensure a sound understanding of the Treasurer Survey questions. HOI was also available to field researchers by telephone throughout the fieldwork periods to clarify issues.

The collection of treasurer survey responses was supplemented by follow up telephone calls by HOI in order to remind respondents to return completed surveys, to provide reassurances about the confidentiality of data where ECE services were hesitant to respond, provide assistance by clarifying information requirements, and/or to seek clarification of responses provided in the survey.

By integrating the collection of financial data into the scheduled visits to ECE services by NZCER and TKRNT field researchers, the data collection process was designed to maximise the efficiency of data collection and facilitate a higher and better quality response rate through face to face contact, than could be achieved solely through remote methods of data collection. The agreed approach and budget for data collection did not make provision for HOI to make individual site visits, nor to conduct audits to verify the accuracy of the data provided.

It was anticipated that obtaining financial data from some cases studies would not be straightforward, due to differences in financial and attendance recording systems (with the result that data could not always be disaggregated in a consistent way to answer the survey questions); differences in the financial expertise of treasurers and the time available to these people (who are often volunteers) to provide the requested data; and sensitivities related to the disclosure of financial information (particularly on the part of private ECSs).

Baseline data collection was completed for 37 of the case studies (an 80 percent response rate). Of these, 30 provided follow-up data (resulting in a final response rate of 65 percent, which is broadly consistent with the expected response rate as determined from a full census of the sector through the operating cost surveys). Responses to both the 2004 and 2006 surveys were received from seven case study kindergartens, seven playcentres, six kohanga reo, five community-based ECSs, two home-based networks, two private ECSs and one playgroup. Some respondents did not complete all questions in the surveys, and some anomalous or poor quality responses were excluded from the analysis where these could not be clarified. Response rates to particular questions are indicated alongside findings in section 4.

The principal value of the case studies for the sustainability evaluation is the ability to link financial changes to changes in outcome indicators from the evaluation conducted by NZCER/TKRNT, in order to explore linkages between sustainability issues and the three goals of the Strategic Plan.

Analysis

The information and data obtained from the stakeholder consultations, secondary data sources and treasurer surveys were first analysed separately. Integrative analysis and synthesis of qualitative and quantitative findings was then carried out to address each evaluation question. Further analysis and context was incorporated in the report through the consideration of findings from the NZCER and TKRNT research on quality, participation and collaborative relationships (through two workshops with the Ministry of Education, NZCER, TKRNT and HOI; and through the analysis of indicator data and review of draft report findings from NZCER).

Thematic analysis of stakeholder feedback was undertaken by reviewing written notes gathered during each interview. Interview data were compiled and analysed according to stakeholder group. Overarching themes across groups were then identified. Two key processes were used in the qualitative analysis of interview data:

• Top-down: Key themes emanating from the consultations were identified and a draft structure was created for their analysis and reporting.
• Bottom-up: Interview notes were thoroughly analysed in a structured manner, addressing each research question in turn and populating the reporting framework (as well as making modifications to that framework as required). 
  

Both forms of analyses were conducted in an iterative fashion until such time as thematic analysis became redundant.

Data analysis was conducted in Excel. Table 1, on the following pages, summarises the research questions and indicators together with the data collection and analysis that were undertaken to support each indicator. As a general principle, the analysis avoided value-judgements about what constitutes an ‘acceptable’ level of sustainability (in the absence of accepted benchmarks for this purpose) and instead focused on identifying improvements in indicators over time, through trend analysis.

Table 1: Research questions, indicators and data sources used in the evaluation

Footnotes

1. The indicators used in the Monitoring Framework are provisional and Monitoring Framework reports are draft. Further development is taking place on the Monitoring Framework.