Term of Reference 2: Assess the methodology and frequency of ERO’s current external evaluation services, and how these can more effectively contribute to improvement in the delivery of education by schools and early childhood centres.
Reviewers as Professionals
Comments seem to reflect experience. Some comments related to the behaviours and activities of reviewers.
- "At all times I found their comments constructive, non-threatening and of considerable assistance. The report itself was detailed and where recommendations were made they were supported by clearly understandable reasons and accurate recitations of fact."
- "The professionalism of the officers reviewing this school has always been of the highest order"
Others perceived a lack of professionalism.
- "Some reviewers demonstrate a real lack of interpersonal skills and there seems to be an almost obsessive desire to identify areas of weakness"
- "One of the reviewers who came to the school had a very high-handed and intimidating manner"
The key concern appears to be reviewer credibility, their knowledge on aspects of the curriculum, philosophy, sensitivity to school needs and whether they have had recent teaching/principal or manager experience in the sector that they are reviewing.
- "For the team to be credible it requires: management expertise in the areas of boards of trustees and principalship with up to date experience in Tomorrow’s Schools; curriculum expertise and current curriculum knowledge delivery; legislative expertise; assessment expertise; learning experience."
- Teachers resent being reviewed by people who are not specialists or ex principals in their sector. There is a lack of trust in the professional judgement of ERO reviewers reinforced by perceptions that some schools have hoodwinked ERO into giving a better report than the school deserves.
- "Concern was expressed about reviewers who have no background in early childhood and/ or with no understanding of the philosophy of the centre being reviewed" .
In relation to consistency, the public meetings provided the following views:
- There is an issue of consistency across the country. For example, a cluster of 3 schools developed a child abuse policy. When ERO carried out assurance audits, one school got a compliance, one got no mention and the third was marked as unsatisfactory.
- Schools in the same area have different experiences - teams focus on different issues and priorities, terms of reference, focus, issues to be examined. There are variations in the level of investigation and a lack of evenness of analysis.
Workloads Imposed
Two types of workloads were drawn to the Panel’s attention.
- The workload of ERO staff. They are not resourced adequately for the desired frequency of reviews and the time spent on site during a review.
"The organisation is seriously under-resourced in terms of funding ...It is a credit to the professionalism and genuine commitment of the management and personnel of ERO that it is able to operate with reasonable success under such stressful and constraining conditions."
- The additional stress on school staff, particularly for teaching principals and small rural schools, in preparing for an ERO visit and during the time the review team is on site.
"ERO have exerted huge pressure on rural schools who have much more limited human resources to meet the same requirements as urban schools. ... we have always had excellent ERO reports but it is only through the staff in particular working very long hours and being extraordinarily dedicated beyond the norm."
Reporting
Reporting was of great interest to all those who made submissions and who attended public meetings. Comments included:
- Reports are regarded as bland and non-specific, don’t notice the positives and do not record student and school achievements.
- Reports seem to use the same format providing little sense of the school’s ethos and uniqueness.
- It is difficult to read ERO reports - full of jargon, often have no definitive evidence to support statements, with opinion set out as fact, problems of sweeping generalisations in reports eg, "some teachers", "most teachers".
- There are dramatic differences between oral feedback and draft report. When positives are reported orally and fail to appear in the written report the credibility of the report and the reviewer is undermined. Frustration was expressed in that the ERO oral report was excellent, the written report very good and the newspaper report very damning.
- No dialogue with ERO or option of discussing/giving further information with the unconfirmed written report.
- There is a power imbalance between ERO and the school here. There is no appeal process re what is in the final confirmed report. "Let there be no doubt that ERO do not alter school reviews nor do they apologise when they are found to be subsequently wrong."
- There can be delays between ERO visits and reports. Where a school has rectified a problem they can still be pilloried for something that is now fixed.
- The suggestion was made that the audience of the report must be clarified and perhaps it was necessary to write multiple reports designed for different audiences e.g. Boards, Principal, teachers, parents, media.
- ERO reports have improved over time.
Publication of Review Reports
There was widespread but reluctant agreement that ERO reports are public documents.
- "We agree that the reports are public documents and should be published and made available to the media in the interest of accountability"
There was recognition by the public that ERO reports have a tremendous impact when printed in the newspaper and negative reporting can be damaging.
- "We do not believe it is productive to a school and its community to be sensationalised by the media due to a perceived poor ERO report. The current media use of ERO reports appears to focus on negative reports or aspects of reports in ways which produce trauma for the school community concerned."
- Reports, "should never be made public to be exploited in the press and other media. This can only lead to further problems. The board of the school is the democratic representation of the local community and they should be informed and asked to correct faults."
- "All findings and reports should be available to the board, staff and parents without all the newspapers selective reporting that presently happens."
- Release to media by school, not by ERO.
"They should be available only on request from the school, that way the school has an opportunity to put things in perspective and has prior knowledge something will be published and not be subject to the sneak attack".
- "In relation to the timing of confirmed reports it seems common sense that reports should not be made public until a school has had time to comply with ERO instructions/act on its recommendations."
- ERO should prepare a separate summary report for the media, with a more detailed report to the school "The ERO reports should be confidential documents that are as helpful as possible to schools. A shorter (two paragraph) summary report using a standardised format could be available for the media".’
- The Commonwealth Press Union however, representing many of the NZ journalists, however, "wholeheartedly supports the current ERO practice of publicly releasing school review reports for the following reasons: the release of reports is in compliance with the intent of the Official Information Act; the information is of considerable public interest; the level of Crown investment in schools is high, which justifies a high level of public involvement and information; the approach is in line with Tomorrow’s Schools in enhancing the performance of schools; this approach is likely to lead to more informed parental choice."
- Some Principals argued that journalists need to be educated as to what reports mean and not used out of context and prepare balanced articles. It was suggested that ERO should follow up negative media reports. The press usually gives the school opportunity to comment but often they do not give it, so an article may be unbalanced. "Schools should be looking for skills on how to handle the media and how to handle the reviewer."
- "The media shackles ERO too - perhaps that’s why the reports are so insipid."
- Media could get reports from schools, not direct from ERO.
- ERO reports could be delayed 3-6 months before publication or not published.
Evaluation Reports
Few comments were made on the national evaluation reports although more had positive than negative comments. Comments from schools indicated they found them useful.
- "Booklets produced by the Review Office such as Professional Leadership in Primary Schools and Reporting Student Achievements are well worth reading. They should be required reading for all aspiring or present principals."
- "ERO publications are excellent vehicles for promoting best educational practice."
Some national organisations, however, questioned the validity and reliability of the research methods used to produce the findings.
- "Generally speaking, ERO publications are too esoteric to be of any use to anybody except politicians and educationalists and as already mentioned are not rigorously researched documents and are often misconstrued."
Advice and Guidance
There is an unmet demand for advice, guidance and support for schools.
Several themes came through in submissions:
- "NZFKA considers that the Education Review Office should not be involved in offering guidance and advice ...the Office would: develop a conflict of interest as both adviser and reviewer."
- There should be a separate arm of ERO to give advice. "An additional arm of the ERO should be created to assist governing bodies implement review recommendations."
- "Reviewers are in a unique position to be able to provide quality advice, guidance and assistance after being in the school for a period of time. It is a waste of a valuable resource not to be able to utilise their knowledge and expertise."
- ERO should direct schools to sources of advice. There is a lack of knowledge about where to get help for problems, especially principal or board problems faced by the board. This is compounded by a perceived reduction in professional contact with other schools.
- ERO should have the power to direct advisors to approach schools to give support.
- ERO does not have a function in advice and guidance.
- The Government needs to give more professional support for schools. Concern came through that access to advisory services was limited. The number of school advisors needs to be increased and their role expanded. There appears to be a need to formalise the relationship between ERO and the Advisory Services. More advice also needs to be given via the NZ School Trustees Association.
Conducting reviews
There was a wide range of suggestions as to the format, processes, approach and criteria suggested for reviews. Some of the repeated themes were as follows:
Timing of Reviews
There were three main messages from the written submissions.
- In the early childhood sector people wanted more frequent reviews, spending more time in centres. Two years was the most common time frame mentioned.
- The school sector wanted reviewers to spend longer time in schools when reviewing. "ERO visits should be held over a period of a few weeks so they can build up a relevant picture."
- Schools were generally happy with a frequency of reviews every three years although some wanted annual reviews and others only reviews every four or five years. "Every three years is a good time frame."
Composition of Teams
Feedback from public meetings suggests:
- Reviewers must be up to date with practice, knowledgeable, have specialist current curriculum knowledge, must have worked in the Tomorrow’s Schools environment and have experience of the new curriculum. ERO teams are professional in behaviour and conduct but their level of expertise is questioned i.e. knowledge, this influences their ability to review accurately. There is a perception that ERO needs reviewers who are experienced in the sector being reviewed - pre-school, primary and secondary. Anything less reduces their credibility.
- Secondment of principals to ERO teams is a good idea and they work well in a team but they should not work in their own area as there is a conflict of roles. However, reviewer salaries have not kept pace and are well below the level of principals so ERO is not likely to attract experienced and respected principals.
Relationships, Partnership and Policing
- ERO needs a more active, ongoing relationship with schools. Rarely do the same review members come back in the next audit.
- Schools must provide reviewers with a professional environment in which to work. The Principal should be the reviewers’ friend, accompanying the team.
- The empowered schools concept is a myth - there is still a lot of central control and schools are sandwiched between ERO and the Ministry of Education. ERO and the Ministry do not talk to each other and need to. ERO needs to work as a team with NZQA and the Ministry of Education.
- The question was asked "Who is the ERO report for?" - the Minister, the Board, the Principal, the parents - the client base needs to be identified.
- ERO should investigate exemptions for home schooling.
- ERO should be an avenue for parental complaints if the Board has given no satisfaction.
- ERO should expand its focus to look at the impact of Government policies on schools, the local community and issues out of the control of the school.
- An alternative view is that ERO should not be handling issues beyond education. A health and safety agency could be appointed and spot check for compliance with non-education requirements.
- The functions of each of the agencies - Ministry of Education , ERO, NZQA etc. need to be clarified and the specific criteria against which the audit was conducted e.g. ERO, ACC, OSH.
- The functions of ERO should be contestable.
- To give schools choice in who carries out reviews "a. Each education district would have a number of reviewers to choose from ... b. These reviewers would still apply the national education and administration guidelines ... c. The reviewers would be accredited by a section within the Ministry of Education."
Term of Reference 3: Recommend any procedural or legislative changes that optimise the educational benefits and maximise the effectiveness of the current usage of the ERO’s resources.
Post review action
- "There should be a statutory requirement for a follow up meeting between ERO and the school to discuss implementation of changes necessary to remedy any failing or deficiencies identified in the report."
- In relation to compliance issues in the early childhood sector, "tougher more stringent requirements be enforced," including withdrawal of government funding where there is failure to respond.
- Links to advice and support "The need for an agency to follow up with professional development in areas recommended."
- If there is a poor report and no change, where does the community go for action? ERO should be able to insist on the recommendations being followed up. Time limits should be in place for ERO’s recommendations to be acted upon.
- If ERO is not satisfied, then another agency can then come in and assist the school to meet the needs as required by ERO. There is a need for advisors to go into schools to give support and advice.
- Boards are aware of the importance of professional development but its costs for remote schools is prohibitive. Resourcing must recognise this. Boards need to know where to go for assistance and sources of best practice.
- The leadership of the Principal is crucial and may be the problem in some instances.
- There is no need for ERO to have teeth. The Minister should act on ERO advice. ERO should have more clout with the Ministry of Education re health and safety concerns identified in recommendations. If there are compliance issues, ERO should come back and check. Where non-compliance is serious it should be possible to cut the funding.
- ERO should comment on Board-Principal relationship. Much of the work of a Board is delegated to a Principal and staff to carry out, including implementation of ERO compliance and recommendations. It is unclear what should be left to the Principal and what to the Board.
- Review teams may not have the expertise or competence to guide but they should be able to direct to sources which are appropriate.
Term of Reference 4: Examine current methodology for evaluating the impact of policy and delivery of services of other state education agencies on educational outcomes for students, and consider how this information should be fed back into the policy process.
Interpretation
A lack of confidence was expressed at public meetings in the differing interpretations of the Ministry of Education and the Education Review Office on some issues e.g. assessment. ERO appear to be imposing their interpretation on schools, contrary to the self management school model. When ERO uses a comply/not comply model for policies under development or partially implemented different reviewers have different standards on what constitutes compliance.
The Ministry of Education and ERO have different expectations, with the school in the middle. There is considerable concern about the differences between advice received from the Ministry and advisory services and ERO expectations. Sometimes the recommendations made in a report are contrary to known good practice. There was a repeated call for clear expectations from ERO, unambiguous language and confluence between Ministry and ERO statements.
Purpose
The purpose of this review is to recommend ways of increasing the effectiveness of the Education Review Office’s contribution to improving the quality of education outcomes for students in schools and early childhood centres.
Terms of Reference
- Examine the nature and scope of the authority of the Education Review Office and assess the degree to which such authority impacts on and contributes to improvement in the quality of educational outcomes.
- Assess the methodology and frequency of ERO's current external evaluation services, and how these can more effectively contribute to improvement in the delivery of education by schools and early childhood centres.
- Recommend any procedural or legislative changes that optimise the educational benefits and maximise the effectiveness of the current usage of the ERO's resources.
- Examine current methodology for evaluating the impact of policy and delivery of services of other state education agencies on educational outcomes for students, and consider how this information should be fed back into the policy process.
Government Educational Objectives
In carrying out the review, the review panel will have regard to the Government’s educational objectives, principally to:
- encourage a more highly knowledgeable and skilled nation;
- lift the quality of the learning environment;
- maximised education gains in cost-effective ways;
- promote an education sector which is responsive to the needs of students, in particular those with special needs or at risk of poor educational outcomes; and
- enable local decision-making at the school/early childhood centre level to enhance the quality of education.
Other Matters to Consider
In addressing the Terms of Reference, the review panel will consider:
- the appropriate balance between self reviews and external evaluation in schools and early childhood centres;
- the functional relationship of external evaluation with other agencies in the education sector relevant to schools and early childhood centres;
- the adequacy of current processes to develop common understandings between agencies for policy formation, policy implementation and advice, and the Education Review Office (ERO) as to what is considered satisfactory performance from schools and early childhood centres;
- the constructiveness of current processes to communicate evaluation findings.
Assumptions
The review is to be based on the following assumptions:
- the Education Review Office remains in existence, as an independent Government agency;
- the objective of the review is to maximise the effectiveness of the office; and
- the review panel will be requested to keep recommendations fiscally neutral.
