Appendix 3: Methodology

Key evaluative questions

The overarching question for ERO’s evaluation of principal appraisal was ‘How effectively does appraisal of the principal contribute to development?’

Separate judgements were made about the contribution of principal appraisal to the principal’s own development, staff and school development, and improved student achievement.

Four other evaluative questions also provided some relevant information:

  • How coherent are the school’s charter, strategic plan, targets, appointments, appraisal, and PLD in supporting high quality teaching, ongoing improvement, and raised student achievement?
  • What factors enhance the implementation of robust appraisal that supports improving student learning?
  • What factors hinder the implementation of robust appraisal that supports improving student learning?
  • How knowledgeable and well-prepared are boards and principals for their role as employer?

Sources of information

Information for this report was drawn from three main sources:

  • an online survey of a sample of board chairs about appraising the principal (154 chairs, 52 percent response rate)
  • an evaluation of appointment processes and practices in 27 selected secondary schools in Term 1, 2013
  • an evaluation of appointment processes and practices in 173 primary schools having a regular review in Term 1, 2013.

Online survey

The online survey gathered detailed information from boards about their documented policies and procedures, the principal’s most recent performance appraisal, involvement of an external appraiser, sources of information and advice, board training, board confidence in appraising the principal, and their view of the appraisal’s effectiveness. The questions were largely based on NZSTA’s Guidelines on Managing Principal Appraisal and Good Practice Framework (see Appendix 4: Useful resources).[30]

Only 52 percent of boards responded. Rural and small schools are under-represented in the responses (see Appendix 3). The number of responses is too small for robust analysis of sub-groups, but there were no consistent trends in terms of location or size of school.

Appendix 2 presents the survey questions and responses of board chairs.

Investigations in schools

The investigations in primary and secondary schools enabled reviewers to interview trustees and staff and review documentation so that they could make judgements about the key evaluative questions. Appendix 1 shows the indicators for each evaluative question.

Appraisal of principals was evaluated in 173 primary schools as part of the school’s regular education review. These investigations did not include any schools with an early return of one-to-two years signalled in their previous ERO review report, or any schools where ERO had signalled in the previous review that their next review would be within four-to-five years as the latter schools were not due for a review in 2013. [31]

As few secondary schools were scheduled for review in Term 1, a sample of 27 secondary schools was selected for a special investigation. Analysis of the most recent review report of these 27 schools showed that their overall performance tended to be more effective than secondary schools overall. The previous review reports of one‑third of these schools had signalled a return after four-to-five years compared with 15 percent nationally. It is therefore probable that the secondary schools visited are more effective than all secondary schools nationally, and give a more positive picture than would be true for all schools.

Schools reviewed and schools responding to surveys

Sampling

An online survey about principal appraisal was provided for a random sample of board chairs. The sample error was six percent for primary schools and 14 percent for secondary schools.

Responses were received from trustees at 153 schools (52 percent response).

Table 1: Schools reviewed and board chairs responding to the survey

 

Primary schools reviewed

Secondary schools reviewed

Board chairs responding

National percentage Term 4, 2012

 

N=173

N=27

153 responses

N=2430

School type

   

%

%

Full primary

Contributing

Intermediate, middle school

Special

99

48

14

2

 

43

37

7

44

32

5

2

Composite (Year 1-15, Year 1-10)

Secondary (Year 7-15)

Secondary (Year 9-15, Year 11-15)

7

3

0

1

7

19

1

3

8

5

4

9

Location of school

       

Main urban

Secondary urban

Minor urban

Rural

80

13

18

62

19

1

5

2

62

10

10

17

53

7

12

29

Size of school

       

Very small

Small (101-400)

Medium (401-800)

Large(801-1500)

Very large (1501 )

17

43

72

28

13

0

2

12

9

4

3

13

55

19

10

10

25

37

19

9

Decile grouping

       

Low decile (deciles 1-3)

Medium decile (deciles 4-7)

High decile (deciles 8-10)

40

83

50

6

10

11

24

39

37

31

40

29

Note: percentages do not always add to 100 because of rounding.

Differences between the responding schools and schools nationally were tested using chi square tests. Differences that were statistically significant (P<0.05) are described below.

The primary schools visited also included seven composite schools, three Years 7-15 secondary schools, and two special schools. More of the primary schools included were full primary schools (Years 1-8) than nationally (61 percent compared with 54 percent), and fewer were contributing schools (Years 1-6) – 30 percent compared with 39 percent. The sample also included slightly fewer low decile and more medium decile schools than nationally. This is consistent with the exclusion from the evaluation of schools with longitudinal reviews (those ERO returns to after one-to-two years), which occur more often in low decile schools.

Trustee survey

Only 52 percent of boards responded. Rural and small schools were under-represented in the responses. The number of responses is too small for robust analysis of sub-groups, but there were no consistent trends in responses by location or size of school. Characteristics of schools tend to be linked. For example, most rural schools are small and most secondary schools are larger and in main urban areas.