5 BACKGROUND

5.1 1995 Trial

In June 1995, following a public call for tenders, the Australian Council for Educational Research (ACER) was commissioned to undertake a trial of procedures for collecting valid and reliable data on English literacy achievements of Australian students in Years 3, 5, and 10. The trial was commissioned by a management committee (Figure 5.1) established by the Steering Committee (see Introduction, page 3).

Figure 5.1Members of the management committee for the 1995 trial

PURPOSE

The purpose of the trial was to explore the feasibility of two alternative procedures for collecting reliable data on students' English literacy achievements with a view to their possible use in a 1996 Survey of the English literacy performances and practices of Australian primary and secondary school students.

AIMS

The specific aims of the trial were to:

TRIAL PARTICIPATION

A total of 174 teachers and 41 external assessors participated in the trial. Each teacher collected data on five students, resulting in 870 participating students. These students came from government, Catholic and independent schools in most States and Territories but did not constitute a nationally representative sample from these systems.

TRIAL PROCEDURES

The Trial investigated two alternative procedures for collecting reliable data on students' literacy achievements. Half the participating teachers and students were assigned to one procedure, half to the other. All students, teachers and principals completed questionnaires to provide information on student backgrounds and schools' literacy policies andpractices. At the completion of the trial, feedback was sought from participating teachers and external assessors.

Procedure 1 involved collecting data on the English literacy achievements of students by giving teachers a list of outcomes on computer disk and asking them to judge how often individuals demonstrated achievement of each outcome. All Procedure 1 students then completed a set of common tasks in reading, viewing, speaking, listening, and writing which were assessed by teachers using provided assessment criteria. Teachers' outcome judgements were compared with students' common task performances to explore the possibility of statistically moderating teachers' outcome judgements if necessary.

Procedure 2 involved collecting data on the English literacy achievements of students by having teachers work with external assessors to assess pieces of student work. The work assessed included common tasks completed under timed conditions, and specified classroom work collected in 'portfolios'. All assessments were made against assessment criteria developed by ACER.

To investigate inter-rater reliability, samples of student work (common tasks and portfolios) were collected centrally and independently reassessed. Teachers also were asked to submit their assessments of videotaped speaking performances to ACER.

The two procedures were evaluated and compared in terms of their validity, reliability, and feasibility as methods of collecting national data on students' English literacy achievements. Did each procedure provide evidence about a broad range of literacy skills? Was it seen by teachers and external assessors as providing a valid picture of individuals' literacy achievements? Were assessments made reliably? Could they be used as a basis for dependable national data? How practicable were the arrangements for data collection in terms of demands on class time and the cost of each procedure?

TRIAL RESULTS

Analysis of trial results showed a trade-off between validity, reliability, and feasibility in the collection of nationally comparable data on students' English literacy achievements.

Procedure 1 (providing outcomes on a computer screen and asking teachers to key in their judgements outcome-by-outcome) was administratively very convenient, but the trial suggested that it produced results which were unreliable and about which teachers were sceptical.

Procedure 2 (teachers and external assessors jointly assessing pieces of students' work against provided criteria) was found to be administratively demanding and resource intensive. On the other hand, teachers saw the common task-portfolio combination as capable of providing valid measures of English literacy achievement at Years 3 and 5. They also were very supportive of the collection of portfolio evidence even though it was time consuming.

The trial suggested that acceptable levels of reliability could be achieved for some categories of student work. The trial results produced high levels of inter-marker reliability on common tasks in reading, viewing, and listening. Acceptable levels of reliability appeared to be achievable on common task writing and perhaps speaking. The only components of student portfolios which produced acceptable levels of reliability were the writing components, and some of the speaking evidence.

The questionnaires developed and used in the Trial were found to be useful for collecting a range of evidence about student backgrounds and schools' literacy practices. Teachers found the questionnaires long and sometimes repetitive.

RECOMMENDATIONS

On the basis of ACER's report on the Trial results, the Management Committee recommended to the Steering Committee that a 1996 Survey proceed, at Years 3 and 5 only, using a modified form of Procedure 2 to collect achievement data, and a modified set of questionnaires to collect background data on literacy achievements. It recommended also that consideration be given to conducting a survey at the secondary level, using a similar methodology, at a later stage, when appropriate assessment materials had been developed.

In April 1996 the Federal Minister for Schools, Vocational Education and Training, Dr David Kemp, announced that the National School English Literacy Survey would take place. A Survey Management Committee (Figure 5.2) was set up to implement the Survey. On its advice, the Commonwealth commissioned ACER to undertake the work to collect valid and reliable data on the English literacy achievement of students in Year 3 and Year 5 only.

Figure 5.2Members of the National School English Literacy Survey Management Committee (1996-7)

5.2 Development of Survey Instruments

In 1995,ACER was commissioned by the Survey Management Committee to develop Year 5 literacy assessment instruments for the proposed 1996 Survey. The materials were to be similar to ACER's Developmental Assessment Resource for Teachers (DART) for upper primary students, which was used in the Survey trial. The assessment tasks were designed to be as appropriate as possible for the full range of students. (The trial materials were not suitable for a survey of this size because they had been published and widely exposed since their release in 1994.)

The developers of the achievement assessment instruments met at regular intervals with the Materials Reference Group, and supporting experts (Figure 5.3) established by the Survey Management Committee. The Reference Group included literacy experts and representatives of multicultural education associations, teachers of English as a second language associations, and Indigenous education groups.

A decision was made to use ACER's new, unpublished DART English kit, for middle primary students, which was under development at the time, for the Year 3 Survey assessment instruments

Figure 5.3 Members of the National School English Literacy SurveyMaterials Reference Group and supporting experts

The Management Committee established a Questionnaire Reference Group with supporting experts (Figure 5.4) to work with ACER to refine the background questionnaires used in the trial for use in the 1996 Survey.

Figure 5.4Survey Questionnaire Reference Group and supporting experts

Copyright © Commonwealth of Australia
DEST Web Site Privacy Statement
Disclaimer