- An introduction to MULTILIT
- Essential constituents of effective reading instruction for low-progress readers
- A description of the MULTILIT Program: predicates, programs and procedures
- General research methods
- Monitoring the performance of low-progress readers: development of the WARP
- Typical rates of progress of low-progress readers
1. An introduction to MULTILIT
For those of us who take literacy for granted, who can literally ‘take it as read’, it is difficult to imagine what it must be like for students who start high school with poorly developed, or nearly non-existent, literacy skills. The main aim of this project was to provide a thoroughgoing evaluation of the efficacy of an intensive, systematic, skills-based literacy program, known as MULTILIT, in redressing the literacy difficulties of older low-progress readers.
MULTILIT stands for ‘Making Up Lost Time In Literacy’. The MULTILIT Initiative is a research and development enterprise directed by Professor Kevin Wheldall from Macquarie University Special Education Centre (MUSEC). This Initiative comprises research and development into more effective ways of teaching older low-progress students experiencing severe difficulties in learning literacy skills, carried out in the MULTILIT classroom programs and clinic at MUSEC, and in MULTILIT outreach programs.
The MULTILIT Initiative has three main foci:
- Service provision
- Research
- Product/Program development
All students attending MULTILIT programs, both within MUSEC Special School and in various outreach settings, do so as a direct result of their special learning needs in the area of literacy, that is, being significantly behind their peers in reading and related skills. This is typically operationalised as being at least two years behind in terms of reading accuracy.
The generic classroom behaviour management model underpinning the Program is predicated on the principles and methods of 'Positive Teaching' while the overall orientation to teaching is avowedly non-categorical. This recognition that the problems of low-progress readers may stem, at least in part, from inappropriate teaching per se is carried through in the academic element of the initiative which provides a strong, systematic, skills-based literacy program predicated on methods and strategies of proven effectiveness as a result of research carried out within the MULTILIT Initiative at MUSEC, and elsewhere. The MULTILIT Reading Tutor Program, the focus of the literacy intervention, comprises instruction in word attack skills, sight word recognition and supported reading of natural language texts.
The Report details evaluations of efficacy which provide strong evidence for the effectiveness of the MULTILIT Program over three years of operation, 1996, 1997 and 1998. Other implementations of the MULTILIT Program (including variations) are also evaluated and reported.
2. Essential constituents of effective reading instruction for low-progress readers
As a result of the considerable body of reading research undertaken over recent decades, a great deal more is now known about both the knowledge and the skills necessary for becoming a proficient reader. The nature of a generic effective model of instruction and the specific knowledge and skills which need to be mastered in order to develop reading proficiency have been verified by a considerable body of research and are well documented and widely agreed upon by many researchers and practitioners.
The focus on instructional issues appears no longer to be confined to the ‘either/or’ dilemma of whole language versus phonics instruction. There is increasing acknowledgment that the ‘both/and’ approach is appropriate and desirable in providing a wide range of effective teaching and learning opportunities for students to enable them to acquire and develop reading skill. There appears to be a growing consensus that component skills of reading learned in isolation are of relatively limited utility in and of themselves, and that they have true value only when practised in meaningful, connected text.
The reading process
When skilled reading occurs, decoding is accurate and automatic, oral reading performance is fluent and expressive, and the cognitive process of constructing meaning is ongoing and purposeful. Skilled readers can effectively integrate information from the text with what they already know to construct ongoing meaning and demonstrate high order comprehension skills.
Reading appears to be effortless to the skilled reader only because the word recognition component of the reading process has become automatic to the point where there is no overt awareness of the complexity of the behaviour which is occurring. The skilled reader has developed automaticity in the lower order word recognition components of reading; he or she is able, quickly and accurately, to recognise individual letters and spelling patterns and to translate them into words with apparent ease.
A minority of learners appear to develop understanding of the alphabetic principle and learn to read with little or no explicit instruction. Probably the majority of children learn to read with instruction and, regardless of mode, acquire the alphabetic principle. For a significant number of struggling, beginning and low-progress readers, however, this is not the case. For these students intervention which delivers well sequenced, structured and explicit instruction in code emphasis activities is required, along with extensive practice to ensure mastery of the component skills of reading and their integration in the reading of connected text. Instruction for beginning readers and low-progress readers alike must be purposeful, strategic and based on methods of proven efficacy.
[Top of Point 2]
Models of reading instruction
Current research findings lead us to believe that a heavy emphasis on systematic phonics instruction combined with the reading of natural language texts is the most effective and efficient way to approach both initial reading instruction for the majority of learners and remedial instruction for low-progress readers. This view of reading, which acknowledges the simultaneous integration of orthographic, phonemic, syntactic and semantic cues, is referred to as an interactive model. In this model, learners are directly and explicitly taught to apply whole word recognition skills and phonic knowledge skills to decode words. These skills are then applied and practised to develop automaticity through the reading of a variety of meaningful, connected texts at an appropriate level.
The Spalding method of teaching reading incorporates a systematic, explicit approach but the component parts are initially taught to mastery in isolation and before progressing to reading words in context in books. Reading Recovery is also considered by some as following an interactive model but research in New South Wales has found it to be effective for only one child in three undertaking the program. The success of students in Reading Recovery programs has been shown, in fact, to be largely dependent upon their level of phonological processing skills on entering the program.
Systematic, skills-based reading instruction, as reflected in a truly interactive approach, is supported by the so-called simple view of reading. Two components of reading are identified in the simple view: first, that which allows language to be recognised through its graphic representations (decoding); and second, that which allows language to be understood (comprehension).
[Top of Point 2]
Decoding
The term decoding is used in its broadest sense to describe the process of translating written text into spoken words. More specifically it is used to describe the process of deciphering words through the use of phonological coding (phonic word attack skills), while the term word recognition is used to refer more specifically to recognising words through graphic information stored in memory (whole word recognition). Both of these strategies are necessary components of a reading program.
Understanding of the alphabetic principle is necessary in order to access words that have never before been seen in print. Beginning and low progress readers need to become proficient in both phonemic awareness and letter-sound knowledge. They need to develop phonic word attack skills.
Whole word (or ‘sight word’) recognition refers to the ability to identify familiar words without the need to analyse their component parts. Since the most frequently encountered one hundred words in print account for around half of all printed words encountered in text, it makes good sense to teach a corpus of familiar or frequently occurring words. It enables the novice learner to acquire a bank of high frequency sight words which can be retrieved directly from memory through lexical access. It introduces students to the idea of reading and facilitates early success in reading text which contains those previously learned words.
[Top of Point 2]
Comprehension
Around 75% of children will, regardless of the type of reading instruction received, discover the advantages of applying the alphabetic principle to reading. There appears to be general agreement that without this ability to identify words accurately and fluently, students will be inhibited in their efforts to develop the higher order skill of comprehending written text. If individual words in a text cannot be identified nothing else in the reading process will happen.
Linguistic comprehension, the second component identified in ‘the simple view of reading’, is the ability to take semantic information at the word level and construct meaning at the sentence level. Reading comprehension requires the use of the same skills as oral comprehension but in response to print rather than in response to speech. Reading comprehension is developed through regular practice in reading natural language texts and the explicit teaching of specific skills. According to the simple view of reading, word recognition skills and linguistic comprehension are the two essential prerequisites for reading text.
[Top of Point 2]
What do we know about low-progress readers?
Across the very substantial body of research on reading there is undeniable evidence of the strong relationship between word recognition and reading comprehension. The vast majority of low-progress readers show deficits in their phonological processing skills, that is, in their ability to use phonological information to decode written text. The primary factor preventing the vast majority of low-progress readers from improving their reading performance is their poor word decoding skills. They are unable to match letters with their corresponding sounds to decode words. In other words, their phonic word attack skills are poor.
[Top of Point 2]
Functional literacy and decoding skills
The term ‘functional literacy’ is a level often used as a benchmark measurement of reading. Functional literacy is probably equivalent to a reading age of around ten to ten and a half years and is achieved by most students by the end of Year 5 (or the sixth year of schooling). A functional level of literacy needs to be attained by a student if he/she is to meet the minimum reading demands which are made in lower secondary school, or even by the final primary years.
Some low progress readers with poor phonic word attack skills can and do rely on other strategies to read text which are less efficient but, none the less, effective to some degree. Some readers may even be able to reach a functional level of reading while maintaining relatively poor phonic word attack skills, by acquiring a very large lexicon of sight words, for example. As a general teaching strategy for reading, however, this is not to be recommended - it is far too risky.
While most reading researchers and practitioners would probably agree that learners should make use of both context cues and letter-sound cues when reading words, the critical point is that the use of phonic word attack skills is the strategy more likely to result in consistent and independent accurate word recognition. Furthermore, there is a general consensus that readers need to develop a bank of high frequency sight words which can be recognised accurately and fluently without the need for analysis of their component parts.
Effective reading instruction for low-progress readers requires: (i) a well developed phonic word attack skills program; (ii) opportunities to acquire and practice a bank of useful, high frequency sight words; and (iii) regular practice in reading meaningful, connected text in a supportive context. Such a truly interactive model is operationalised within the MULTILIT Reading Tutor Program. Instruction focuses on phonic word attack skills (understanding and applying the alphabetic principle to decoding words), sight word recognition (developing lexical access in memory) and the practice of these skills through reading natural language in meaningful and age appropriate text matched to the child’s reading level, within a supportive tutoring context.
3. A description of the MULTILIT Program: predicates, programs and procedures
Children who have failed to learn to read by the usual methods in the first few years of school need intensive systematic reading instruction. MULTILIT aims to provide precisely that and is predicated upon:
A non-categorical approach to instruction
Categorising or labelling a child with a disability type or diagnosis identifies that child with a group who, in theory, have similar instructional needs. It also suggests that students who have been diagnosed with different learning problems will respond differentially to different interventions. Research shows that this is not, in fact, the case. The majority of special educators today favour what is known as 'a non-categorical approach' to teaching students with special needs and are committed to the conviction that all children can learn, given effective instruction. The forms of pedagogy to be employed, however, are determined not by the nature of the child's disabling condition but by a needs-based appraisal of the student's current level of functioning. The categories commonly employed to describe students with apparently similar disabilities and difficulties are of little or no use in determining the appropriate pedagogy to be employed with a particular student. Categorisations based on measured IQ, for example, have proved to be particularly unhelpful in terms of informing instructional decision making. Measured intelligence has very little relevance to how we go about teaching a child to read.
[Top of Point 3]
Positive Teaching for effective classroom behaviour management
Effective teacher management of classroom behaviour is an essential prerequisite for effective classroom teaching and learning to take place. If the teacher is prevented from teaching, or students are prevented from getting on with their academic work, as a result of either their own inappropriate behaviour or that of other students, then clearly little of educational value is likely to be achieved. Amount of time spent appropriately academically engaged is vital to the progress of both able and less able students.
The child whose behaviour is continually disruptive, or who is even quietly, but regularly, ‘off-task’ is seriously educationally disadvantaged, since academic engaged time is one of the most important correlates of academic progress. Successful classroom behaviour management ensures that maximal learning time is delivered. We advocate Positive Teaching to maintain attention to task.
Positive Teaching derives from the 'Positive Teaching Project' conducted by Kevin Wheldall and Frank Merrett from the Centre for Child Study at the University of Birmingham during the 1980s. Wheldall and Merrett produced a series of intensive, skills-based training packages for teachers in effective classroom behaviour management, based on their program of observational and experimental research in classrooms.
'Positive Teaching' is based on applied behaviour analysis and observed classroom processes. In essence the procedures advocated 'accentuate the positive' by requiring teachers to focus their attention on students when they are behaving appropriately ('catch them being good') rather than continually being on the lookout for, and reprimanding, inappropriate behaviour, in line with traditional reinforcement theory. Positive Teaching advocates increasing teacher praise and approval and decreasing disapproval and reprimands. Reprimands are used very sparingly, specifically and privately and in a positive context overall. A number of procedures are advocated which have been shown experimentally to bring about improved classroom behaviour, often requiring teachers to change their own teaching behaviour or to make changes in the classroom environment. Classroom seating arrangements, for example, has been an area of particular interest in which Wheldall has carried out a series of studies. Positive Teaching procedures are thus characterised by a concern with ecological classroom variables and setting events for classroom behaviour, as well as contingency management procedures. They are also child centred in the sense that student initiations and negotiations constitute critical components of the approach. For example, students themselves can be directly involved in bringing about behaviour change by employing self-monitoring strategies to determine whether they are on- or off-task.
Positive Teaching is taught to teachers by means of skills-based training comprising five one hour sessions, preferably held weekly after school. Following course attendance, teachers appreciably decrease their use of reprimands, increase their use of praise and reward and, most importantly, bring about substantial positive changes in the levels of on-task behaviour of their classes.
All MULTILIT teachers attend Positive Teaching Package courses and are thoroughly trained in Positive Teaching methods. Classroom observations have shown that, consequently, MULTILIT teachers typically praise about ten times the rate typically observed in regular classrooms. In MULTILIT programs, extensive use is made of student self-monitoring and self-recording strategies. Extrinsic reinforcement is employed at the beginning of the program and is gradually faded, with points and stickers being harder to earn towards the end of a student’s time in the MULTILIT program. MULTILIT teachers are also aware of the key antecedents influencing classroom behaviour, such as classroom seating arrangements and using appropriate curriculum materials.
[Top of Point 3]
A balanced, ‘interactive’ model of effective literacy instruction
The theoretical rationale for effective literacy instruction underpinning MULTILIT emphasises the need for a balanced approach combining the best of both ‘bottom-up’ and ‘top-down’ methods, as described in Chapter 2.
The careful planning and systematic delivery of an intervention is not inconsistent with teaching in natural environments and using natural routines. Systematic instruction that can be accurately described and independently verified is certainly not inconsistent with flexible or learner-centred interventions.
The components of effective instruction may be summarised as planning instruction, managing instruction, delivering instruction, and evaluating instruction. Instruction is planned for each MULTILIT student based on criterion-referenced tests which identify the skills the student needs and help place the student at the appropriate level in the various programs employed. Managing instruction is carried out by means of Positive Teaching, as previously described, to create a positive classroom atmosphere with a task-oriented focus, and a clear definition of rules. Systematic approaches to delivering instruction are a key feature of teaching in MULTILIT, including some use of direct instruction. This involves explicit teaching of rules and strategies, using sufficient examples and non examples, frequent responding by students (both chorally and individually), and frequent verbal praise. Students are given clear, unambiguous instructions and are kept informed of their academic performance through the use of corrective feedback. When assessing a student’s instructional environment, the focus should be on gathering information relevant to instruction rather than on complex diagnostic procedures. Formative evaluation necessary to make such ongoing instructional decisions is taken regularly in MULTILIT. This formative evaluation is used to make instructional decisions so that each student is consistently placed at the appropriate level in each component of the program.
MULTILIT teachers are trained in effective instruction and data-based teaching, in Positive Teaching, and have particular specific expertise in implementing the MULTILIT Reading Tutor Program which forms the core of the MULTILIT curriculum.
[Top of Point 3]
The MULTILIT Reading Tutor Program
The most effective remedial programs for low-progress readers involve intensive, systematic instruction in three main areas: phonic word attack skills; sight word recognition; and supported book reading in a one-to-one context. The MULTILIT Reading Tutor Program incorporates all three of these key features and forms the core of what we offer to low-progress students accessing the classroom programs and clinic within the MULTILIT Initiative at MUSEC and in MULTILIT outreach settings. The program was specifically designed for teaching low-progress readers in Year 2 and above (about seven years upwards) who are reading at a level considerably below what might be expected for their age and who have not acquired the basic skills needed to become functional readers. The MULTILIT Reading Tutor Program comprises three distinct elements.
MULTILIT Word Attack Skills
Children learning to read primarily need to learn how to ‘crack the code’ - how to decode words they have not previously encountered by breaking words down into their component phonic parts. Low-progress readers need intensive, systematic instruction both in how to break up (‘segment’) words into their component letter sounds and, even more importantly, how to ‘blend’ component letter sounds into words. MULTILIT Word Attack Skills is designed to do precisely this: to teach older low-progress readers the phonic skills essential for rapid decoding.
MULTILIT Sight Words
Sight words are words that can be read automatically on sight without recourse to decoding strategies. When learning to read, it makes good sense for children to learn a small corpus of very common sight words so that they will not need to struggle to decode every single word that they encounter in a sentence. MULTILIT Sight Words systematically teaches the automatic recognition of 300 high frequency sight words.
MULTILIT Reinforced Reading
Reinforced Reading is a program developed to enhance the student’s independent reading skills and is based on the set of tutoring strategies for use with older low-progress readers known as Pause, Prompt and Praise (PPP). The aim of the tutoring session is for the tutor to listen to the low-progress reader read natural language books at an appropriate level of difficulty for up to fifteen minutes. The tutor is trained to pause for up to five seconds or wait until the end of a sentence when a mistake is made to permit time for self-correction. If no self-correction occurs, the tutor supplies up to two prompts in the form of a graphophonic prompt ('How does this word begin?', 'What sound do these letters make?'), a contextual cue ('Does that word make sense?') or a re-read prompt ('Read that again from the beginning of the sentence.'). When the student correctly reads a sentence or paragraph, self-corrects without a prompt, or successfully uses a given prompt to identify a word, specific praise is given.
The three elements of the MULTILIT Reading Tutor Program: MULTILIT Word Attack Skills, MULTILIT Sight Words, and MULTILIT Reinforced Reading, form the bedrock of any MULTILIT program.
[Top of Point 3]
Daily activities in the MULTILIT Program
MULTILIT students spend the first two hours of their school day in their home group of eight students of varying ages and abilities under the direction of one teacher. During this first session of the day, students are called individually in turn to their one-to-one session with their teacher (typically for about 15 minutes), and to their Reinforced Reading (PPP) session with a teacher’s aide or parent/community volunteer (15 minutes). In addition, while in their home group, students spend approximately one and a half hours carrying out individual work or activities set out in a work contract. Contracts are designed to encourage responsibility for organising and completing work independently. Each student has an individually designed contract tailored to meet his/her particular needs and students are rewarded when they complete all contract work assigned for the week.
MULTILIT one-to-one teaching activities provide sequential learning for students who are behind in reading skills. The MULTILIT Reading Tutor Program forms the core for these lessons as previously noted. At MULTILIT, the program is implemented by a teacher, but a teacher’s aide, trained volunteer, trained parent, or skilled peer tutor working under direction of a teacher could also implement the program. Each child is tested on entry to the MULTILIT Program to place them at the appropriate levels in each component of the MULTILIT Reading Tutor Program. As we have already seen, the lessons in this program concentrate on decoding skills (MULTILIT Word Attack Skills), accurate and automatic recognition of sight words (MULTILIT Sight Words), and the practice and generalisation of these skills using the connected reading of real text (in MULTILIT Reinforced Reading). Also included in the one-to-one session, and depending on the student’s need, is an auditory awareness training program, and repeated reading, designed to build reading fluency of text and help in the generalisation of decoding and comprehension. Others will require repeated reading designed to improve reading accuracy. Some students, who specifically need to improve their comprehension skills have appropriate activities included in their one-to-one session. Together, the components of the MULTILIT Reading Tutor Program and the supporting materials used during the one-to-one session comprise a set of very powerful teaching tools.
MULTILIT utilises other teaching strategies and programs to support and extend the instruction provided by the MULTILIT Reading Tutor Program and to provide opportunities for students to generalise their skills. Following a brief recess break, the daily routine resumes in MULTILIT with the format changing from individual to group instruction. Students work in small groups of approximately eight students, based on ability level. Each group sits in a semi-circle desk formation, with the teacher in the middle of the circle so that eye-contact can be made with each student. There are three 30 minute lessons between recess and lunch, all of which are carried out in groups. These lessons are spelling, reading, and comprehension/writing. In addition to these lessons, students and teachers engage in 10 minutes of Uninterrupted Sustained Silent Reading (USSR), spend five minutes having reward stickers awarded for appropriate behaviour during the morning session, and enjoy a 15 minute session of Serial Reading immediately prior to the lunch break. Four hours of intensive instruction in literacy are thus delivered between 9.00 and 1.15 pm daily, with only a 15 minute break for recess. The specifics of the group lessons are detailed below.
Spelling Mastery
The main spelling component of the MULTILIT program is delivered by means of the SRA Spelling Mastery program. Teachers also program for generalisation using a variety of activities such as dictation, story writing and computer spelling games. These activities allow the student to practise their newly acquired spelling skills in less structured, more naturalistic and meaningful contexts.
Reading Rigby lesson
During group reading lessons, books are introduced and discussed, opportunities are taken to build vocabulary, and the story is read aloud by the students in turn with the teacher using the Reinforced Reading strategies of Pause, Prompt and Praise. Although any graded reading scheme could be used for this group lesson, we have found the (now out of print) Reading Rigby series to be finely levelled and very appropriate for low-progress readers.
Comprehension
Different skills are targeted during the comprehension lesson using the graded SRA Multiple Skills comprehension program. These include understanding the main idea of a story, finding facts located in a passage, understanding context clues, making inferences, and the extension of vocabulary. Students are exposed to a variety of comprehension formats to help them generalise their skills.
Uninterrupted Sustained Silent Reading (USSR)
Teachers demonstrate the importance of reading as a recreational activity by modelling the desired behaviour of quiet reading. This is sometimes referred to as USSR or Uninterrupted Sustained Silent Reading. For 10 minutes each day, all MULTILIT students and teachers engage in Uninterrupted Sustained Silent Reading. USSR promotes reading as a pleasurable activity, and, along with Serial Reading, provides an important element in the interactive model that MULTILIT seeks to deliver.
Serial Reading
For the purposes of Serial Reading, MULTILIT students are divided into junior (Years 3 and 4) and senior (Years 5 and 6) groups. For 15 minutes a day a teacher reads from a novel selected from books aimed at the interest level of the students.
Although the primary focus of MULTILIT is literacy, the full-time program includes instruction in mathematics and other key learning areas in the afternoon sessions where MULTILIT students practice their reading and writing skills.
[Top of Point 3]
4. General research methods
The same basic evaluation procedures were employed for all efficacy evaluations on all of the various MULTILIT sites. All students were thoroughly assessed on entry to and on exit from the MULTILIT programs on a variety of standardised or curriculum-based measures including the Neale Analysis of Reading Ability - Revised, the Burt Word Reading Test, the South Australian Spelling Test, the Wheldall Assessment of Reading Passages (WARP), the MULTILIT Word Attack Skills Placement Test, and book level assessments. Such measures allow for comparisons of overall growth in literacy skills against standardised levels of performance, to ensure that MULTILIT is meeting the objective of rapid progress toward functional literacy.
Students were typically assessed on entry to the programs, retested after two terms, which for most students was the end of their program, and then retested twice subsequently at two term intervals (roughly every 5-6 months) to assess maintenance and generalisation of literacy skills learned.
We endeavoured to demonstrate the low rates of progress typically made by groups of low-progress readers in regular school programs. Where such groups are comparable to treatment groups, we may compare progress during MULTILIT programs with progress typically made by low-progress readers not receiving MULTILIT intervention.
Limited use was also made of (more qualitative) small-scale survey and focus group techniques to add a more descriptive dimension to this efficacy research.
[Top]
Target group and composition of the samples of students
The target group for the MULTILIT programs comprised older, low-progress readers; specifically, students in Years 3 to 7 (aged 8-13 years) who were at least two years behind their chronological age in terms of reading age as measured by Neale Analysis of Reading Ability - Revised (accuracy) but who (preferably) had made a start in learning to read and who could read simple text (ideally have a minimum reading age of, say, six and a half years). Having said this, complete non-readers were also included in the sample, as were a small number of Year 2 students.
[Top of Point 4]
Assessment procedures
Students involved in this research project were tested on the battery of tests of reading and related literacy skills and were typically retested five months later after two terms (20 weeks of instruction), to determine how much progress had been made. Data collection was completed by trained and experienced research officers or by trained research assistants supervised by members of the highly experienced team of MULTILIT literacy research officers.
[Top of Point 4]
Test instruments/measures employed in the studies
The following reading and spelling tests were administered individually to the students in the period 1996-98:
1. The Neale Analysis of Reading Ability - Revised
The principal reading assessment utilised in this evaluation was the Neale Analysis of Reading Ability - Revised which aims to provide global indicators of performance in two of the main skills involved in reading: reading accuracy and reading comprehension.
2. The Burt Word Reading Test
The Burt Word Reading Test has a long history in educational research and is a measure of single word recognition in isolation.
3. The Wheldall Assessment of Reading Passages (WARP)
The WARP is a new curriculum-based measure of reading which consists of a series of five 200 word passages, each passage comprising an entire story. The development of this test is described in some detail in Chapter 5.
4. South Australian Spelling Test
Spelling performance was assessed using the South Australian Spelling Test. The revised norms for this simple spelling test provide estimates of spelling age (based on a sample of South Australian students) from 6 to 15.5 years.
5. Book level
Book level data were also collected to measure progress in the curriculum. When students enter a MULTILIT program, they are placed on appropriate book levels at instructional level (90-95% accuracy) and are promoted to succeeding levels as they progress. Reading Rigby levels were employed in this research.
The assessment battery was adjusted over the course of the evaluation. Some data included in the project were collected prior to the evaluation study being contracted, hence there is variability in sample sizes for some measures. The Neale Analysis of Reading Ability - Revised was the constant dependent variable throughout the research.
[Top of Point 4]
Follow-up testing
Where possible, students completing the standard two term MULTILIT program were followed up twice at approximately six monthly (two terms) intervals to test for maintenance and possible generalisation of the skills learned during the MULTILIT program. The Neale Analysis, only, was administered, for reasons of economy and comparability across program intakes, at six month and twelve month follow-up testing.
[Top of Point 4]
Effect size
In line with contemporary best practice in empirical educational research, we have sought to go beyond the concept of mere statistical significance and to provide estimates of the size of the gains or effects obtained. To this end we have employed the concept of the effect size. The effect size is the size of the difference between experimental and control group means divided by the standard deviation of the control group yielding, in effect, a Z score or normalised standard deviate. Effect sizes are typically classified as small (0.20), medium (0.50) or large (0.80).
[Top] [Top of Point 4]
5. Monitoring the performance of low-progress readers: development of the WARP
Effective monitoring of the progress of low-progress readers is critical for effective remediation. If we are to remediate effectively with low-progress readers it is essential to know whether the individual child is progressing and, if so, at what rate the child is progressing. Empirical research strongly supports the use of such ongoing measurement and has found that it leads to measurable gains since it provides direct feedback to teachers on the efficacy of the instruction they are providing.
Curriculum-based measurement (CBM) is a set of procedures for measuring student proficiency and indexing growth in the curriculum. This set of procedures was developed in response to the problems associated with the use of standardised, norm-referenced tests.
The Passage Reading Test (PRT) represents an alternative, curriculum-based measure of reading progress to more traditional reading tests. The PRT requires students to read from any appropriate grade level basal reader for just one minute. The number of words read correctly in this period is the index of the student’s current level of reading performance. This simple measure of reading has been shown repeatedly to correlate highly with other measures of both reading accuracy and reading comprehension. Moreover, such measures may be used very effectively to track reading progress and, thereby, the efficacy of reading instruction provided.
If performance on curriculum-based passage reading tests is to be used to monitor progress toward a long term goal, that goal needs to be identified. We suggest functional literacy as an appropriate long term goal for low-progress readers.
[Top]
Development of the WARP
In 1995 the first author (Wheldall) commenced development of a set of generic passages for the curriculum-based measurement of reading. In its original form, the Wheldall Assessment of Reading Passages (WARP) consisted of fourteen 200 word passages. These passages, of roughly equal difficulty and each comprising an entire story, were specially written to preclude the possibility of students having encountered the passages before. The development of the WARP is summarised here since it was a major dependent variable in the efficacy evaluations of MULTILIT subsequently reported.
Two preliminary small scale pilot studies were conducted to provide data on the degree to which the passages were similar in terms of difficulty level, both across and within passages. In the first pilot study, a group of 21 low-progress readers from Years 5 and 6 were tested on the original 14 WARP passages. The number of words read correctly in the first minute (WPM) and also the number of words read correctly per minute averaged over the whole passage (WPP) were calculated for each passage. The two alternative scoring methods, WPM and WPP correlated highly on all 14 passages (0.94 to 0.99). Moreover, the 14 passages scored for WPM intercorrelated at 0.84 to 0.98 and at 0.89 and 0.98 for WPP. This indicates excellent parallel or alternate form reliability.
Five passages identified in the first pilot study as being the most similar from the original fourteen passages were subsequently administered to a second incidental sample of 21 low-progress readers from Years 6 and 7. Once again, the means and standard deviations for the five passages were very similar. Correlations between the two forms of scoring were exceptionally high (0.96-0.99) and highly significant (p<0.001), as were intercorrelations between the five passages both for first minute only (0.83-0.92) and whole passage modes of scoring (0.91-0.96).
A larger sample of 190 participants in Years 3 to 7 were subsequently assessed on the same five passages. (38 classroom teachers had been asked to identify four "average readers" and one "low-progress reader".) The high alternate forms reliability found in the pilot studies was replicated with this larger sample with coefficients ranging from 0.95-0.96 and 0.96-0.98, for WPM and WPP respectively. In addition, internal consistency was extremely high, with correlations between WPM and WPP of 0.99 for all five passages. Mean scores (and standard deviations) for WPM and WPP provided further support for the similarity of the five passages. It was also shown that the WARP very effectively differentiated low-progress readers from regular progress readers, the average older low-progress reader reading the WARP passages at below Year 3 level for regular readers. On the basis of these studies, it was also concluded that the small increase in reliability and validity gained by having students read the entire passage was not worth the extra time and effort involved, given the very high correlations found between WPM and WPP. In the WARP, a measure of oral reading fluency is gained by having students read passages for one minute only.
In order to be confident that the WARP is valid (that is, that it measures what it claims to measure) performance on the test was compared with performance on other established tests of reading ability. This exercise establishes criterion-related validity. The first validity study, involving 146 low-progress readers, examined the relationship of the WARP to the Neale Analysis of Reading Ability - Revised. Alternate forms reliability was again found to be extremely high (0.94-0.96). Similarly, mean scores for the five passages ranged from 77 to 82 and the standard deviations were very similar. Highly satisfactory correlations were found between the WARP and reading accuracy on the Neale (0.78-0.80). The relationship between Neale comprehension and the WARP, however, was lower than expected (0.49-0.55).
The main purpose of a second validity study was to examine the criterion validity of the WARP on a sample that was representative of the full range of reading ability. This was achieved by examining the relationship of the WARP to the Neale Analysis and the Burt Word Reading Test. The sample comprised a group of 50 regular students from a Sydney primary school; 10 students from each of Years 2 to 6. The WARP passages, and the mean score over the five passages, correlated highly with Neale accuracy scores at 0.85 on average, with coefficients ranging from 0.84-0.87 (p<0.001). Good correlations were also found between the WARP and Neale comprehension scores (0.67-0.72, p<0.001). The Burt Word Reading Test also correlated highly with the WARP, with coefficients ranging from 0.83-0.85 (p<0.001). The five passages of the WARP also correlated highly with each other, with coefficients ranging from 0.94-0.96 (p<0.001). The similarity of the passages was further supported by the closeness of the mean scores which were all within 3 words per minute of each other on average, with very similar standard deviations.
[Top of Point 5]
Comparison data on the WARP by grade
In the large scale study completed during the initial development of the WARP, some preliminary, tentative findings on performance on the WARP across grade (Years) were provided. A more extensive data collection exercise on the WARP was subsequently undertaken which provided comparison data based on a much larger sample of 1011 students from 43 classes of primary school students in Years 2 to 6.
The five passages again yielded very similar means and standard deviations, the means differing by no more than four words per minute. Moreover, the inter-correlations between the passages were all over 0.95. These results confirm on a very large and representative sample that the five passages yield highly similar results and may be regarded as parallel forms.
The five passages also yielded very similar results within each age group and WARP score was shown to rise steadily by grade level up to Years 5 and 6 where a ceiling effect appeared to set in. There was also a strong effect for gender of students with girls outperforming boys on all passages by around 13 words per minute on average. The table below presents rounded mean values for the WARP, averaged over the five passages, for Year 2 to Year 6, together with the 50% range limits. Values below the lower 50% range figure indicate levels achieved by the bottom 25%; values above the upper figure indicate levels achieved by the top 25%. These figures provide comparison data by which to evaluate the performance of individual students. Low-progress students may be operationally defined as those scoring below the 50% range limit for their age, ie. the bottom 25%.
Average reading levels for the WARP for Years 2 to 6 (words read correctly per minute)
|
Year |
Rounded Mean |
50% Range |
|
2 |
84 |
51 - 109 |
|
3 |
109 |
85 - 135 |
|
4 |
118 |
93 - 144 |
|
5 |
135 |
109 - 163 |
|
6 |
139 |
112 - 172 |
[Top of Point 5]
Sensitivity of the WARP to progress in reading performance
One of the key incidental questions to be answered by this research program was to determine the sensitivity of the WARP to increasing change/progress in reading performance. Chapter 7 of the Report describes in some detail the performance of students attending the MULTILIT programs who attended for the whole year in 1997 and in 1998. This provides evidence for the sensitivity of the WARP.
For 19 low-progress readers from the 1997 MULTILIT cohort we have full data sets at February intake, in June after almost two terms in the program, and in November, almost at the end of the program. With a mean age of just under 11 years, this group were, on average, over three years behind in terms of reading accuracy. Increasing mean reading age for accuracy from 87 to 100 to 110 months was accompanied by a steady increase from 45 to 77 to 100 in the mean number of words read correctly per minute as measured by the WARP.
Similarly, 18 students attended the MULTILIT Program for the whole of 1998 with a mean age just under 9 years and who were nearly three years behind in terms of reading age for accuracy. These students were assessed on the Neale Analysis, the Burt and the WARP on entry in February, in July after two terms in the program, and again in December at the end of the year after two further terms in the program. Students were also assessed on the Burt and the WARP after one term in April and after three terms in September.
These students made mean gains over the school year (just over ten months between initial and final testings) of around two years in reading performance. On the Burt, students averaged about six months’ growth each term. Reading performance is tracked over the equivalent of two years’ growth in reading performance by successive growth in mean WARP scores in successive terms from 23 to 39 to 62 to 80 to 89 words read correctly per minute, thereby demonstrating the sensitivity of the WARP to change in performance. The influence of practice effects on these results is unlikely to be very great for low-progress readers since practice effects are dependent upon the very form of incidental learning which low-progress readers find particularly difficult.
The development of a reliable, valid and sensitive measure of reading progress which takes literally only one minute to administer will present realistic opportunities to special educators, reading educators and, ideally, regular classroom teachers for tracking the reading progress of their students, on a regular basis.
[Top] [Top of Point 5]
6. Typical rates of progress of low-progress readers
It is clearly anticipated that students who are experiencing extreme difficulty in acquiring the basic skills of literacy will achieve lower scores on standardised measures of reading and on curriculum-based measurements than students who are progressing at a rate roughly commensurate with their chronological age. It is unclear, however, just what rate of growth is typical for low-progress readers in the primary/early high school years. Such information is important in assessing the efficacy of specific literacy interventions such as MULTILIT by providing a true baseline against which such interventions may be evaluated; a yardstick, in effect, by which to compare the effect of literacy intervention. Previous research provides evidence in support of the obvious (that low-progress readers make low rates of progress) but varies in the estimates provided of the rate of progress. We wanted to know the typical rates of progress targeted low-progress readers would make prior to entry into MULTILIT programs.
[Top]
A sample of Year 4 and 5 students assessed prior to, and subsequently upon, entry into the MULTILIT Program
This study aimed to establish typical rates of literacy growth of low-progress students failing to acquire basic literacy skills, prior to their entry into the MULTILIT Program (see Chapter 7). Twenty-three students in Years 4 and 5, identified as low-progress readers by their teachers, were assessed in late September 1996 on the Neale Analysis of Reading Ability - Revised and again in early February 1997, prior to entry into MULTILIT. These students, after having attended school for an average of 6.14 years had gained only six months on average per year in reading accuracy and only six and a half months per year in reading comprehension. The mean gain over the 4 month period prior to MULTILIT entry was minus 0.78 months (standard deviation 8.10 months) for Neale accuracy and 2.91 months (standard deviation 9.24 months) for Neale comprehension. By Years 5 and 6, these low progress readers had basically stopped making any progress in reading accuracy but were still making limited progress in reading comprehension.
[Top of Point 6]
A sample of Year 5 and 6 students assessed prior to, and subsequently upon, entry into the Schoolwise Program
Seventeen students in Years 5 and 6 were assessed on the Neale Analysis in October 1998 to confirm their eligibility for entry into the Schoolwise Program (see Chapter 8) and were assessed again in early February 1999. On the Neale accuracy measure, they had typically been gaining only six months on average per year. On the Neale comprehension measure, these results showed that the students had typically been gaining less than eight months per year. The mean loss over the four month period prior to Schoolwise entry was minus 3.82 months (standard deviation 6.99 months) for Neale accuracy and minus 6.53 months (standard deviation 0.08 months) for Neale comprehension. At the very least, this study casts doubt on the instruction provided being sufficiently robust, from a maintenance point of view, if it washes out over the period between testings (including the holiday period).
[Top of Point 6]
A sample of low-progress Year 7 students
Data informing this issue were also available from a sample of 57 very low-progress Year 7 students assessed before and after an eight month interval on measures of reading and related skills. The participants in this study were identified by their high schools as being very low-progress students in terms of literacy, falling within the bottom 6% for NSW. All students were assessed in Term 3 when they had been attending high school for over six months and were retested at the commencement of Term 2 of the following year, after approximately eight months, to determine how much progress had been made following low-intensity literacy intervention carried out by the schools. For reading accuracy and reading comprehension as measured by Neale Analysis, there was evidence of four and five months average growth respectively, over the eight months. These very low-progress readers in Year 7/8 typically made only very modest progress even when afforded targeted literacy intervention (albeit non-intensive in nature).
[Top of Point 6]
A planned study specifically designed to address the issue
As a follow-up to these findings based on incidental samples of students, we initiated a study specifically addressing this topic. The aim of the study was to attempt to establish typical rates of literacy growth in low-progress primary aged students (Years 3 to 5) who were struggling to acquire basic reading skills. Of 68 students originally tested toward the end of Term 2, only 24 met the criterion of being at least two years delayed in reading performance, with a further 17 being at least one year delayed. These 41 participants were retested six months later in December (Term 4). These students, after being at school an average of 5.3 years had been falling behind by five to six months on average per year.
The mean gain over the 6 month period for Neale accuracy was 5.51 months and 4.34 months for Neale comprehension. The mean rate of progress was thus greater during the 6 month period covered by the study than gains typically made over 6 month periods during the previous years at school. The gains made by the Year 5 students on Neale accuracy and comprehension appeared to be somewhat lower than for Years 3 and 4 suggesting that progress declines as low-progress readers reach the final years of primary schooling where the emphasis on basic skills acquisition will have been reduced. It should also be noted that a condition of being permitted to carry out this research study was that the results were to be made available to the teachers immediately after the first testing and, hence, teachers/schools might have been prompted to initiate reading interventions for the targeted students which would not otherwise have occurred.
In summary, some groups of low-progress readers appeared to make little or no progress and some even appeared to go backwards. Others made modest gains, on average, of about three to four months over two terms while other groups of students (many of whom were not low-progress readers as defined for the purposes of this research) made almost average gains once their teachers were alerted to the problem. If pressed to provide an estimate of the likely progress of older low-progress readers (Years 5 to 8) who are at least two years behind in terms of reading skill, and who are offered either no or only limited non-intensive remedial support, then we would conclude that progress of about half normal rate is probably typical.
[Top] [Top of Point 6]
7. Evaluations of MULTILIT Programs and follow-up studies
The various programs and manifestations of the ‘Making Up Lost Time In Literacy’ or MULTILIT Initiative all have their origin in the University-based MULTILIT Program which operates on a full-time basis for primary aged students in our Special School at Macquarie University Special Education Centre. Research and development over many years has led to what we believe to be, and what is now supported by a substantial body of evidence as being, a highly effective set of instructional strategies. Clustered together, these instructional techniques constitute the MULTILIT Program.
Evidence for the efficacy of the full-time MULTILIT Program is presented over the three years 1996 to 1998. Over the period of this evaluation, various models of the MULTILIT Program were trialed, including a program for younger students known as the FAST (Focus on Academic Skills Teaching) Program. Full-time programs for primary school aged students having difficulty in acquiring basic literacy skills varied in length of time and were offered to different aged groups of students over the three years covered by this report. For the purposes of this evaluation, we have reported the performance of students after two terms (about 20 weeks of instruction or one semester) in the program. Some data are available for a smaller sub-group of students who participated in the program for four terms (or a whole school year). The results for each of the six successive intakes are detailed in the main Report, as well as a thorough analysis of the total sample of 142 students which is summarised here. Students from Years 2-6 are represented in the total sample although the full-time MULTILIT Program principally targets students in Years 3-6.
[Top]
Results of the MULTILIT Program 1996 to 1998
Students participating in the University-based program were referred by teachers, school counsellors or parents who were concerned that the student was not developing adequate literacy skills in their regular school environment. The group of students on whom we have data were, as a result, a clinical sample. The mean chronological age of the 142 students comprising the total sample at the commencement of their two term programs was 10 years and 5 months. Boys accounted for 75% of the group, with students from Years 5 and 6 being heavily represented as a result of the intake policy (Year 5 and 6 students only) in 1996.
At program commencement, the average reading age for reading accuracy for this group of students was 87 months (7 years: 3 months), and for reading comprehension 94 months (7 years: 10 months). Students were over three years (38 months), on average, behind their chronological age in reading accuracy, and over two and a half years (31 months) behind for reading comprehension.
After two terms of instruction within the MULTILIT Program, the 142 students made mean gains of 15 months in reading accuracy, and 13.5 months for reading comprehension. On a smaller subset of 97 students for whom WARP data were available, students could read 38 words more words correctly per minute than at pre-test, an increase of 96%. For the 85 students who completed the South Australian Spelling Test, a mean gain of 16 months was made in two terms of instruction, and a mean gain of 15 months was made on the Burt Word Reading Test for the 66 students who completed it. The results for the total group after two terms of instruction are presented in the table below together with gain scores, and statistical information, including effect size. All of the gains made were highly significant (p<0.001).
Means (and standard deviations) of the Relevant Literacy Variables and the Resultant Gains for the Total Sample
|
Literacy Variable |
N |
Pre-test |
Post-test |
Gain |
F |
p |
ES |
|
Neale Accuracy (months) |
142 |
87.13
(15.49) |
102.21
(17.67) |
15.08
(9.46) |
58.52 |
0.001 |
0.97 |
|
Neale Comprehension
(months) |
142 |
94.44
(17.69) |
107.94
(20.15) |
13.49
(13.21) |
35.95 |
0.001 |
0.76 |
|
WARP
(wcpm) |
97 |
39.73
(29.69) |
77.71
(36.44) |
37.98
(16.46) |
63.31
|
0.001 |
1.27 |
|
SA Spelling
(months) |
85 |
88.92
(12.00) |
104.60
(11.38) |
15.68
(7.23) |
76.47 |
0.001 |
1.30 |
|
Burt
(months) |
66 |
85.27
(14.08) |
100.00
(19.43) |
14.73
(9.10) |
24.87 |
0.001 |
1.05 |
Powerful effect sizes were evident on all literacy variables, but particularly for gains made in reading accuracy, reading fluency (the WARP), single word recognition (the Burt) and in spelling (very nearly 1.0 or higher). The effect for reading comprehension was lower but still appreciable.
In terms of reading accuracy, 87% of students accessing the MULTILIT program over the three years made gains of at least 6 months or more and 61% made gains of 12 months or more, in approximately 20 weeks of instruction. For reading comprehension, 71% made gains of at least six months and 51% made gains of 12 months or more. Curriculum-based assessment data on book level were also collected which are detailed in the Report.
We may estimate the typical rate of progress this sample would be likely to have made, if the students had remained in regular school, by reference to rate of progress of low-progress readers prior to entering the MULTILIT Program in February 1997 as Year 5 and 6 students, as previously discussed. This sample of 23 students had typically been gaining, on average, about one month for every two months in school over their school careers thus far and had made little or no progress over the previous term in reading accuracy and only small progress in reading comprehension. Consequently, we may estimate that the present total sample would be unlikely to have progressed by more than about three months, at most, over the two terms had they not undertaken MULTILIT programs, compared with the mean gains they did achieve of 15 months in reading accuracy on the Neale and the Burt and 13.5 months in Neale reading comprehension. This is about five times the rate of progress they probably would have made and about three times the average rate of students progressing normally in regular schools. To this extent, our goal was achieved of greatly accelerating the rate of progress of these low-progress readers beyond the normal or average rate for primary students, thereby allowing the opportunity for Òmaking up lost time in literacyÓ.
Differential gain across gender for the total MULTILIT sample was also analysed, notwithstanding the fact that the sample size for girls (35) was far lower than that for boys (107), the results of which are detailed in the Report. In brief, no gender differences in gain were apparent; boys and girls benefited equally from the Program. Similarly, an analysis of referral information, detailed in the Report, showed that MULTILIT appears to be effective as an intensive intervention for low-progress readers regardless of the apparent reasons for their disability or learning difficulty.
[Top of Point 7]
But what happens after students leave the MULTILIT program?
There is no point in investing significant resources if improved performance is either short-lived or dependent on the context in which the program is delivered. Consequently, we followed up as many students as possible from the total sample, 6 months after leaving a two term program, and then a smaller subset after 12 months.
We were able to follow up and assess 57 students from the total sample of 142 MULTILIT students on the Neale Analysis (Revised) (only), 6 months after leaving their two term MULTILIT programs and returning to their regular schools. This reduced sample of 57 students comprised 44 boys (77%) and 13 girls (23%) and had a mean chronological age of very nearly 11 years. Performance on the Neale Analysis showed that these students, at pre-test on program entry, had been about three years behind their chronological age in terms of reading accuracy and about two and a half years behind in reading comprehension, on average. Highly significant mean gains had been made by this group of students in both accuracy (15 months) and comprehension (about 16 months) during the program (p<0.001). These results suggest that, to this extent, the students comprising the sub-sample followed up at 6 months were typical of the total sample.
After being back in their regular schools for about six months, these former MULTILIT students, on average, clearly maintained their program gains in reading accuracy and in reading comprehension, reading age scores improving by a further 2.5 months and 1.0 month respectively. These gains were not, however, statistically significant suggesting no real further improvement in reading skill. The gains had clearly not washed out but nor did the students continue to gain appreciably when they left the program, on average. In this case, however, the devil is in the detail and, in particular, the high variability in follow up gains evidenced by the very high standard deviations for both accuracy (9.57) and comprehension (15.39). For accuracy, the top quartile had made additional gains of over 8 months whereas the bottom quartile had lost at least 4 months at 6 month follow-up. For comprehension, the situation is even more extreme whereby the top quartile (25%) had made additional gains of over 9 months whereas the bottom quartile had lost up to 8 months.
It appears as if there are three distinct groups of students, for accuracy and comprehension separately: those who do not hold their gains, those who essentially maintain their gains and those who continue to progress after leaving the program. The results for gains from pre-test to 6 month follow-up reflect the overall efficacy of the intervention six months after leaving the program. The overall gains from pre-test to follow-up were 17.54 months for reading accuracy and 16.81 months for reading comprehension. The top quartile for accuracy gained at least two years and for reading comprehension at least two years three months; the middle 50% gained 11 to 23 months and 6 to 27 months respectively. Thus very nearly 75% made and held gains of at least a year for accuracy and 75% made and held gains of at least six months for comprehension.
A sub-sample of 37 students from the total sample of 142 MULTILIT students were re-assessed on the Neale Analysis (only), 12 months following their discontinuation from the two term program. The gains made by this sub-sample of students on both accuracy (14 months) and comprehension (about 17.5 months) during the program were similar to the gains made by the total sample and to the larger sub-sample (n=57) followed up at 6 months, as reported above.
Twelve months after discontinuation from the program, this group of students, on average, clearly maintained their original program gains in reading accuracy and in reading comprehension. The gains over the intervening period, however, were minimal, reading age having improved in 12 months by only a further 4.5 months for accuracy and not at all for reading comprehension. The gains clearly did not wash out, on average, over the period but nor did the students continue to gain appreciably after leaving the program.
[Top of Point 7]
Results for the 1997 MULTILIT Whole Year Program
Results were available for a sample of students who attended MULTILIT for the whole year in 1997. These students had also been assessed prior to entry into the Program. It proved possible to re-assess 13 of the original 19 students at periods of both 6 and 12 months following their discontinuation from the Program. These 13 students were assessed on the Neale on six occasions and on the WARP on the last five of these at roughly 5-6 month intervals as follows:
- In late September, 1996 prior to entry into the MULTILIT Program;
- In early February, 1997 on entry;
- In June 1997 after two terms in MULTILIT;
- In late November, 1997 after four terms in MULTILIT;
- In May/June, 1998 at six month follow up; and
- In November/December 1998 at 12 month follow up.
This provided a unique opportunity to track the progress of a group of students for over two years prior to, during and following exposure to the MULTILIT Program. The mean chronological age of this group was just over ten and a half years at program entry, when they were, on average, about three and a half years behind chronological age in reading accuracy and over two and a half years behind in reading comprehension. Average performance for this group of 13 students is shown in graphical form on the following page so as to illustrate progress over the two years.
For reading accuracy and comprehension, there was little or no change in performance in the period prior to program entry, as we have come to expect for older low-progress readers. On all three measures, dramatic growth was evident during both the first two terms and the second two terms of the MULTILIT intervention. Over the six month period back in regular schools, following discontinuation from the program, however, there was little evidence for further gain but the existing gains, made in MULTILIT, were clearly maintained. This was also true for accuracy in the following six months up to 12 month follow up but for both reading comprehension and reading fluency (WARP) further appreciable gains were apparent.Mean scores for Neale Accuracy and Comprehension (months) and the WARP (wcpm) for 13 MULTILIT students prior to, during, and following a four term MULTILIT Program
[Top of Point 7]
Consumer feedback on MULTILIT
In order to ascertain parents’ (and caregivers’) responses to their child’s participation in the MULTILIT programs at MUSEC, we asked them to complete a questionnaire when they brought their child back to MUSEC for follow-up testing at 6 month (and in some cases 12 month) follow-up. Data were available for students participating in the various MULTILIT programs in 1997 and 1998. Ninety four percent of parents surveyed considered that the program had benefited their child and over half believed that their child’s skills were now adequate to access their regular classroom curriculum. Parents did, however, comment that their child had experienced certain difficulties as a result of participating in the program. These issues were subsequently addressed and the program modified accordingly. Reflections on MULTILIT from the students’ perspective are reflected in the Report by way of inclusion of student speeches at graduations over the period 1996-8. In the words of one of our students from 1998, after two terms in MULTILIT:
One of the main things that I have found is that I am not as dumb as I thought I was. I now know that I can do great work and achieve many things.
[Top] [Top of Point 7]
8. Evaluations of Schoolwise Programs and follow-up studies
Initial disaffection and subsequent alienation from high school may lead some students into increased truancy and a potential for a generic decline into petty crime/delinquency, substance abuse and 'streetlife'. For many such students, problems begin early as a result of initial academic failure in learning basic literacy skills and are then exacerbated by the increasing demands made by a largely text-based curriculum. Schooling can become an increasingly aversive experience for many such 'marginalised' students.
The 'Schoolwise' Project, developed by Macquarie University Special Education Centre in collaboration with the Exodus Foundation, addressed this problem directly by providing intensive literacy intervention programs for students in Years 6 and 7, who were experiencing such severe literacy problems that they were at risk of becoming seriously disaffected from school.
Consequently, MULTILIT programs were provided by MUSEC for groups of twenty such students each semester at the Exodus Foundation Tutorial Centre in Ashfield, known as the Schoolwise Program. The MULTILIT program offered, designed, developed and delivered by the staff of Macquarie University Special Education Centre (MUSEC), provided systematic intensive literacy instruction. Each Schoolwise Program ran for about twenty weeks (two terms) from 8.45 am until 1:00 pm daily, students returning to their home schools in the afternoons. The Schoolwise Program was typically staffed by one senior teacher and two or three assistant teachers for two groups of ten students. Postgraduate students from MUSEC on practicum placement also typically assisted in the programs. The program was an attenuated version of what is offered in the MULTILIT Program at MUSEC (see Chapter 3), with a two hour ‘home group’ independent session at the commencement of the day; one half hour spelling lesson (taught in groups); group reading, comprehension and language lessons, for about an hour; and serial reading.
[Top]
Results of the Schoolwise Program 1996 to 1998
The results for six successive intakes into Schoolwise between 1996-98 are presented in the Report. Successive groups of Year 6 and 7 students admitted to the two term MULTILIT programs consistently made major gains on the literacy measures employed in the evaluation. For reasons of economy, we present here the pooled results of these intakes only, describing a total sample of 106 students.
Of the total sample, 66 (62%) were Year 6 students and 40 (38%) were Year 7 students; 67 (63%) boys and 39 (37%) girls. The mean chronological age of the group was 11 years and 11 months. Performance on Neale Analysis showed that these students were, on average, just over four years behind their chronological age in terms of reading accuracy and nearly four years behind their chronological age in terms of reading comprehension.
All 106 students whose data are included in this total sample experienced two terms in the program. The means and standard deviations for all measures at pre-test and at post-test are shown in the table below. In under five months of participation in the program, these students made average gains in Neale reading accuracy of 15 months, 11 months in Neale reading comprehension, 15 months in Burt word reading, 35 words read correctly per minute on the WARP and 14 months in spelling. It bears repeating that these were students who had made little or no progress in recent years and who were around four years behind in reading and related skills when they commenced the program. Some were complete non-readers at program commencement.
Means (and standard deviations) of the Relevant Literacy Variables for the Total Sample and the Resultant Gains
|
Literacy Variable |
N |
Pre-test |
Post-test |
Gain |
F |
p |
ES |
|
Neale Accuracy
(months) |
106 |
93.60 (13.03) |
108.71 (16.16) |
15.10
(9.01) |
56.13 |
0.001 |
1.16 |
|
Neale
Comprehension
(months) |
106 |
96.85 (14.81) |
107.58 (16.52) |
10.73
(11.89)
|
24.77 |
0.001 |
0.72 |
|
Burt (months) |
56 |
99.11
(15.81) |
114.18 (20.59) |
15.07
(6.99) |
18.88 |
0.001 |
0.95 |
|
WARP (wcpm) |
74 |
70.61 (30.92) |
105.78 (33.11) |
35.18
(14.20) |
44.62 |
0.001 |
1.14 |
|
SA Spelling
(months) |
73 |
98.15 (14.51) |
112.03 (11.40) |
13.88
(8.57) |
41.28 |
0.001 |
0.96 |
All of these gains were highly significant statistically (p<0.001) and the effect sizes for all five dependent variables may be classified as large in that they are all over 0.8, except for reading comprehension (0.72). The average effect size was 0.99 which means that as a group they gained almost a whole standard deviation between testings. These ‘low-progress’ students made as large a gain in five months as regular students typically make in one year.
For Neale reading accuracy, 88% made gains of six months or more and 65% of students made gains of a year or more. Similarly, for Neale reading comprehension, gains of six months or more were made by 73% of students and 47% of students made gains of a year or more. On average, they could now read nearly 50% more words correctly per minute on the WARP than they could when they first entered the program.
These students would typically have gained only about five months in reading age each year. We might then have expected the total sample to have made perhaps about two and a half months progress during the course of their two term MULTILIT program at Schoolwise compared with the 15 months they actually made, on average, for both Neale accuracy and on the Burt; six times the rate of progress we might have predicted.
Comparisons of Year 6 with Year 7 students showed that there was no evidence to suggest differential gains by one group over the other. MULTILIT appeared to benefit both Year 6 and Year 7 groups in the Schoolwise Program evenly. Likewise, in terms of sex differences there was no evidence to suggest differential gains by one group over the other. MULTILIT appeared to benefit both boys and girls equally.
[Top of Point 8]
But what happens after students leave the Schoolwise Program?
Follow-up assessments were completed (where possible) on students who had attended the standard two term MULTILIT program after six months and twelve months after students had left Schoolwise.
We were able to collect follow-up data on 50 students who had completed the standard two term program at both six and twelve months following departure from the program. At pre-test on program entry these students were, on average, over four years behind their chronological age in terms of both reading accuracy and reading comprehension. Highly significant gains had been made by this sub-group on both accuracy (about 16 months) and comprehension (about 10 months) during the program. These results suggest that, to this extent, the sub-sample followed up at six and twelve months were typical of the total sample.
After leaving Schoolwise and returning to their regular schools for about six months, these students, on average, made minimal further improvement (about a month on average) but clearly maintained their gains in both reading accuracy and in reading comprehension. The substantial gains made in the program clearly had not washed out. High variability in follow up gains was apparent; the standard deviations being high for both accuracy and comprehension. The top quartile (25%) for accuracy gain had made additional gains of over 8 months whereas the bottom quartile had lost at least 3 months; for comprehension the top quartile had made additional gains of over 6 months whereas the bottom quartile had lost at least 5 months.
Twelve months after discontinuation from the program, this group of students, on average, had still maintained their original program gains in both reading accuracy and comprehension with average further gains again being around one month only. The gains clearly had not washed out over the 12 month period but nor did the students continue to gain much after leaving the program. The top quartile had made additional gains of at least 5 months for accuracy and 9 months for comprehension whereas the bottom quartile had lost at least 3 months in accuracy and 5 months in comprehension.
The overall gains from pre-test to follow-up at 6 months for the Schoolwise students were 16.76 months for accuracy and 14.68 for comprehension. Eighty (80%) per cent had made and held gains of at least a year for accuracy and 55% had made and held gains of at least a year for comprehension (nearly 70% at least six months). The parallel gains overall at 12 month follow-up were 18.16 months for accuracy and 15.74 for comprehension. Seventy-five percent (75%) had made and held gains of at least a year for accuracy and 64% had made and held gains of at least a year (75% at least six months) for comprehension.
[Top of Point 8]
Consumer Feedback on the Schoolwise Program
Teachers at students’ regular schools were supplied with questionnaires at the time of follow-up, either six or twelve months after students had completed the Schoolwise Program. Each teacher was asked to complete one questionnaire referring to all of the Schoolwise students who were enrolled in his/her class. Ninety six percent (96%) of teachers considered that the program had been of benefit to students. The great majority of teachers believed that participation in the Schoolwise program had not led to the stigmatisation of students (94%). Further, two thirds (66%) also believed that it was advantageous for students for the program to be situated separately from the school. In terms of students’ views, three focus groups were completed with groups of students. In summary, students clearly enjoyed the clear instructions provided by teachers, the individual attention, praise, rewards and encouragement. As one of the students put it, in the context of coping with regular school work following attendance at Schoolwise: 'It makes it very easier.'
[Top] [Top of Point 8]
9. Evaluations of other MULTILIT programs
In addition to the evaluations completed within MUSEC Special School and the Exodus Foundation Tutorial Centre, we have been able to trial MULTILIT in other locations and forms.
[Top]
The MULTILIT Program at an independent primary school
Following extensive negotiations with an independent primary school (‘Valleyville’) in the Sydney area, an official MULTILIT outreach facility was established which allowed an evaluation of a small-scale implementation of MULTILIT for eight low-progress readers which operated in the school in Terms 3 and 4, 1998.
A MULTILIT teacher from MUSEC was seconded to deliver the program on-site for a total of 18 weeks. Instruction took place between 9.00 am - 12.30 pm; just over three hours of MULTILIT instruction being provided per day. Each student had a one-to-one individualised session with the teacher, a MULTILIT Reinforced Reading session with a trained tutor, a group reading lesson, a group spelling lesson, a group comprehension lesson, and an independent contract to complete daily.
The mean chronological age of the eight students from Years 5 and 6 comprising this sample at program entry was 11 years and 10 months. At the commencement of the program, reading accuracy for the group averaged 102 months and reading comprehension 106 months. These students were nearly three and a half years behind their chronological age for reading accuracy and three years behind for reading comprehension, on average.
Highly significant gains (p<0.01) and very large effect sizes (1.33 to 1.90) were in evidence for all literacy measures. After 18 weeks of instruction, the mean reading age had increased by 15 months for reading accuracy and by 16 months for reading comprehension. All eight students made gains of 10 months or more, with three students making gains of 12 months or more, in reading accuracy. Similarly, for reading comprehension, seven of the eight students made comprehension gains of 6 months or more, while five made gains in excess of 12 months. For word recognition, students averaged a gain of 14 months while reading fluency increased by a mean of 54% in terms of wor