Appendix 1: Summary of Mr Glenn Cupit's Report to the Faculty of Aboriginal and Islander Studies on the Performance of Aboriginal and Torres Strait Islander Students at the University of South Australia by Samantha Moore, NAU

[next chapter] [previous chapter] [contents]

Preamble
Barriers to Ascertaining Level of Performance
Relevance of Statistics
Correction of Attrition Formula
Reliability of Attrition Figures
Evidence from Pass Rates
Indigenous Performance in the University of South Australia
Intra-Faculty Comparisons
Recommendations

Preamble

Mr Glenn Cupit, Head, De Lissa Institute was commissioned to report to the Faculty of Aboriginal and Islander Studies on the performance of Aboriginal and Torres Strait Islander students at the University of South Australia. His report was an assessment based on the existing statistical record of Indigenous attrition at the University of South Australia to determine what conclusions could be drawn from the statistical data. This case study was to be a basis for an enlarged project comparing Australian wide attrition rates of Indigenous university students.

A profile of age and educational attainment of Indigenous students upon entry to the university was compiled for investigation. The project reviewed Indigenous attrition rates in an endeavour to discover potential patterns and common factors that contributed to attrition based on the statistical records. Some of the factors to be considered were:

• academic preparedness for tertiary studies;
  
• family or cultural pressures;
  
• inappropriate instruction;
  
• courses too demanding or not demanding enough;
  
• adequacy of support provisions; and
  
• other factors such as gender, age, prior education, distance from permanent home, health, family pressures, isolation, and any other issues that appeared during the course of the case study.

The project aimed to identify, in specific terms, such variables as: when students withdraw, which courses and subjects they withdraw from, and what role non-academic concerns play (e.g. accommodation, finances etc) as opposed to purely academic concerns. The issues of where the students go after leaving the institution and whether they return to their studies in the future were also to be examined. Finally, the results were expected to provide a basis for recommendations regarding future policy in the areas of pre-testing potential applicants, selection criteria, pre-enrolment counselling, enrolling successful applicants, and supporting students once on course.

The first step of the research project was a case study of the attrition statistics available for Indigenous students at the University of South Australia. Through analysis of the available statistical records comparisons of attrition rates between and within faculty courses and non-faculty courses were to be made.

The scope of the project was quite broad and required an accurate statistical foundation to begin the case study with. The first barrier encountered was manifested in inaccurate statistical reporting.

Barriers to Ascertaining Level of Performance

The first barrier was a statistical distortion of the actual performance of Indigenous students in courses within the University of South Australia. The Planning Unit of the university provided statistics on Indigenous student performance. The performance was expressed in two ways: as attrition rates and pass rates. Even a cursory examination of the statistics revealed anomalies in the figures; for example retention rates of 150 per cent and 200 per cent. Members of the reference group providing advice to Mr Cupit included faculty staff members, and they also reported marked discrepancies between their own records and those provided by the Planning Unit.

Two sets of comparisons had been envisaged in the wider project; inter-university and intra-university. Both required the use of accurate statistics. The reliability and validity of the statistics provided was seriously questioned. Two sorts of error infected the original data. The first error caused a significant number of students in the statistics labelled as 'completed' in 1991 (and therefore excluded from consideration in attrition rates) to show on 1992 class lists. This program error was corrected and the corrected data provided the basis of the report. Mr Cupit recommended a future audit of the accuracy of the statistical programs in order to minimise these types of errors.

The second error was identified by the Planning Unit as inaccuracy of student responses. The apparently high attrition rate of Indigenous students was explained as the university's loss of interstate students returning home as courses for Indigenous students opened elsewhere. Students who had left the University of South Australia to attend other universities near their permanent home were included in the attrition rate. The number of students who returned to their permanent address to study is not reflected in the statistics because the home address may be incorrectly listed as an Adelaide address. This error surfaced when students were asked for their home address, and the interstate or country students reported their Adelaide home address. Mr Cupit found that the wording on university data collection instruments should be considered in order to increase the validity of information provided.

An additional barrier lay in the discrepancy of the definition of the term 'attrition'. The operational definition of 'attrition' used by the Planning Unit was not consistent with the term's meaning in polemic discourse. In common discourse an 'attrition rate' is seen as an assessment of the global performance of a group of students. The assumption is that 'attrition' is a measure of the proportion of students who fail to complete a course of study. A high success rate should correspond to a low attrition rate.

The Planning Unit's measure of attrition was a simple ratio that did not reflect the attrition rate accurately. It compared the number of students who enrolled in year X with the number who did not re-enrol in year X+1. This ratio specifically excluded students who successfully completed their course of study in year X. This measurement of attrition was misleading in that, students who successfully completed a one year course were not counted as successful, but instead added to the attrition rate.

The apparent attrition rate only measured the students who re-enrolled in the same university or left the University of South Australia. Therefore, students who did not actually leave the university (they may have taken approved leave, changed schools, or even completed their course) were included in the attrition rate. These students did not re-enrol in the following year, and were added to the attrition rate.

The result of measuring the attrition rate in this manner showed anomalies in the Planning Unit data. The original data indicated that the better the prior education level of Indigenous students the worse their attrition rate. This finding is not an accurate reflection of actual student performance.

It can reasonably be argued that students studying part time or through distance education, are female, mature age, or Indigenous are more likely than others to require legitimate leave. All of these categories are relevant to intra-university comparison of performance because students in these categories were erroneously included in the attrition rate.

Relevance of Statistics

The Planning Unit measure of attrition rates contained a systematic bias against shorter courses. Any faculty that offered more short courses than the average will have higher attrition rates then faculties that offer degree length awards as a minimum. In addition, courses with small enrolments, as is the case when considering Indigenous students in FAIS awards, were subjected to substantial statistical percentage changes when a few or even one student leaves the course.

Correction of Attrition Formula

The operational definition of attrition should be based on the following formula to reflect the rates more accurately:

Attrition = 100 - (C+R+L)%
                          (     N    )

Where:

C = number successfully completed

R = number continuing study (whether in same or different award)

L = number on approved leave

N = total enrolment in unit of study.

This formula would be appropriate for intra-university comparisons in the short term. In the longer term and for any inter-university comparisons a valid common measure would be required. A standard program such as COHORT, described in a DEETYA occasional paper (June 1993), would be appropriate for university comparisons.

Reliability of Attrition Figures

While the amended figures for attrition provided a more accurate picture of the performance of Indigenous students, the fact that figures were only available for two years was disappointing. This limitation of the data calls into question the representativeness of the statistics. The figures provided were small, and far below a level where significance tests would be meaningful. If even one year in the figures was unreliable, it would have significantly skewed the general trend of attrition rates as a whole.

Evidence from Pass Rates

Some aspects of the statistics provided lead to inaccurate representation of the performance of Indigenous students. The Planning Unit ignored subjects in which Indigenous students performed at a higher success rate than non-Indigenous students. Unfortunately, it is possible to assume from the data provided that Indigenous students never performed better in subjects than non-Indigenous students. To rectify this situation, a print-out of all subjects listing the performance of Indigenous and non-Indigenous students at a specified pass rate along with a comparative print out of subjects with Indigenous enrolment should be used.

The figures also unfairly represented the comparison between Indigenous and non-Indigenous students by attributing a 100 per cent pass rate to non-Indigenous students in subjects where enrolment by those students was zero. If we were to follow this line of reasoning, then a 100 per cent pass rate should have been attributed to Indigenous students in all subjects where they were not enrolled.

Indigenous Performance in the University of South Australia

Some tentative conclusions can be expressed in terms of apparent performance according to the statistics provided by the Planning Unit. The pass rates were examined in terms of above and below 50 per cent and 70 per cent.

When comparisons were made between the Faculty of Aboriginal and Islander Studies (FAIS) and other faculties the relative number of cases of Indigenous students above and below a 50 per cent pass rate did not differ, but when non-Indigenous students were considered, Indigenous students were less successful in FAIS subjects then non-Indigenous students. Indigenous students in general were less successful then non-Indigenous students, and Indigenous students were less successful in both FAIS and non-FAIS subjects.

The comparison of FAIS and other faculties and Indigenous and non-Indigenous students on pass rates above and below 70 per cent revealed that Indigenous students did better in non-FAIS subjects than in FAIS subjects. The statistics also revealed that both Indigenous and non-Indigenous students had less success in FAIS subjects. Finally Indigenous students were also reported as less successful in general, in both FAIS and non-FAIS subjects.

These results suggest that the issue of performance of Indigenous students both within and outside the faculty needs to be addressed and improved. The most urgent issue though, is the suggestion in the figures that Indigenous students had better pass rates outside the faculty. This is consistent with the finding that the attrition rate for Indigenous students was higher than for non-Indigenous students within the faculty.

Intra-Faculty Comparisons

The comparison of pass rates between students of the two schools, Aboriginal Studies and Teacher Education Centre (ASTEC) and School of Aboriginal and Islander Administration (SAIA) revealed no difference between ASTEC and SAIA at either 50 per cent or 70 per cent pass rates for Indigenous and non-Indigenous students. For all comparisons in the faculty, Indigenous students were less successful than non-Indigenous students independent of site or level of pass rates considered.

For future comparisons it would be useful to identify subjects with sufficient enrolments for the statistics to be meaningful (at least ten students) and identify general superior and inferior performance in courses among Indigenous and non-Indigenous students. Where necessary enrolments over a number of years may need to be aggregated to provide sufficient numbers for statistical analysis.

Recommendations

Based on the case study of the statistical assessment of attrition rates at the University of South Australia, the following eleven recommendations were made by Mr Cupit for any future studies on Indigenous student performance.

Recommendation 1: Improved Statistical Package

• That the faculty request the university to press DEETYA to adopt a statistical package such as COHORT for inter-university comparisons and to refrain from drawing conclusions about relative performance until a standard, valid and reliable instrument is in place.

Recommendation 2: New Definition of Attrition

• That until such a measure is adopted, intra-university comparisons be based on a definition of attrition that does not exclude completers, does not count approved leave as attrition, and includes students with status in the cohort with which they share course work.

Recommendation 3: Pass Rates Data

• That the faculty request the Planning Unit to provide pass rates for Indigenous and non-Indigenous students in all subjects in which a minimum of ten Indigenous students are enrolled but with the following stipulations:
  

a) students on approved leave are excluded;

b) separate rates are provided inclusive and exclusive of 'withdrawal' grades as failures; and

c) that no pass rate be provided for non-Indigenous students where there is not a relevant enrolment.

Recommendation 4: Experiential Data

• That the working group continue to meet to generate, on the basis of experience with the students, a set of proposed explanations for the difficulties in Indigenous performance at the university as a basis for subsequent detailed review.

Recommendation 5: Cohort Studies

• That, as soon as valid and reliable measures are available, cohort studies should be undertaken which track each student's career from initial contact to activity in the year subsequent to exit from the award. This should include source of entry, prior qualifications and experience, academic performance and relations with the institution, reasons for leaving, life consequences of university involvement, etc. This cohort study should use both quantitative and qualitative techniques to identify career 'choice points' and both objective and phenomenal variables relevant to career directions.

Recommendation 6: Extended Case Studies

• That a subset of students within such cohort studies should be selected for involvement in extended case studies to document relevant attitudinal and motivational variables, difficulties encountered and support received, career decision points and their perceived meaning, etc.

Recommendation 7: Detailed Study of Practice

• That, as soon as valid and reliable measures are available, the faculty should identify examples of superior and inferior performance at a subject and award level and commission a comparative study of practice in those areas at a detailed level including such issues as staff demographics, qualification and attitudes, student demographics and attitudes, pedagogy, assessment and infrastructure support. This will be potentially threatening to some staff, and possibly students, and should be conducted with sensitivity by a research group including persons experienced in Indigenous education and coordinated by an experienced researcher identified as being devoid of vested interest in the outcome.

Recommendation 8: Detailed Administrative Review

• That the faculty request the university to undertake a detailed administrative review of:
  

a) University policies and procedures which have the potential to have an impact on the career paths of Indigenous students (note this is wider than specific equity and access policies);

b) Faculty, school, department, award, and administrative group (e.g. Library) policies and procedures which also have that potential;

c) implementation, non-implementation or variation of policies and procedures where Indigenous students are involved; and

d) differential treatment of Indigenous and non-Indigenous students at the level of policy, procedure or practice.