Department of Employment, Education, Training and Youth Affairs
Influences on
the Course Experience
Questionnaire Scales
Michael Long
Trevor Johnson
Australian Council for Educational Research
January 1997
96/20
Evaluations and Investigations
Program
Higher Education Division
©Commonwealth of Australia 1996
ISBN 0 642 23612 7
This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the Australian Government Publishing Service. Requests and inquiries concerning reproduction and rights should be addressed to the manager, Commonwealth Information Services, Australian Government Publishing Service, GPO Box 48, Canberra ACT 2601.
This report is funded under the Evaluations and Investigations Program of the Department of Employment, Education, Training and Youth Affairs.
The views expressed in this report do not necessarily reflect the views of the Department of Employment, Education, Training and Youth Affairs.
[next chapter] [top]
2. The Course Experience Questionnaire
3. Characteristics of Graduates
4. Comparisons between Universities
[next chapter] [contents] [top]
The introduction of the Course Experience Questionnaire has contributed to a focus on the quality of education offered by the Australian university sector and by individual universities. Such attention can only be to the good of higher education. Research and consultancy are activities which are perhaps more easily demonstrated. If the quality of teaching is not measured, it will be given less than its due regard.
Measurement of the quality of teaching is not easy. There will always be doubts about the validity of comparisons between courses consisting of students of different backgrounds, abilities and experiences. There will always be doubts about the validity of the instruments used and the extent to which they can be meaningfully interpreted by students and graduates. The difficulties encountered with measurement of course quality are common to the study of many social processes and are not insurmountable.
This volume provides an empirical investigation of aspects of the validity of measures derived from the Course Experience Questionnaire. We would like to thank officers of the Department of Employment, Education, Training and Youth Affairs (DEETYA), particularly within the Evaluations and Investigations Program (EIP) Section and within the Higher Education Division more generally, for their encouragement to pursue these issues.
Acknowledgement must be made also to Roger Bartley, Director of the Graduate Careers Council of Australia (GCCA), under whose auspices the information is collected, Bruce Guthrie, the GCCA survey manager, and Michael Koder, the chair of the GCCA Survey Management Group. Their efforts were essential for the introduction and survival of the Course Experience Questionnaire. Paul Ramsden has devoted many years of his professional life to developing and investigating measures related to the quality of learning. The Course Experience Questionnaire is to a considerable extent the product of this work. The work of survey managers at every Australian university is essential to the survey. In the last instance, however, we are always indebted to the people who take time to respond to the questionnaires—in this case, more than 60,000 recent graduates of Australian universities.
[next chapter] [previous chapter] [contents]
The Course Experience Questionnaire was added to the Graduate Destination Survey—an annual survey of recent graduates from nearly all Australian universities—in response to Government policies which demanded greater public accountability of Australian universities and which therefore needed indicators of university performance. The questionnaire focuses on the quality of teaching, nature of assessment, level of workload, level of skill formation and graduates’ overall satisfaction with their courses.
The strength of the instrument for system and institutional management lies with the potential for comparisons between universities. One of the major features of the indicators derived from the Course Experience Questionnaire is that they vary substantially between fields of study. This suggests that comparisons of courses within universities may be misleading.
Comparisons between courses conducted by different universities, however, are not necessarily any less problematic. The graduates of the various universities may differ substantially in terms of personal, family and educational background. If these differences exert a substantial influence on graduates’ perceptions and hence on the measures derived from the Course Experience Questionnaire, then comparisons between universities may be equally as misleading as comparisons within universities.
This report examined the effect of a number of different characteristics of graduates on their responses to the Course Experience Questionnaire. The characteristics were age, gender, English language background, previous educational qualifications, full- or part-time attendance, and participation in further study and employment after course completion. The major finding was that any relationships with Course Experience Questionnaire measures were generally quite small and were rarely consistent across the ten fields of study for which analyses were conducted.
The effects of these characteristics of graduates on between-university differences were small. Generalising somewhat over fields of study and measures, the percentage of variation explained by between-university differences was relatively unchanged by consideration of graduate characteristics and was large relative to the variation explained by graduate characteristics. Graduate characteristics had little effect on the mean scores for individual universities in the overwhelming majority of cases.
Two further possible sources of bias were examined. Arguments could be advanced for either positive or negative effects of response rates and the academic ability of students undertaking a course on Course Experience Questionnaire measures for a particular course. Measures of response rates and academic ability (based on Tertiary Entrance Scores) were constructed and analyses of relationships with Course Experience Questionnaire measures undertaken.
These relationships were examined at the level of courses. The existence of a relationship at this level is a necessary, but not sufficient, condition for there to be some effect on between-university means. In both instances, graduate-level information is required to establish any definite effect.
The major finding for both these variables was that there was substantial variation in the magnitude and direction of relationships with the Course Experience Questionnaire scale scores for the various fields of study. Such variation is not explicable in terms of reasons that may be advanced for expecting relationships between mean scale scores and either of these variables.
Nevertheless, the magnitudes of the relationships for some fields of study for selected measures were sufficiently large to indicate the need for further research based on information for individual graduates rather than on the averages for entire courses used in this report.
The Graduate Destination Survey collects information on a number of important background variables. There is, however, the opportunity for universities to do more with their own results. Student identification numbers are included on the main questionnaire in order to allow the removal of respondents from follow-up mailouts. These could provide the basis for merging student administration information onto responses to the Graduate Destination Survey and the Course Experience Questionnaire. This would permit further analyses of potential sources of response bias and provide a basis for identifying correlates of non-response. Given the undertakings of confidentiality on the questionnaire, it is unlikely that there would be concerns about privacy.