2. The Course Experience Questionnaire

[next chapter] [previous chapter] [contents]

The Policy Background
The Course Experience Questionnaire Scales

The Good Teaching Scale (GTS)
The Clear Goals and Standards Scale (CGSS)
The Appropriate Assessment Scale (AAS)
The Appropriate Workload Scale (AWS)
The Generic Skills Scale (GSS)
Comparison of Scales

Fields of Study
Summary

This chapter provides background information about the Graduate Destination Survey (GDS) and about the development, purpose and content of the Course Experience Questionnaire (CEQ). The CEQ records the opinions of university graduates about aspects of the courses they have recently completed. It consists of 25 questions which are included with the Graduate Careers Council of Australia’s (GCCA’s) annual GDS.

The GDS is a mail survey of recent graduates of nearly all Australian universities. The universities invite recent graduates to participate in the survey a few months after the completion of their course. The questionnaire takes approximately 20 minutes to complete and focuses on the current activities of graduates: whether they are employed, studying, or unemployed. It seeks information about occupation, industry, income and further study. As its name implies, it is concerned with the destinations of graduates. The GCCA acts as survey manager. It provides the questionnaire, coding instructions, and a recommended methodology for conducting the survey. Each university is responsible for surveying its own graduates. Given that the response rate to the 1994 GDS was 62.2 per cent and 55.0 per cent for the CEQ, the survey provides almost complete coverage of recent graduates of Australian universities.

The Policy Background

The Green and White Papers on Higher Education (Dawkins 1987 and 1988) provided much of the impetus for the implementation of the CEQ. The publication of Higher Education: A Policy Discussion Paper (Dawkins 1987) expressed the view that there was a need for greater public accountability in the higher education sector. The Green Paper proposed that funding for universities be linked to institutional performance. This proposal was carried forward into the White Paper, but substantially modified later. In the policy statement, student satisfaction was included in a list of output-based measures of institutional performance along with completion rates, relative staffing levels, and research and publication rates (Dawkins 1988: 85–86). The White Paper also pointed to the need for the development of appropriate indicators of university performance, and endorsed the initiatives of the Australian Vice-Chancellors’ Committee (AVCC), and the Australian Committee of Directors and Principals in Advanced Education (ACDP), in attempting to define a workable and comprehensive set of indicators.

The Performance Indicators Research Group (PIRG) was commissioned by the Commonwealth Government to examine a number of quantitative indicators that could be used to evaluate relative performance in higher education at both system and institutional levels. The work of the PIRG was based on the definitions and classification of potential indicators developed by a joint working party of the AVCC and the ACDP (AVCC/ACDP 1988). The PIRG conducted an extensive trial of the CEQ in several Australian universities (Linke 1991).

The questionnaire had previously been used to good effect in the Discipline Review of Accounting (Mathews 1990), and was based on the work of Paul Ramsden and his colleagues (Ramsden and Entwistle 1981; Ramsden, Martin and Bowden 1989; Ramsden 1991). On the basis of their trial of the CEQ, the PIRG recommended that:

an appropriate indicator of perceived teaching quality ... be included as part of any national system of performance indicators; ... as a first step in the development of an effective indicator of teaching quality the Commonwealth seek to have an appropriate instrument, structured initially along the lines of the Course Experience Survey, incorporated in, or administered in conjunction with, the GCCA graduate survey.

(Linke 1991: xviii–xix).

These recommendations were given effect when the CEQ was included in the 1993 GDS. In its final form, however, the questionnaire consisted of a subset of the items from the original trial version, and a further six items relating to work skills which had not previously been tested. The same set of items was used in the 1994 survey and, with one exception, in the 1995 survey. The 25 items are shown in Table 1 together with the pattern of responses and summary statistics for pass degree and honours graduates from the 1994 survey of graduates who completed their courses in 1993.

A cursory review of the published research in this area will reveal that the results reported in Table 1 form part of the comprehensive theoretical and empirical framework describing student evaluations of teachers and their teaching practices that has developed over the last 25 years. For example, Smith (1980) discussed student assessment of university teachers. Ramsden and Entwistle (1981) examined the effects of academic departments on students’ approaches to studying. Marsh and Overall (1981) reported on the relative influences of course level, course type, and instructor on students’ evaluations of college teaching in the United States. Angus (1984) investigated student attitudes to teachers and teaching. Marsh (1987) reviewed work in the area, discussed methodological issues, and proposed directions for future research. Ramsden, Martin and Bowden (1989) considered the relation between social environment and the approaches to learning of senior secondary school students. In a longitudinal study of the stability of student ratings of the same teachers over a 13 year period, Marsh and Hocevar (1991) collected data from a total of 6,024 graduate and undergraduate classes taught by 195 teachers representing 31 faculties. They found little change over time with respect to nine content specific dimensions, an overall course rating, and an overall teacher rating. More recently, Marsh and Roche (1994) reported on the use of students’ evaluations of university teaching to improve teaching effectiveness. Thus, it would seem that research in the area of student ratings of teachers and courses is flourishing.

Table 1: Course Experience Questionnaire Items: Percentage Responses, Means and
               Standard Deviations: Degree Graduates, 1994

See Notes to Tables

The basic form of the CEQ was developed for initial graduates, and tested in Australian universities during 1989. Ramsden (1991) described the theory of teaching and learning that underlies the questionnaire. It links perceptions of courses to learning outcomes. Since its inception the content of the instrument has been progressively refined. Items included in the questionnaire have changed, and the wording of some items has been modified, in response to data obtained from its application in different contexts. It may not be important whether graduates agreed that It was always easy to know the standard of work expected (the first item on the questionnaire). While it could be argued that vagueness of standards encouraged students to work to the limits of their own ability, the evidence in the literature reviewed above supports the general view that clarity of goals and standards in courses is associated with better learning outcomes.

Table 2: Characteristics and Constituents of the Course Experience Scales: Degree                Graduates, 1994

See Notes to Tables

The Course Experience Questionnaire Scales

Table 2 shows the grouping of the items in the CEQ, and the name of the scale to which each item contributes. The structure and existence of the scales were confirmed by factor and item analyses. Item 16, The course was overly theoretical and abstract, did not show suitable properties for inclusion in any of the scales and Item 25, Overall, I was satisfied with the quality of this course, was retained separately as a global assessment of the course.

The item scores were recoded from 1, 2, 3, 4 and 5 (from strongly disagree to strongly agree as per the questionnaire) to -100, -50, 0, +50, and +100 respectively. The means and standard deviations recorded in Table 1 are based on this recoding. The procedure has the effect of removing decimal points from results, and making negative scores correspond to disagreement and positive scores correspond to agreement.

The score for a scale was calculated as the mean of the recoded scores of the items which constituted the scale. For example, if a respondent circled ‘5’ for Item 3, circled ‘4’ for Item 7, circled ‘5’ for Item 15, circled ‘4’ for Item 17, circled ‘2’ for Item 18, and circled ‘4’ for Item 20, that individual’s score for the Good Teaching Scale (GTS) would be: (100 + 50 + 100 + 50 + (-50) + 50) / 6 = 50. The coding of items which had a meaning which was opposite to the meaning of the scale, for instance, Item 13, It was often hard to discover what was expected of me in this course, was reversed before being combined with other items. In Table 2 these items are indicated by an asterisk.

The use of scales, rather than individual items, has four benefits. The first is parsimony. Instead of discussing say six items, we can refer to a single measure of the quality of teaching. Second, scales, by focusing on what is common between items, tend to reduce the effect of any idiosyncrasies associated with particular items. Third, the standard deviation of a scale is usually less than that of the items from which it is constructed. The standard deviations of the CEQ scales are around 40 compared with the standard deviations of the constituent items which are approximately 50 or more. Fourth, the extent to which items share a common variation can be measured. Table 1 shows Cronbach’s alpha (standardised) which is a measure of the internal consistency of the items which form the scale. Using single items does not solve the problem of low reliability. It simply means that the reliability of the item is generally unknown. The reliabilities of the CEQ scales are satisfactory.

The Good Teaching Scale (GTS)

The GTS is defined by behaviours such as providing students with feedback on their progress, explaining concepts, making the course interesting, motivating students, and understanding students’ problems. High scores on the GTS are associated with the perception that these practices are present. Lower scores reflect a perception that these practices occur less frequently.

It looks as if changes in teaching (good teaching, greater freedom in learning and an avoidance of overloading) are likely to move students away from surface and towards deep approaches to learning, and also to improve attitudes, thus improving the quality, at least, of what is learned.

(Ramsden and Entwistle 1981: 381)

The Clear Goals and Standards Scale (CGSS)

The definition of clear goals and standards for a course is a part of good teaching. It is, however, conceivable that lecturers could engage in those activities incorporated in the GTS and yet fail to establish clear goals for the course and clear expectations of the standard of work required from students. The items which form the CGSS jointly measure this component of teaching practice.

The Appropriate Assessment Scale (AAS)

The items which comprise the assessment scale focus on the extent to which higher-order thinking was being assessed in the course. An over-emphasis on the memorisation and recall of factual material was not seen as appropriate for university level courses. High scores on this scale indicate that the assessment methods employed are not simply testing memory.

The wording of the AAS items suggests that assessment which does not focus on facts instead concentrates on higher-order processes. The first item, To do well in this course all you really needed was a good memory (coded so that strongly disagree corresponds to a high score) suggests that students required something in addition to good memory to do well in the course. Strong agreement with the item implies that good memory was necessary and sufficient to pass the course—that good memory was exclusive of other learning requirements. The second item in the scale, The staff seemed more interested in testing what I had memorised than what I had understood (coded so that strongly disagree corresponds to high scale scores) establishes the assessment of facts in contrast to the assessment of understanding.

Strong agreement clearly implies that understanding is not being assessed. The third item in this scale is Too many staff asked me questions just about facts (coded so that strongly disagree corresponds to high scale scores). This item does not establish a clear dichotomy between assessment of learning based on facts and memory, and assessment of understanding and higher-order processes. Nevertheless, its wording suggests an over-emphasis on assessment of facts—and the high correlation of this item with the other two items in the scale suggests that it is effectively measuring the same dimension. The wording of these items establishes a spectrum with assessment of learning based on facts and memory at the low scale score end and assessment of understanding and higher-order mental processes at the high scale score end.

This scale is not comprehensive in its measurement of assessment. Other features, such as the relationship of the assessment to the material actually taught, or the frequency and burden of assessment, or of the level of difficulty and consistency of assessment, could have been included under this heading.

The specific nature of this scale reflects several features of the survey. First, it is impossible to measure the universe of assessment with a subset of 25 items. Second, it is important to use measures with a proven track record in prior research. Third, the context within which this instrument developed was concerned with the relationship between the learning environment and the type or style of learning in which students engage. A measure of assessment which focuses on whether higher order thinking is being tested is a natural consequence.

The Appropriate Workload Scale (AWS)

High scores on the Appropriate Workload Scale indicate reasonable workloads. Graduates who disagreed with the proposition that The workload was too heavy and who agreed that I was generally given enough time to understand the things I had to learn will have high scores on this scale. The interpretation of this scale rejects the view that heavy workloads are a reflection of a course demanding high standards from its students and therefore producing exceptional results.

The evidence from research on student learning is that such high standards may be a mirage. Heavy workloads require students to adopt an approach to learning which emphasises skimming across the surface of topics without being able to spend the time to truly engage and understand the material they are meant to be learning. We are not talking about workloads that interfere with a student’s social life, the scale is tapping a level of workload that has an adverse influence on students’ learning. Two of the four items in this scale explicitly make this link.

It is possible that workload can be too low as well as too high. It is true that a course which required little work from its students would score very high on this scale, and that this could hardly be described as a good learning situation. It appears, however, that courses rarely, if ever, reach the too low level of workload.

If they did, then we would be unlikely to observe the positive correlations between the Appropriate Workload Scale and the Good Teaching, Clear Goals and Standards, and Appropriate Assessment Scales that are reported elsewhere (Ainley and Long 1994).

It has been suggested that university departments might attempt to maximise their scores on the Appropriate Workload Scale by making their courses less intellectually demanding (Magin 1990). Workload is not the same as the level of intellectual input required from students. Such a response would almost certainly be counter-productive in any case. The effect on some items in the GTS, those which emphasise student achievement in particular, would no doubt be negative.

The Generic Skills Scale (GSS)

The generic skills scale and the items which comprise it are specific to this questionnaire and this survey. The inclusion of this scale in the survey is an attempt to take into account the extent to which university courses add value to the generic skills which their graduates might be expected to possess. While discipline-specific skills and knowledge are often crucial to prospects for employment and further study, the emphasis on generic skills stems from the belief that knowledge quickly becomes obsolete. The generic skills that may have been developed while acquiring that knowledge endure.

A report from the Higher Education Council (1992) emphasises the importance that should be attached to such generic skills. It cites a number of studies of the skills that employers (and academics) look for in graduates. The lists of desirable skills variously include communication skills, the capacity to learn new skills and procedures, the capacity to make decisions and solve problems, the ability to apply knowledge to the workplace, theoretical knowledge in the professional field, and the capacity to work with minimum supervision (Higher Education Council 1992: 23); and professional qualifications, analytic ability, current knowledge in the field, ability to learn, achievement orientation, independence, career orientation, and ability to innovate (Higher Education Council 1992: 24). The Generic Skills Scale measures the extent to which graduates believe that their course has enhanced some of these skills.

Comparison of Scales

The means presented in Table 2 for the five scales cannot be compared directly. Simply because the mean of the GTS is 6 and the mean of the AAS is 36, it cannot be concluded that in some sense the appropriateness of assessment is higher than quality of teaching. The means of the scales are an artefact of the wording of the items from which they are formed. It is not the overall means that are important, but rather the size of the differences between the means of various categories, be they categories of gender, field of study or university attended.

Table 3: Course Experience Questionnaire Scales by Selected Fields of Study: Mean,                Standard Deviation and Sample Size, Degree Graduates, 1994

See Notes to Tables

Fields of Study

Table 3 shows the means, standard deviations and sample sizes of the five CEQ scales and the Overall Satisfaction Item (OSI) for ten fields of study for pass degree and honours degree graduates. These are the fields of study that form the basis for the analyses in this report. The criteria for selecting these fields of study were that each contained substantial numbers of respondents from a reasonable number of universities, that the fields were broadly spread across the range of courses offered by universities, and that corresponding information was available from DEETYA’s Higher Education Student Data Collection.

One of the major determinants of CEQ scale scores is the field of study of the graduate’s course. Table 3 shows substantial variation between the mean scale scores for the ten fields of study. For the GTS, the means vary from 33 and 23 for History and Literary Studies respectively, to -16 for both Law and Medicine. In the context of a measure with a standard deviation of 40, these are quite substantial differences. These differences are paralleled, more or less, for the CGSS and the OSI, albeit with somewhat less variation between the fields of study. For the AWS, Law has a substantially higher mean compared with its relative position for the other scales and Chemistry has a relatively lower mean. For the AWS, Engineering has a relatively lower mean.

There are several possible reasons for these differences. The most obvious is simply the course content. While it could be argued that this should not affect the quality of teaching, there may be subjects which are inherently more difficult to explain or present in an interesting manner. It might be expected that the content of a field of study would have more influence on the assessment of the course. A second reason for these variations in CEQ scale means might be the culture of the field of study. The importance that is attached to teaching. It may be that, within history and literary studies, it is relatively more important to be seen as a good teacher than in say medicine or law, which may emphasise other criteria. A third reason may be the educational background of students undertaking the course. The order of fields of study on the GTS broadly corresponds the minimum Tertiary Entrance Scores (TES) required for admittance to the course. Medicine and Law, which have very high cut-off points, have low GTS means, while History and Literary Studies, located with Arts and Humanities faculties, have lower tertiary entrance scores. Since TES is a matter of interest for later analyses, we will explore this argument a little further now.

The claim that differences in TES may be at the root of CEQ scale differences between fields of study is based on the claim that students (and hence graduates) with higher TES scores have higher expectations than students with lower TES scores. The word critical is sometimes used in this context, but care is required because this is an over-determined concept. Presumably what is meant by critical in this context is an ability to offer a more accurate assessment of phenomena, or situations, rather than some predisposition to negativism. That is, after all, what might be expected of the best graduates.

If there is an effect due to differing expectations, it does not explain, for instance, the substantially lower between-field-of-study differences for the CGSS or the OSI compared with the GTS, unless the effect is claimed to be differentially teaching specific. Nor does it explain the differences in relative ratings for chemistry, engineering and law across the various CEQ scales. Nor does it explain the lack of any substantive difference between pass degree and honours degree students for CEQ scale scores (Ainley and Long 1994: 16).

The likely effect of TES needs to be put in context. Much of the variation in TES is removed when dealing with the population of university students because candidates with low scores are not admitted to university study. Hence the lower-end of the distribution is truncated. This is a reason for the quite modest relationships observed between TES and first-year university performance and between TES and the probability of graduation. To the extent that there is any (positive) effect of TES on the probability of graduation, variation in TES is likely to be further reduced in the graduate population. However, it should also be noted that not all undergraduate students enter university on the basis of their TES.

Nevertheless, if there is an inverse relationship between TES and CEQ scale scores, it is possible that this could be responsible for differences in between-university CEQ mean scale scores, even if such effects were not observed to any great extent within a particular university. This would require quite substantial between-university differences in mean TES, a situation which may exist for some fields of study. There are, however, fields of study for which the between-university variation in TES is likely to be minimal, medicine and law, for instance. Yet there is little evidence of substantially lower between-university variation for these fields of study.

The substantial variation in CEQ scale scores between fields of study within a university is one reason why the CEQ is important. It indicates that within-university comparisons of measures of student or graduate assessment of the quality of instruction in courses for different fields of study are likely to be misleading. Like is not being compared with like. In this sense, the possibilities of between-university comparison offered by the CEQ may be more valid than within-university comparisons, provided university comparisons are made within a given field of study and for a given level of study.

Summary

The Course Experience Questionnaire was added to the Graduate Destination Survey as part of a policy which required indicators of university performance. The 25 items on the questionnaire cover aspects of teaching, assessment, workload, skill formation, and overall satisfaction with the course. combined to form scales which previous research has They are suggested is environments. The scores for these CEQ scales vary related to effective learning substantially across the ten fields of study investigated in this report. Hence it is important to focus on analyses within fields of study.

[back to top]