5. Course Characteristics

[next chapter] [previous chapter] [contents]

Hierarchical Linear Modelling
Response Rates

The Good Teaching Scale (GTS)
The Clear Goals and Standards Scale (CGSS)
The Appropriate Assessment Scale (AAS)
The Appropriate Workload Scale (AWS)
The Generic Skills Scale (GSS)
The Overall Satisfaction Item (OSI)

Tertiary Entrance Scores (TES)

The Good Teaching Scale
Other CEQ Scales

Summary
Conclusions

This chapter examines the effect of two variables—response rates to the survey and mean Tertiary Entrance Score (TES)—on Course Experience Questionnaire (CEQ) scale scores. Information which assisted in the calculation of both these variables was drawn from the Department of Employment, Education, Training and Youth Affairs’ (DEETYA’s) Higher Education Student Data Collection—an administrative database of enrolments and completions in all Australian universities. These two variables are characteristics of the courses rather than of graduates and this raises special issues in their analysis.

Hierarchical Linear Modelling

Hierarchical linear modelling (HLM) is a statistical technique that has applicability to a wide variety of research situations which involve nesting of individuals within groups (and perhaps the nesting of those groups within larger organisations). It can be used, for instance, to model the academic performance of students within schools. In this context, there are said to be two levels of variables. There are student-level variables (such as age, gender, ethnicity, and so on) which refer to the characteristics of individual students. There may also be school-level variables (size of school, whether the school is government or non-government, and so on).

There are parallels between modelling student achievement in schools and the examination of factors which affect CEQ scale scores. Graduates exist within universities. There are a number of variables which describe the characteristics of individual graduates and there are some which relate to the university course they have completed (for instance, the mean Tertiary Entrance Score (TES) of students entering the course) or which relate to characteristics of the data collected for that course (for instance, the response rate for that course).

HLM offers a number of advantages compared with earlier approaches (Bryk and Raudenbush 1992: 5–7). From the perspective of the present analyses, the major advantage is in dealing with problems of aggregation bias—the inaccuracy of estimates of relationships that can result when analyses are performed on data obtained by aggregating information collected at the graduate-level to the university- or course-level. This problem has been recognised for some time in quantitative educational research. The methodological issues highlighted by practices such as these are commonly referred to as ‘unit of analysis’ or ‘level of analysis’ problems. They originate from the incompatibility between the assumptions underlying the tools of analysis and the natural grouping of the data. Bias introduced by the process of aggregation may lead to inappropriate inferences when data collected at one level are analysed at another. Typically aggregation produces apparently stronger relationships between variables.

Table 9 shows unstandardised regression coefficients for relationships between the CEQ scales and response rates and TES means. These coefficients were produced from HLM models which fitted response rates and TES means separately, together with selected graduate characteristics. The graduate characteristics incorporated in the various models are detailed in the notes to Table 9. They were selected because they showed the greatest relationship with the corresponding CEQ scores in the analyses presented in earlier chapters.

Table 9: Regression Coefficients for CEQ Scales by Response Rates and TES Means:
               Degree Graduates for Selected Fields of Study, 1994

See Notes to Tables

Response Rates

The overall response rate for the CEQ was about 55 per cent of all graduates in 1993. Institutional response rates, however, varied from a high of 86 per cent to a low of just 18 per cent. There was further variation within particular fields of study. Such variation may reflect the nature of the graduate population—for instance, whether it is highly mobile after course completion. Because each university is responsible for surveying its own students, variation in response rates may reflect differences in the methodology used to conduct the survey at particular institutions, particularly the number of follow-up questionnaires sent to non-respondents.

The interest in response rates is prompted by speculation about possible differences between respondents and non-respondents in terms of their attitudes to their course. If, for instance, individuals with a more positive attitude to the university and their course are more likely to respond, then approaches to the survey that are persistent in seeking responses will eventually elicit responses (on the second or third follow-up) from graduates with less positive attitudes. All else equal, mean CEQ scale scores for universities with higher response rates will be lower.

The reverse might also be true—graduates with a negative attitude to their course may be eager to vent their spleen. If this is the case, then, all else equal, higher response rates may be associated with higher CEQ scores. Further effort in pursuing non-respondents elicits responses from graduates with more positive attitudes to their course.

There are other possibilities too. If differential non-response is not the result of differences in effort to obtain a response, but more a reflection of respondent predisposition to respond, and that predisposition is based on attitude to their course (either positive or negative) then differential non-response will simply serve to suppress between-university differences for means of the CEQ scales given a constant survey methodology across universities.

Universities will also be differentially able to contact recent graduates. Those universities with a substantial proportion of undergraduates who continue directly with postgraduate studies will be more likely to have current addresses for their graduates and hence be more likely to obtain responses. To the extent that continuance with further study is associated with differential responses to the CEQ (which, as suggested in previous chapters is small), mean CEQ scores may be biased up or down.

In any case, if the predisposition of graduates to respond is related to their perceptions of the course, then there should be an observable correlation of mean CEQ scale scores with response rates. Observation of this relationship at the aggregate level (such as in this report) is a necessary, but not sufficient, condition to establish this relationship. In dealing with aggregated data it is not possible to establish the extent to which any relationship between the aggregated variable and mean CEQ scores influences between-university differences in mean CEQ scores. Any observed correlation may be simply coincidental.

In order to establish the existence of an effect of response rates on between-university CEQ scale scores, additional information needs to be provided at the level of individual graduates. For instance, the responses of graduates could be mapped against the length of time required to respond or by whether they responded to initial contact or a first or second follow-up contact. It might also be possible to compare the CEQ scores of ordinary respondents with the scores of graduates who would ordinarily be non-respondents but who were pursued intensively by, say, phone interviews. Aggregated data is not sufficient by itself.

The response rate to the survey is the number of graduates who returned a completed CEQ divided by the number of students who completed their course in a given year and converted to a percentage. The number of graduates who completed their course was obtained from the course completion file of DEETYA’s 1993 Higher Education Student Data Collection.

The ideal would be to create a response rate specifically for each field of study. The level at which information is recorded in the Higher Education Student Data Collection did not always allow this to be achieved. The level of detail provided for the field of study of graduates varied between universities and across fields of study. Graduates of medicine, for instance, were coded specifically to medicine. Graduates of accounting, however, were rarely recorded separately and were instead included generically with all business graduates.

Response rates were therefore calculated according to the most precise detail available. For accounting and economics, the response rate is for business studies, for chemistry and computer science, the response rate is for science as a whole, for civil engineering, the response rate is for engineering overall, for history and literary studies the response rate for the humanities and social sciences has been used, for law and medicine, specific response rates were able to be calculated, and for psychology—which often enrols students from both the science and arts faculties (or their equivalents)—a combined science and humanities and social science response rate was used. For some few institutions for some fields of study, coding of field of study in the course completions file did not appear to correspond to the coding available in the CEQ survey and hence results for these universities could not be used in these analyses.

The Good Teaching Scale (GTS)

The values in Table 9 are unstandardised regression coefficients. These have a direct interpretation in terms of the scales of the variables involved. For instance, the very first value of -0.21 in Table 9—for the GTS and accounting graduates—can be interpreted as saying that for every increase of one percentage point in the response rate, the mean GTS for a given institution declines by 0.21 points. If, for instance, the response rate for a given course moved from 40 per cent to say 60 per cent (a quite large increase), the mean score for accounting for that university should decline by 4.2 points on the GTS scale or a little over one tenth of a standard deviation (a modest decline).

The coefficient has two asterisks next to it, indicating that it is statistically significant. Table 9 differs from other tables in this report in that a double asterisk corresponds to significance at 0.05 and one asterisk corresponds to significance at 0.10. Less stringent criteria are used because the effective sample size (in this case 33 universities) is substantially smaller than the samples of graduates that underlie the values in previous tables.

As noted previously, the use of the term sample in this regard is a little misleading. The survey is a population study. The statistics are population values and hence issues of statistical significance are irrelevant. In a sense, non-response makes the achieved number of responses a sample, but certainly not a simple random sample as assumed by the inferential statistics used to calculate values of statistical significance. Even if it were, estimates of sampling variability would have to be reduced quite markedly through a finite population correction (because a substantial proportion of the population is in the sample). The only reason for including indications of statistical significance is a what if approach. If the survey participants were a simple random sample, then the corresponding values are (or are not) statistically significant. It provides more in the way of psychic comfort than an indication of rigour.

The direction and magnitude of the regression coefficients for the GTS in Table 9 are not constant across fields of study. Economics, history and medicine all have regression coefficients similar to those for accounting. Computer science, civil engineering, law and literary studies have more modest coefficient of varying signs. Psychology and chemistry, on the other hand, have positive coefficients which indicate increases in mean GTS scores with increased response rates.

There may be some reason for this variability across fields of study (i.e. some genuine reason why the relationship should be positive for one field of study and negative for another). It is difficult, however, to advance any reason why this should be the case. Further analyses across other years of the survey and examination of changes in mean scores within universities as response rates have changed may establish that this is a stable structure. The more likely explanation is that these are relatively small effects, close to zero overall, and displaying substantial variability because they are based on relatively few courses.

The Clear Goals and Standards Scale (CGSS)

The CGSS displays this pattern more clearly. The coefficients are smaller and, if they were standardised, none would be larger than negligible (i.e. over 0.05). There are about as many positive coefficients as negative—a pattern corresponding to no relationship.

The Appropriate Assessment Scale (AAS)

The coefficients for the AAS show a slightly different pattern. To begin with, there are four small negative coefficients and three negligible negative coefficients. This is balanced against only one small and one negligible coefficient (the model for computer science failed to converge to produce any estimates). There is at best a hint of some small negative relationship between AAS scores and response rates.

The Appropriate Workload Scale (AWS)

The AWS shows a similar pattern—three small to moderate negative relationships, five negligible relationships, and one moderate positive relationship. On balance, there is little to suggest any consistent relationship, although the balance of evidence comes down marginally in favour of a negative relationship.

The Generic Skills Scale (GSS)

The story is somewhat monotonous, but again there is little indication of any relationship. Chemistry has a positive coefficient, medicine a negative coefficient, and the remainder are negligible. Again, the pattern of coefficients suggests no relationship between the GSS and response rates overall.

The Overall Satisfaction Item (OSI)

The regression coefficients for the OSI provide some indication of a negative relationship with response rates. Chemistry and economics provide the exceptions. Given that accounting and economics have the same set of response rates (i.e. for business studies overall) and that chemistry and computer science have the same sets of response rates (i.e. for science overall) but the coefficients for accounting and economics are quite different, as are the coefficients for chemistry and computer science, there appears less consistency than might be expected.

Tertiary Entrance Scores (TES)

The earlier discussion of differences in CEQ scale scores between fields of study raised the possibility that academic ability might influence CEQ scores. Fields of study which were more competitive to enter seemed to have lower CEQ scales scores, particularly for the GTS. This suggests the possibility of a negative relationship between ability and CEQ scores. In secondary education, however, there appears to be little relationship between ability and attitudes to schools (Ainley and Sheret 1992). Whether this result can be generalised to university graduates is unclear.

In this report TES will be used as measures of academic ability. TES are the values generated from Year 12 (and sometimes Year 11) assessment for entry to university and other tertiary courses. There are quotas on entry to most courses and applicants with the highest TES are admitted. A cut-off score for a particular course is the lowest TES which will provide entry to that course. Given that in Australia secondary education is the responsibility of State and Territory governments, the construction of TES is similarly a State-based activity. This means that Australia has eight sets of TES. Schema have been developed for translation of the scores for one State into those of another, although such translation has usually been problematic.

The TES were obtained from the enrolment file of DEETYA’s 1990 Higher Education Student Data Collection for students commencing their courses in that year. Prior to analysis the data were processed to remove dummy scores and other inconsistencies. The mean TES from four years previously were taken as an indication of the academic ability of students who completed their course in 1993. The choice of 1990 rather than 1991 enrolment data is somewhat arbitrary. However, while the majority of students enrol in courses of three years full-time duration, mean completion times exceed three years and some students are enrolled in courses of four years duration or longer. Thus, an overall four-year lag was considered to represent the best compromise. It should be recalled, though, that the mean TES for a course is likely to be more stable than cut-off points which may be fairly volatile even over the space of only a few years.

TES were examined for universities within States and converted to percentiles scores within States. These scores were then taken as equivalent across States, so that a score of 27 (that is, corresponding to the 27th percentile) in South Australia was taken as equivalent to a score of 27 in New South Wales. It should be noted that the percentiles were based on the Higher Education data collection, not the whole candidature for a given state. Using percentile scores has the advantage of removing the influence of outliers (distributions were often left-skewed) and made the mean a more appropriate measure to use.

The approach used here for standardising TES across states has some difficulties. It does not allow for between-State differences in retention to Year 12 and entry from Year 12 to higher education. It might be felt, for instance, that the Year 12 higher education entrant with the lowest TES in a State in which only 10 per cent of the cohort entered higher education could have a higher ability than the student with the lowest TES in a State in which 20 per cent of the cohort entered higher education. The percentiling approach does not allow either for any overall differences in school achievement between States, or for between-State differences in the extent to which students with the highest TES apply to enter higher education.

Information for some universities for some fields of study was not available. This made estimates in Tasmania, the Australian Capital Territory and the Northern Territory particularly problematic and universities in these States and Territories

were omitted from all analyses. Some other universities were dropped from specific analyses because information on TES was not available. For some fields of study the proportion of courses dropped from the analyses approached 25 per cent.

As with the determination of the number of completing students, information on specific fields of study was often not available on the enrolment file. This is only to be expected. First-year students rarely enrol in history per se, but rather in a bachelor of arts within which they may elect to undertake history. Similarly, it is a rare university that enrols first-year students in chemistry. They enrol in a bachelor of science course and may choose to study chemistry along with other science subjects. Mean TES were therefore often aggregated to major or broad fields of study in the same way as for the calculation of response rates.

The regression coefficients for TES in Table 9 are derived from HLM with CEQ scale scores as the dependent variable predicted by mean course TES at the university-level and graduate characteristics as specified in the notes to Table 9 at the graduate-level.

The Good Teaching Scale

The regression coefficient for TES and the GTS for accounting graduates is -0.22. For every increase of one point in the mean TES percentile score for a course, the mean GTS for that course declines by 0.22 units. A movement of one standard deviation of the between-university means (18.9) would produce a decline of about 4 points on the GTS or about 0.1 standard deviations.

There are, however, considerations which modify this interpretation. First, the TES coefficient for accounting is not repeated across all fields of study. Chemistry, economics, literary studies, psychology and to a lesser extent computer science show a similar direction and magnitude of effect. In contrast, history, law and medicine (because of its relatively small distribution of score) show negligible effects while civil engineering has a positive coefficient.

Second, with graduate-level information, it would be possible to explore the effect of TES on between-university differences as was undertaken in previous chapters. The information on TES used here is aggregated to the level of course. It is therefore not possible to determine the extent to which between-university differences in TES would affect between-university differences in CEQ scale scores. Any correlation between mean TES percentile scores and mean GTS scores which underlies the regression coefficient could just be happenstance.

Ideally what is required is graduate-level data on TES or some equivalent measure of ability. While the absence of a TES-GTS relationship (within given fields of study) would suggest that between-university differences in TES do not affect between-university CEQ scale scores, a complete test would require this information for graduates from all universities. The absence of this information, particularly in the context of inconsistent coefficients across fields of study makes it difficult to draw firm conclusions.

Other CEQ Scales

The remaining CEQ scales can be dealt with briefly. For all scales there are instances of inconsistency of relationship between mean TES and the particular scale. Given this caveat, the balance of coefficients for CGSS and TES suggests a possible negative relationship, for AAS a positive relationship, a negligible relationship for AWS and the GSS and possibly some indication of a negative relationship for the OSI.

Summary

This chapter examined the relationship between mean Course Experience Questionnaire scale scores for universities within particular fields of study and:

• response rates to the Course Experience Questionnaire for those fields of study for each university; and
• mean Tertiary Entrance Scores (as an indicator of academic ability) for those fields of study for each university.

There were difficulties with the measurement of both response rates and Tertiary Entrance Scores. For most fields of study neither variable could be defined at the student-level, but had to be defined at the level of broad fields of study such as engineering or science. It proved difficult to standardise Tertiary Entrance Scores between States and information on Tertiary Entrance Scores was not available for all courses.

Thus, these relationships were examined at the aggregate-level of courses. The existence of a relationship at this level is a necessary, but not sufficient, condition for there to be some effect on between-university means. In both instances, graduate-level information is required to establish any definite effect. The major finding for both these variables was that there was substantial variation in the magnitude and direction of relationships with the Course Experience Questionnaire scale scores for the various fields of study. Such variation is not explicable in terms of reasons that may be advanced for expecting relationships between mean scale scores and either of these variables.

It is possible to describe the balance of coefficients—an averaging of relationships across fields of study. If this is done, the most consistent effects are:

• negative relationships between response rates and the Appropriate Assessment and Appropriate Workload scales; and
• negative relationships between Tertiary Entrance Scores and the Good Teaching, Clear Goals and Standards and Generic Skills Scale scores.

Conclusions

This report investigated the possible influence of a number of different attributes of university graduates on their responses to the Course Experience Questionnaire (CEQ). The particular characteristics considered were student age, gender, English language background, previous educational qualifications, work and further study activities following course completion, and mode of attendance. The major finding was that any relationships with CEQ measures tended to be quite small, and were rarely consistent across the ten fields of study for which analyses were conducted. Furthermore, the effects of these graduate characteristics on between-university differences were also small. Generalising somewhat over fields of study and measures, the percentage of variation explained by between-university differences was relatively unchanged by consideration of graduate characteristics, and was large relative to the variation explained by graduate characteristics. In the majority of cases graduate characteristics had little influence on the mean scores for individual universities.

As a consequence of the small and inconsistent relationships between the graduate characteristics and the CEQ scale scores two further possible sources of bias, were considered - the effects of response rates and the academic ability of students undertaking a course on the CEQ scale scores for that course. As reported above, there was substantial variation in the magnitude and direction of their relationships with the CEQ scale scores for the various fields of study. Nevertheless, the magnitudes of the relationships for some fields of study for selected measures were sufficiently large to indicate the need for further research based on individual student-level information rather than on the course average scores used in this report.

Thus, the opportunity exists for individual universities to do more with their own results. Student identification numbers are included on the main questionnaire in order to allow the removal of respondents from follow-up mailouts. These could provide the basis for merging student administration information onto responses to the Graduate Destination Survey and the Course Experience Questionnaire. This would permit more appropriate student-level analyses of potential sources of response bias to be undertaken, and provide a basis for identifying correlates of non-response.

[back to top]