4. Comparisons between Universities

[next chapter] [previous chapter] [contents]

Percentage of Variation Explained
Changes in University Means

Engineering
History

Summary

This chapter examines the consequences of any relationships between CEQ scale scores and characteristics of graduates for comparisons between the mean scale scores of universities. It asks, and answers, the question Are differences in mean scores between universities due to the different characteristics of the graduates of those universities? The previous chapter examined the relationship between seven characteristics of graduates and CEQ scale scores and found evidence of relationships between some of these characteristics and CEQ scale scores for selected fields of study. The weakness of these relationships suggests that any effect on university differences is likely to be small, although this is not necessarily true.

The CEQ focuses on the evaluation of courses (rather than, say, of particular units within a course) creates the possibility of averaging. Graduates may find it difficult to summarise their experiences of a course which consists of many units into a single response for each item. If the responses of graduates are averages, there is a likelihood that the items will fail to discriminate between courses. There is also the difficulty that in some instances two graduates of the ‘same’ course may have undertaken a different set of subjects. Thus, when computing graduates of a given university respond to the CEQ they may be referring to quite different courses.

Previous investigations of CEQ results have usually treated campuses of the same university separately. In the analyses presented in this report, university means sometimes combine the results from several campuses of a single university. This approach has been taken to ensure consistency with later analyses which use DEETYA data from the Higher Education Student Data Collection which does not permit campus-level analyses. The combination of results from several campuses further increases any problems of averaging, and reduces the likelihood of detecting between-university differences.

Percentage of Variation Explained

One approach to examining the effect of a set of variables on another group of variables is to examine the change in the percentage of variation explained when the set of variables is added to the regression equation. The values in Table 5 show the percentage of variation explained in the CEQ scales by various groups of variables. Characteristics refers to the seven factors discussed in the previous chapter, age, gender, non-English-speaking background, prior educational qualifications, mode of attendance, and subsequent study and employment. University refers to a set of dummy variables which represent the universities, and permit the calculation of the variation for the differences in CEQ scores between universities.

In Table 5 the row headed by Characteristics & University shows the percentage of variation in CEQ scale scores explained by the seven graduate characteristics and university attended, Characteristics shows the percentage of variation explained by the graduate characteristics only, University | Characteristics shows the variation uniquely explained by differences between universities controlling for any differences in graduate characteristics, and University shows the variation explained by differences between universities without any controls for graduate characteristics (the difference between the values in the first and second rows). To reiterate, the row headed by University shows the percentage of variance explained by differences between universities without making any adjustment for any differences due to graduate characteristics. If graduate characteristics have any substantial effect on university differences, then there should be notable differences between corresponding values in the third and fourth rows.

There is, however, little difference, either in absolute or relative terms, between the values in the third and fourth rows in Table 5. This suggests that the set of seven graduate characteristics examined previously has little effect on between-university differences for scores of the CEQ scales. The first entry in Table 5 is for scores on the Good Teaching Scale (GTS) for graduates of accounting courses. Differences between courses account for 4.53 per cent of the variation in scores on the GTS for accounting graduates. Controlling for the seven characteristics of graduates reduces the percentage of variation to 4.30. This is a very small decline of 0.23 percentage points or about five per cent of the total relationship. The overwhelming majority of differences in Table 5 are of this order. Some of the larger relationships have somewhat larger absolute differences, but rarely larger relative differences.

The largest relative effect of controlling for graduate characteristics not based on a very small initial relationship is for the GTS for graduates of literary studies. The initial relationship falls from 6.90 per cent of the variation explained by university differences to 5.08 per cent after controlling for graduate differences, a decline of 26 per cent of the initial relationship.

Another approach to assessing the relative effect of graduate characteristics on differences between universities is to examine the relative sizes of university effects, and the effects of graduate characteristics. If the size of the effects of graduate characteristics were as large as the effects of university attended, there would be substantial variation in CEQ scale scores due to factors other than the major differences of interest that is, between-university differences. The values in Table 5 suggest that this is rarely the case. The variation associated with between-university differences is generally at least two or three times larger than the variation associated with graduate characteristics. By way of example, for the first entries in Table 5 for accounting graduates, 1.58 per cent of the variation in GTS scores is explained by graduate characteristics while 4.30 per cent (or more than twice as much) is explained by between-university differences in graduates’ perceptions of teaching.

Table 5: Percentage of Variance Explained in Course Experience Questionnaire Scale
               Scores by Graduate Characteristics and University Differences

See Notes to Tables

Table 5 presents a great deal of information which, however, does not necessarily bear on the issue of spurious sources of between-university differences in CEQ scale scores. For instance, the extent to which between-university differences explain the variation in CEQ scale scores is larger for some fields of study, and for some scales, than others. By way of illustration, the percentage of variation in GTS scores explained by university differences for chemistry, law and psychology (over ten per cent for each) is considerably greater than for history and medicine (about two per cent for each). The values in Table 5 also show that between-university differences tend to explain more of the variation in the GTS scores than in Clear Goals and Standard Scale (CGSS) scores, or responses to the Overall Satisfaction Item (OSI).

The absolute size of the effect of university differences on CEQ scale scores may appear low. If, for instance, for law graduates between-university differences (controlling for graduate characteristics) explain 12.53 per cent of the variation in the GTS scores, and graduate characteristics collectively explain a further 2.64 per cent, the total variation explained is only 15.17 per cent of the total variation. This leaves 84.83 per cent of the variation unexplained. In a sense, what is known about sources of variation in perceptions of good teaching is relatively small compared with what is not known.

Table 6: Correlations between Observed and Adjusted University Mean Scores for 
               Course Experience Scales: Degree Graduates for Selected Fields of Study

See Notes to Tables

Such an interpretation is a little unfair. The reliability of the scales (as measured by Cronbach’s alpha shown in Table 2) sets an upper limit on the variation that can be explained. That limit is the square-root of the reliability (Zeller and Carmines 1980: 10). As detailed above, the concept of course may also involve a certain level of summarising by graduates, and even variation in the stimulus to which they are responding for the same course. Given these considerations, it may seem surprising that between-university differences explain as much variation as they appear to do.

Explanation of ten per cent of the variation of some dependent variable in studies based on survey data is not quite as small as it might appear. There are some widely accepted results based on much smaller percentages of explained variation. Socioeconomic status of parents generally explains less than ten per cent of the variation in school achievement (Glass, McGaw and Smith 1981: 148). There is substantial individual variation in student achievement, yet family background is still accepted as an important determinant of school performance.

The results presented in Table 5 suggest that overall, generalising somewhat over fields of study and CEQ scales, the percentage of variation explained by between-university differences is relatively unchanged by consideration of graduate characteristics, and is large relative to the variation explained by graduate characteristics.

Changes in University Means

Although the level of variation in CEQ scale scores explained by between-university differences is important in itself, the percentage of variation explained is an aggregate or overall measure of relationship. It would still be possible for there to be a number of universities with very little difference in mean scores for a particular CEQ scale, but with one or two universities with either particularly high or low means. This might well correspond to a fairly low level of percentage of variance explained by university differences, but contain information that is valuable from a particular university’s perspective. In general, it will be outliers, courses that appear to be perceived relatively well or poorly compared with similar courses at other universities, that the CEQ can help identify. This section moves from overall relationships to a focus on the effect of graduate differences on the means of particular universities.

Table 6 shows the correlation coefficients for the relationships between mean university scores for the CEQ scales and mean university CEQ scale scores after adjustment for graduate characteristics that is, with the variation due to graduate characteristics removed. To the extent that there is little effect of graduate characteristics on CEQ scale scores, there should be relatively little difference between the two sets of means. The values in Table 6 broadly support this expectation. Most of the correlation coefficients are 0.98 or greater.

Table 7: Observed and Adjusted University Scale Means for the Course Experience
               Questionnaire: Civil Engineering Degree Graduates

See Notes to Tables

This is prima facie evidence of little movement in the relative sizes of university means due to the effects of graduate characteristics.

However, the approach taken in Table 6, is not conclusive. Adjusted university means might have a somewhat lower dispersion because of the removal of variation due to graduate characteristics. Correlation coefficients would not reflect this kind of shift because they are unaffected by any linear transformation of the scores of one or both variables. The values for the standard deviations of the observed and adjusted scores shown in Table 6, however, show relatively little difference, which suggests that this is not a problem. There is the further consideration that any comparison of university means after adjustment for graduate characteristics might well be undertaken in terms of means standardised on the distribution of adjusted scores in any case.

The purpose of Table 6 is to present a broad-brush picture to identify fields of study which are likely to show the greatest relative changes in mean university scores due to adjustment for graduate characteristics. There are several fields of study in Table 6 with correlation coefficients of 0.96 or less for at least one coefficient. Two fields of study, however, have been selected to illustrate the effects on the mean scores of universities. Engineering has coefficients of 0.94 for the CGSS and 0.95 for the Appropriate Workload Scale (AWS), and history has a coefficient of 0.92 for the GTS, the lowest correlation coefficient in Table 6.

Engineering

Table 7 shows the observed (unadjusted) and adjusted university means for the CEQ scales for engineering graduates in terms of standard deviations from the overall mean. The values for all scales will be discussed because it is just as important to convey a sense of the changes corresponding to a correlation coefficient of 1.00 as it is for a coefficient of 0.94.

The correlation coefficient for the observed and adjusted means for the GTS in Table 7 is 0.97. The greatest changes after adjustment for differences in graduate characteristics, for universities 3, 6 and 7, are a little less than 0.2 standard deviation units. Thirteen of the eighteen universities show differences of less than 0.1 standard deviation units. The means for the observed scores in Table 7 lead to a focus on university 10 as having a particularly high mean (more than half a standard deviation above the overall mean) and universities 15 and 16 as having particularly low means (more than half a standard deviation below the mean). These interpretations still hold after removal of any differences due to graduate characteristics.

Engineering was selected in part because the correlation coefficient for the CGSS was amongst the lowest values in Table 6. However, the largest differences are still less than 0.2 standard deviation units, and are marginally smaller than for the GTS. Interpretation of the observed means would focus on universities 12 and 14 with relatively high values (0.42 and 0.49 standard deviation units above the overall mean respectively) and, possibly, university 16 which has the lowest mean (0.34 standard deviations below the overall mean). Interpretation of the adjusted means would focus on the corresponding values, although the absolute differences in all cases are somewhat reduced.

The correlation coefficient for the Appropriate Assessment Scale (AAS) is 0.99. As might be expected with such a high correlation coefficient there is a strong correspondence between observed and adjusted means. The adjusted means are within 0.05 standard deviations for sixteen of the eighteen universities. Graduate characteristics make no substantial difference of to the means of any of the universities and any interpretation is, therefore, unaffected.

A correlation coefficient of 0.95 is relatively low in the context of the size of other coefficients in Table 6. The discrepancies between observed and adjusted means are still relatively small except for university 3 and university 7. The mean for university 3 is of particular interest because it is an outlier and declines from 0.73 to 0.52 after adjustment. It remains, however, an outlier. Once again the

interpretation of values is unchanged regardless of adjustment for graduate characteristics. University 3 is the high outlier and universities 14 and 16 are at the lower end of the distribution, although adjustment increases the mean for university 16 while reducing even further the mean for university 14.

Table 8: Observed and Adjusted University Scale Means for the Course Experience
               Questionnaire: Degree Graduates in History