10. Analysis of System-wide Data

The students: characteristics of the student body

In 1995, 448,630 students were enrolled in bachelor courses, and 162,901 of these commenced in 1995. Nearly 15,000 were international students, most of whom being full-fee paying students, leaving 147,920 local commencing bachelor students. Table 1 examines students’ enrolments according to their ‘basis of admission’ to university. Universities assign a basis of admission value to each student’s record at the time they commence their courses.

Table 1: Commencing Bachelor Students, 1995 by Basis of Admission

This table demonstrates clearly the diversity of students’ backgrounds in terms of their basis of admission to university. About 22% of commencing bachelor students had prior experience of higher education, comprising 11,744 who had previously completed a higher education course, and 20,421 with an incomplete higher education course. Of those who had completed a course, some were students returning to undertake professional degrees, such as law; some were students entering graduate-entry courses (such as some of the medical schools); and some might be described as ‘hobby’ students. Most of the students with an incomplete higher education qualification are course switchers. Their reason for switching courses could vary from failure in previous course, at one extreme, to switching into courses with extremely high post-school tertiary entry score cut offs, such as law.

Students entering higher education with a TAFE background comprised 10,426, or 7% of the 1995 intake. This group is one increasing in importance and size, but it is nonetheless small in the overall scheme of things. ‘Special Entry’ students comprised about 9% of the total admissions into bachelor degrees, about 6% being ‘Mature Age’ entrants, and 3% ‘Other Special Entry’.

The numerically largest group entering university were School Leavers, comprising 54% of the commencing cohort in 1995. Although the majority of university commencers were school leavers, it should be remembered that 22% of entrants have prior experience in higher education. Controlling for university-experienced students means that school leavers represented 69% of ‘new to higher education’ commencers. But it is also true that were one to look more closely at the experienced higher education commencers, it would be found that the majority had been school leavers when they first entered universities. This merely confirms the importance of the school leaver cohort as the principal student input source for universities.

Table 2 shows that in terms of the basis of admission there is considerable variation between states. In 1995, Victoria drew 65% of its overall commencing cohort from among school leavers, whereas NSW took only 48% from this source. (By contrast, 15% of the NSW intake was drawn from special entry categories, compared with 5% in Victoria. The NT, WA and Queensland were the states/territories which drew the largest portions of their intakes from higher education-experienced students, with 36%, 31% and 27% respectively).

Table 2:Commencing Bachelor Students, 1995 by Basis of Admission & State

Table 3 suggests that there is also a wide variation between university as to the proportion of the intake drawn from school leavers. Of the larger universities, Melbourne (74%) and Deakin (72%) were the institutions relying most heavily on school leavers. Australia’s largest, Monash University took 65% of its intake from among school leavers. At the other extreme, Charles Sturt drew only 26% of its intake from school leavers. Edith Cowan (40%), Griffith (44%) and QUT (46%) also drew relatively low proportions from school leavers.

Table 3: Commencing Bachelor Students, 1995 by Basis of Admission &University

The final characteristic examined here is Enrolment Type. Table 4 shows the commencing bachelor population according to basis of admission and enrolment type.

Table 4: Commencing Bachelor Students, 1995 by Basis of Admission & Enrolment Type

About 75% of all commencing bachelor students were enrolled full time, with 15% part time and 10% external enrolments. Among school leaver commencers, the proportions enrolled were 92%, 6%, and 2%, respectively. External and part-time enrolment were more commonly the choice of students with prior experience of higher education. Only 11% of external students were school leavers.

How well do commencing students do?

One of the measures available to assess students’ progress through their courses is the Student Progress Unit (SPU). SPU is generated when subjects are successfully completed by students so one SPU is produced by the successful completion of subjects weighted at one EFTSU.

The SPU concept has been in use since at least 1979, when an AVCC Working Party chaired by Professor John Scott of La Trobe University was established to consider the proposition that universities should provide information on attrition in triennial submissions to the Commonwealth Tertiary Education Commission (AVCC 1985, p2). The Working Party considered that graduation rates were an inadequate measure of student progress and of institutional performance. The SPU was suggested as a more valid indicator of the extent to which students completed their studies with maximum efficiency and emphasised the positive nature of student progress.

The concept was rejuvenated in the 1980s by the Linke Report, and the measure used to produce the tables below is based on Linke’s Performance Indicator P2b: the SPU Ratio. (Linke, 1991). This ratio compares successfully completed student load with total assessed student load, using data file supplied by each university to the Government in the February of the year following that in which the subjects were taken by the student. As such, this measure provides a useful indicator of which students do better than others.

Table 5 summarises the relative performance of commencing as opposed to returning students. As can be seen, the performance by Returning Students (SPU = 0.89) was considerably better than for Commencing students, be they school leavers (SPU = 0.82), or the other categories of Commencing students (SPU = 0.83).

Table 5 Student Load and SPU Ratios 1995 by Basis of Admission

nb rounding errors apply

It should be noted here that there has to be some uncertainty about student load which has been reported as "Incomplete". This student load is left out of the equation for calculating SPU ratios, acknowledging that some subjects may be spread over more than one calendar year. In fact when dealing with first year bachelor (pass) courses one would not expect there to be many bachelor subjects STILL without results at the end of February in the year after the subjects had been taken by students. It is much more likely that most of these ‘incomplete’ results are in reality "failed" results for students who had long dropped out of their courses.

Table 6: SPU Ratios by university within state for Commencing and Returning Students.

Table 6 reveals that the smaller UNS institutions had the highest SPU ratios. Among the universities, Queensland and Melbourne appear to have had the highest overall rates. In terms of the difference between the performance of commencing students compared with returning students, James Cook, UWS and Macquarie showed the largest performance gap. The performance of Melbourne and UTS commencers was closest to that of their returning students. These two tables clearly demonstrate that commencing students did not perform as well as returning students, and that there is little difference in performance between school leaver commencers and other commencers.

The tables below concentrate on commencing students only. Table 7 shows the relative performance of commencing students by Enrolment Type. The fact that commencing external students demonstrate a fairly low level of performance, especially among school leavers is starkly obvious. External study would appear to be an acquired art, perhaps for more mature students. School leavers also performed relatively poorly as part time students.

Table 7: Commencing Bachelor Students 1995 by enrolment type

Are some disciplines harder than others? Are there ‘soft options’? Do technology students find the going harder? From reference to Table 8, it can be seen that the lowest performing AOU groups, for both school leaver and other commencing students were Mathematics/Computing and Engineering, although not far behind was Business/Administration/Economics, which also had a relatively low SPU output.

Table 8: Commencing Bachelor student, 1995, by AOU Group

The analysis in this section confirms that in terms of student performance, there was considerable variability within the system, between commencing and returning students, between universities, between enrolment types and between AOU groups. Although external and part time enrolments form only a minority of school leaver enrolments, it would seem that these are less comfortable pathways to a higher education for school leavers. Some AOU Group areas also appear to be more troublesome than others, and in particular Mathematics/Computing, Engineering and Business/Administration/Economics. The results on a university by university basis were no doubt influenced by the discipline mix (AOU groups) of offerings at a given university. The figures suggest that a university offering Mathematics/Computing, Engineering and Business/Administration/Economics in large quantities will demonstrate lower SPU output rates. Also, it could be that the relatively low SPU productivity in Business/Administration/Economics was caused by a relatively high proportion of part time enrolments, as part-time students under-perform full-timers, especially among school leavers. The variations demonstrated here need to be borne in mind by universities, when devising strategies to improve the performance of students at their institution.

Tertiary entry scores (TES) as an indicator of future success

Another important variable in terms of school leavers’ transition from school to university is their tertiary entry score (TES) in Year 12. There is evidence that there is a correlation between success at university and the level of success at school (Eg Dobson & Sharma, 1993, p208), and if this situation still holds, universities should be conscious of the fact, and how it might affect the performance of the students they admit. Universities’ reporting of entry score data is patchy, and for this reason, it was not a viable proposition to use aggregated system-wide 1995 data to test the correlation between TES and student progress. The material which follows is based on three Victorian universities.

Figure 1 confirms that there is a strong relationship between TES and SPU at all three universities, in particularly that SPU ratios were higher for students who scored higher TESs. This information is important, particularly for universities which attract students from the lower end of the TES scale. The whole concept of ‘value adding’ and being aware of appropriate levels of resourcing of first year teaching are intertwined.

The correlation between TES and SPU is a strong one, and the question must be asked: what would the effect on the curves be if more resources were spent on the preparation of first year students? It is hard to answer this question, because the level of funding devoted to first year can not be calculated from outside a university. On the basis of Figure 1 (above), it is not possible to tell whether Institution 1’s students apparently perform better because that institution spends more on first year teaching and support compared to Institutions 2 & 3. Of students with TESs between 70 and 90, Institution 2’s students do not appear to perform as well. How much might this be due to the composition of Institution 2’s courses and student body? Or does this indicate poor resourcing? Should Institution 3 look more closely at its ‘value adding’ if it intends to persist with students with relatively low TESs. Would increased expenditure improve the SPU productivity of Institution 3’s low-TES students? Universities need to look closely at the situation in their own institution to answer these (and many other) questions. What follows is a set of calculations which attempts to establish funding and staffing levels which would be applied to first year if the income generated by first year students were to be applied directly to them.

How much funding should be devoted to commencing undergraduates?

This question can be answered only by universities themselves, and as has been demonstrated above, there is considerable variation between university, enrolment type and AOU Group. Each university will need to establish for itself the appropriate level of resourcing based on the discipline mix of course offerings, modes of delivery (on- and off-campus), and the individual characteristics of the student body, (including the distribution of designated equity groups). No single figure is likely to be appropriate to every university. On the basis of the SPU analysis, and the TES/SPU correlations (above), an institution whose offerings were based around science, technology or business, (especially if it were also using distance education to any large extent) would seem likely to generate lower SPU ratios. If that institution also attracted primarily low TES-scoring students, the relative student success problem could be exacerbated.

If universities were required to supply more detailed student, course and financial data to DEETYA, it might be a simple matter to create a model in which student progress and the direct financial resourcing of teaching could be correlated. However, detail at this level is not collected (nor would universities be happy to supply more information than they do already), therefore what follows is a methodology which could help a university to assess the funding levels appropriate to the aim of ensuring adequate support for commencing students, using system-wide data to generate a benchmark.

In using system-wide data, a level of homogeneity across the system which does not exist (as demonstrated by the examination of the characteristics of the commencing bachelor student population) is assumed. However, the methodology will provide a system average which universities could thereafter adapt to suit their own situation. The analysis below seeks to establish (theoretically) the level of resources (teaching staff and funding) which universities might consider to be appropriate for committing to the teaching of commencing students, including school leaver commencers.

Universities have certain funds available to them, which they use to provide teaching, to conduct research, and to engage in various ways with the outside community. The greatest proportion of the funding for these activities comes from the Commonwealth Government in one form or another, even in an era which has seen the diversification of the funding base. The enrolments and performance analysis above suggested that commencing bachelor students didn’t perform as well as students in later years. In other words, first year proves to be a stumbling block for many. It could be argued that failure at the first year level simply reflects poor selection processes by universities, or perhaps poor preparation of students at secondary school. But in a climate in which government policy has been directed at enhancing opportunities for school leavers in particular, perhaps relatively high failure rates at first year merely reflect an absence of "nurturing" by universities. Whereas some students will have no difficulty in coping with university, other students (perhaps particularly those at the margin of selection) may find the adjustment they have to make to be greater.

It is not possible to derive from central data bases how much any university spends on first year teaching and support. It is therefore not possible to use statistical data reported by universities to correlate students’ success with actual expenditure for DEETYA. Such figures can only be calculated by universities themselves. However, what it is possible to calculate remotely is the ‘revenue’ generated by students, using the statistical collections (student, staff and finance) to notionally apportion income according to how the funding was generated by universities in the first place. By combining various data elements from aggregated student load files and staff files with expenditure estimates derived from finance statistics, it is possible to derive a dollar figure which reflects the cost of teaching in different disciplines.

The student and staff statistical data reported to DEETYA by universities each year include:

• academic and general staff by function/classification by department (AOU),

• student load by discipline by AOU, and

• a variety financial information.

Staff and student data can be linked by AOU, for the purpose of ratio calculation.

University funding is loosely linked to ‘relative teaching costs’, which were weighted according to a two dimensional matrix: discipline by course level. As such, it has been determined (via various studies of the relative costs of teaching students enrolled in subjects) that some disciplines cost more to teach than others, and that undergraduate teaching can be provided at a lower cost than postgraduate teaching in the same discipline. In particular, higher degree by research is expensive compared to other levels, in all discipline clusters. The methodology behind this ‘relative funding model’ was exposed in Baldwin (1990) and the ‘relative teaching costs matrix’ which provided a summary version of relative costs, is shown in Appendix 5.

The task is to establish some method for calculating how much commencing bachelor students "earn" for their university, and consequentially how much teaching and financial resource should be applied to them. A series of tables, showing student load, staff and financial resource figures for each AOU Group have been derived by using:

• staff data files,

• student load data files, and

• 1995 expenditure by universities on teaching academic staffing.

Staff data

The principal staffing resource pertinent to students are teaching academics (i.e., those engaged in the functions ‘Teaching-Only’ and ‘Teaching-and-Research’) working in AOUs. In 1995, there were 25,607 teaching academics in the Australian higher education system (excluding TAFE staff in universities).

Table 9: Teaching Staff by AOU Group -1995

Source: DEETYA Selected Higher Education Staff Statistics 1995, Table 8

The 463 teaching staff reported by universities as ‘unallocated’ have been excluded from here on in. These are staff designated by universities as teaching academics, but not linked to an academic department.

Student Data

Student load (EFTSU) records were sorted into disciplines within AOU Groups. By applying student load (EFTSU) by discipline to the factors outlined in the Relative Teaching Cost Matrix, weighted student load Weighted EFTSU (WEFTSU) was calculated, for each AOU Group. (See Table 10, below). By using the teaching staff numbers (summarised in Table 9), it was possible to calculate a ratio of WEFTSU to FTE Teaching staff, creating of student:staff ratio for each AOU Group. The resulting ratio then provided the basis for establishing the average amount of teaching resource to apply to each AOU Group. Table 10 below compares total EFTSU and total WEFTSU, and the ratio of WEFTS students to teaching staff.

Table 10: EFTSU, WEFTSU And Teaching Staff - 1995

The calculations in Table 10 are averages of the distribution of academic teaching staff over all course levels, to establish a ‘student:staff ratio’ which is based on weighted student load. The assumption is that universities should maintain at least this ratio in its provision of teaching to commencing bachelor students.

Financial data

Some basis for calculation of financial appropriate resource distribution was needed. The exercise to calculate the sums which might be considered to relate directly to teaching of students could take many forms. The model used here, as summarised in Figure 2, is based on Total University Operating Expenses in 1995 (DEETYA 1996). It was then assumed that 67% of this sum was required for academic purposes, the balance being used to fund academic support activities, infrastructure requirements and central administration. It is also known from reported finance statistics that about 17% of the expenditure on teaching staff costs were for non-salary items, leaving 83% of the expenditure relating to direct staff costs. The result of dividing the remaining sum by the total WEFTSU is a cost of $3357 per WEFTSU. This process is summarised in Figure 2, below:

Figure 2: Calculation of System-Wide Funding per WEFSU

1. Source: DEETYA Finance Statistics 1995, Table 1c

2. Based on various university allocation models

3. See DEETYA Finance Statistics 1995 Table

4. See Table 10 (above)

On the basis of these calculations, each, weighted student unit (WEFTSU) should be funded at about $3357. This calculation takes account of the relative costs of teaching at different levels in different disciplines, as outlined in Figure 2, above. Table 11(below) takes the analysis one stage further by using the WEFTSU/FTE ratio from Table 10, to establish the staff resourcing levels which would be appropriate in each AOU Group for All Commencing Bachelor Students, and for School Leaver Commencing Bachelor students. Table 11 indicates how much funding and how much staff would be allocated to each AOU Group on this basis.

Table 11 Commencing Bachelor Students - 1995

This analysis suggests that over $374M would have been the appropriate amount for universities to commit to commencing undergraduates. This provides the basis for two important performance indicators for universities, based on the allocation of resources. The calculations and methodology above provide a mechanism for comparing performance over time and/or between institutions, but in a sense hard data can only be calculated from within. If universities do not already have the information at their disposal, they could be well served in calculating just how much staff and financial resources are devoted to first year teaching, and comparing this with the later year teaching. These national figures suggest that $3357 should be spent on each Weighted EFTSU, and that there is an appropriate ratio of teaching staff to Weighted EFTSU in each AOU Group.

Individual universities would need to establish the appropriate numbers of academic teaching staff and financial resources which should be applied to first year teaching.

Key Performance Indicators

The analysis above has suggested several avenues which might be appropriate for the development of institutional performance indicators. Universities have to calculate for themselves how much staff and financial resource to devote to students in transition from school, so these indicators would not necessarily generate the same absolute values or proportions at every institution. The indicators suggested here focus on the principal variables. They will differ between universities, because of the differences between universities’ breadth of disciplines taught, and the diversity in the characteristics of the student population at each university.

The first two relate to resources usage for the provision of first year teaching. The third looks at institutional populations. Number 4 seeks to identify and compare ‘excellence’ for school leavers and non-school leaver commencing students. Indicator 5 is intended to help universities to focus on the students selected last into courses.

Indicator 1

Proportion of Teaching Staff attributed to first year bachelor teaching using a student:staff ratio (based on Weighted EFTSU) for each AOU Group.

By applying this ratio to sub-populations of WEFTSU, such as for commencing bachelor students, it is possible for universities avoid under-resourcing any area of teaching. The "norm" suggested by the analysis above might provide universities with a starting point for calculating their own resource allocations.

Indicator 2

Proportion of the Academic Teaching Resources applied to first year bachelor teaching

This indicator relates to the calculation of a dollar value for WEFTSU for each AOU Group, which is based on the level of university expenditure on academic teaching staff. System-wide figurings above suggested a dollar amount which should be applied to (weighted) teaching at all levels. Universities could apply this or a similar figure to their own situation, to see whether or not there are sufficient teaching resources being applied to first year.

Indicator 3

Composition of university student population.

This indicator is in reality a range of characteristics of the student population.

Among the important variables to monitor are the following:

• Proportion of school leavers

• Range of TESs by AOU Group/Discipline

• Proportion of external students

• Equity characteristics of the student body.

Universities would need to identify the variables appropriate to their own situation.

Indicator 4

Proportion of School Leaver and other Commencing Bachelor students generating 0.9 SPU or higher.

For the system, this indicator (and Indicator 5) can be calculated from aggregated data set, available from DEETYA, or by universities using their own DEETYA-format data files. For individual universities, the figures can be calculated from the Enrolment files, as provided to DEETYA each year. It is intended to identify the proportion of student "high flyers".

Indicator 5

Value Adding: Proportion of bottom quartile of school leaver commencing bachelor students by TER generating 0.9 SPU or higher.

This indicator is to let universities measure the improvement between cohorts of "last pick" students. The measure is suggested for school leavers only, because of the moderating effect that inclusion of non-school leavers might have on the total.

Conclusions

This section of the study started by confirming the diversity of Australia’s higher education system. Not only is there diversity in access to university, but also in performance. First year is a particularly testing year for school leavers, and there is an obvious correlation between Year 12 Tertiary Entry Scores and First Year SPU ratios.

Ideally, the proportion of staff and financial resources available at each university which should be applied to the teaching of first year students, would be known. Arguably, this would optimise the transition experience of students, particularly among those groups of students who do less well than other groups (eg: students with TESs below 60 in the institutions examined above). It was noted also that some universities might need to apply more or fewer resources than others, dependent on course, delivery and student characteristics.

Using system-wide data, it has been possible to generate average distributions of staff and funds between AOU Groups. Universities could adapt the methodology in order to provide themselves with a benchmark. Of course, if universities put together the appropriate data on transition (or other) students, they would indeed be able to test the affect on student performance of varying funding or staffing levels.