3: Methodology

[next chapter] [previous chapter] [contents]

• Introduction
• Statistical Analysis
• Student Survey

Introduction

This study was conducted in two phases. The first analysed historical and current statistical data in order to determine patterns of participation and outcomes for particular groups of students. The outcomes of this phase of the study are discussed in Chapter 4 of the report. The second phase analysed survey data collected from students who withdrew from the University during the first semester, 1995, and from a group of students who persisted with their studies in the same period. This analysis was conducted in order to develop an understanding of the factors that contribute to student withdrawal and to inform strategies designed to reduce student attrition. The results of this analysis are presented in Chapter 5.

Statistical Analysis

Data from the University’s Higher Education Student Collection Database and its Student Records Information System (1991–1995) were analysed for the study. Consistent with current national equity planning and reporting practice, fee-paying overseas student data were not included. Whenever possible, the definitions recommended by Martin (1994) and now being used nationally, have been applied in collating data. There may, however, be some inconsistencies in relation to historical data prior to 1994, particularly for those groups defined by postcode, such as low socio-economic status and rural and isolated students. Because the University’s information systems have not historically separated rural from isolated students, the two groups have been collapsed for most aspects of this study. Similarly, until 1995 disability status was not recorded on the student information systems and hence no analysis of students with disabilities has been possible.

Some analysis of equity performance indicators was also undertaken, but as relevant indicators have only recently been introduced across the sector and still require some fine tuning for accuracy within this institution, these were not a major source of data. The success rate and attrition rate algorithms employed in the analysis, however, are the same as those used for calculating the Equity Performance Indicators (Martin 1994). In a number of cases the indicators of student performance were tested for statistical significance using chi squared tests at the ninety-five per cent confidence level.

Student Survey

The survey reported in Chapter 5 employed both quantitative and qualitative research methods built into a two stage research design. The first stage involved posting a questionnaire to all commencing undergraduate students who withdrew from the University of South Australia during the first semester of 1995. A control group made up of ten per cent of the students who were still active at the start of semester 2 was also surveyed. The survey instruments are included as Appendices 3 and 4. The aim of the survey was to generate information about commencing first year students, including their reasons for deciding to study at university and those factors that had some effect on their decision to either persist or withdraw. A small number of respondents to the postal survey were interviewed in the second stage of the research. The aim of this stage was to develop a deeper understanding of the reasons for student withdrawal and any actions the University could take to improve student performance and to reduce attrition rates. Key informants such as student counsellors and study skills advisers were also interviewed to further inform the discussion in Chapter 5.

The date that students withdraw from the University of South Australia is significant with respect to various matters related to data records and financial liability. A number of students enrol but withdraw before the Higher Education Contribution Scheme (HECS) cut-off date of 31 March. These students do not incur a HECS liability, their records are subsequently deleted from the Student Records Information System and their enrolment or withdrawal is not reported to DEETYA. Their withdrawal does not, therefore, contribute to official attrition statistics. Students also withdraw without fail or withdraw and fail after the HECS deadline, depending on the date at which they withdraw. Because of the possibility that these three groups may have different characteristics, it was decided to survey students grouped to coincide with four critical dates. The first three groups were identified by the HECS deadline (group 1), the last day for withdrawal without fail (group 2), and the last day of semester 1 (group 3). Those who persisted and were still enrolled on the first day of semester 2 formed the fourth, or control group.

In respect to these groupings, questionnaires were distributed as follows:

• group 1, the HECS cut-off date, 283 questionnaires posted with 167 usable responses received (59.0 per cent response rate);
• group 2, the last day to withdraw without failing, 75 questionnaires posted with 40 usable responses received (53.3 per cent response rate);
• group 3, the last day of first semester, 81 questionnaires posted with 41 usable responses received (50.6 per cent response rate); and
• group 4, the first day of second semester, 618 questionnaires posted to a one in ten random sample of commencing students who were still enrolled with 391 usable responses received (63.3 per cent response rate).

Questions in the survey instruments were derived from related studies, in addition to some that were generated by the research team. A five point Likert type scale was used to elicit answers to many of the questions on both of the instruments. A mixture of positive and negative statements was included in the relevant questions in order to reduce bias in the responses. During the analysis, however, these statements were reworded and recoded to reflect a consistency in scores. The responses were averaged, analysed and are discussed in Chapter 5. The analyses of the survey results give an indication of the differences between groups of students which in turn suggests possible policy or strategy interventions by the University.

Likert type unidimensional scales are particularly useful in this type of study because responses to questions are readily coded as a numerical value enabling computer analysis. They are designed to produce a quantitative measurement from analysis of a set of qualitative responses. However, it needs to be remembered that these values are simply numerical codes for response categories that have been chosen by respondents from a limited range of options to suggestions framed by others. The response categories are often open to wide interpretation by respondents and hence consistency between responses cannot be assured. There is a temptation, nevertheless, to subject such qualitative data to sophisticated statistical analysis; for example, to calculate the mean of the responses for two or more groups of respondents and then use statistical techniques, such as analysis of variance, to test for significant differences between groups. While the application of techniques such as this is suitable for continuous data, they are not reliable if applied in this study as the interval scale data fall between categorical and continuous data. Their use was not, therefore, considered appropriate for the survey analysis.

Further difficulties arise in using Likert type unidimensional scales in studies such as this when data have a bimodal distribution. For example, if half the respondents indicated that they knew what was expected of them in class and the other half that they did not, then students’ understanding of their study expectations will appear to be average using the mean of their responses. If these results were compared to a study in which most respondents indicated that their understanding of expectations was at an average level, one could draw the conclusion that there was uniformity in group responses when major perception differences actually would have existed between the two sets of data. A number of the responses to the questionnaires used in this research have resulted in bimodal distributions. Consideration of this has been given in the analysis where appropriate, through closer investigation of the raw data in order to develop clearer interpretation of the responses.

The authors are aware that surveys are subject to bias, irrespective of the sophistication of the survey design and analysis. Efforts have been made to eliminate such bias, but it is acknowledged that some exists and this will affect the interpretation of the results. There are three main sources of identified bias in this study. The first concerns the sampling method and the profile of respondents. The mailing lists used for each of the four groups were generated from the University’s Student Records Information System in which an indicator identifies those students who have formally withdrawn from their course by completing a withdrawal form. However, not everybody who discontinues their studies at the University of South Australia either completes a withdrawal form or applies for leave within the prescribed timeframe. Some students simply cease their studies and disappear. Student numbers in this category are known to be small and over time their records are corrected (with a withdrawn fail grade). Nevertheless, it is likely that more students withdrew from the University in the period covered by the study than were identified on the Student Records Information System. If the group of students who withdrew without completing a withdrawal form share a common characteristic, this could then affect the validity of the results.

A second source of bias relates to the characteristics, experiences and views of those who responded to the survey. While those surveyed were broadly representative, it is possible that a disproportionate percentage of respondents with a particular view or characteristic may have replied. Table 3.1 and Table 3.2 below provide a breakdown of respondents to the survey discussed in Chapter 5. As can be seen, response rates for women were higher than the total for all groups, except those who withdrew without fail after the HECS deadline. External student response rates were high for group 3, but low for groups 2 and 4, while Indigenous students (ATSI), particularly those who withdrew after the HECS deadline, had very low response rates. Response rates were also low for non-English speaking background (NESB) students, with no response at all from those who withdrew without fail. The rate was high for rural and isolated students who withdrew before the HECS deadline (group 1) while the rate was low for socio-economic status students in group 2. In general, the response rates declined as the negative implications of withdrawal increased, although this pattern also varied across equity groupings.

A third source of bias results from the construction of the survey instruments and the wording of the questions themselves. While these questions were generated to reflect the aims and intentions of the research, they inevitably reflect the authors’ own assumptions about the nature and relevance of various factors in relation to students’ experiences.

Nevertheless, and despite the biases discussed above, given the focus of the study and the nature of the analyses of data, the results provide useful insights into the factors that lead to some students’ withdrawal from their university studies. They also provide insights into the nature of the educational environment that affects the performance of students who have been admitted to the University through its diverse admissions policies, particularly equity group students.

Table 3.1 Respondents to Questionnaire by Survey Group and Equity Category

Table 3.2 Response Rates by Equity Group Category