Executive summary
The Course Experience Questionnaire (CEQ) has been included with the Graduate Destination Survey since 1992 to attempt to capture graduates’ perceptions of their higher education experience. However it is widely recognised that the CEQ is inappropriate in a number of ways for the increasing number of postgraduate research students in Australia.
The Graduate Careers Council of Australia (GCCA) proposed in 1996 that a parallel Postgraduate Research Experience Questionnaire (PREQ) be developed, and this report details the development of this instrument.
After consultations with focus groups and an advisory committee comprising the GCCA’s Survey Management Group and representation from the tertiary sector, a list of items to be included in a questionnaire were developed.
As some criticisms of the CEQ related to the use of an agree-disagree scale, two lists of items were developed with slightly different wordings so as to test the appropriateness of both an agree—disagree and a satisfied—unsatisfied scale. The aspects of the research experience examined in these draft questionnaires included:
- Supervision;
- The thesis examination process;
- Issues about student goals and expectations;
- Faculty/department ethos and intellectual climate issues;
- Infrastructure issues; and
- Skills development issues.
All Australian institutes of higher education were invited to participate in the pilot survey. Twenty-eight universities participated, involving some 2336 postgraduate research students. The Australian Council for Educational Research (ACER) was commissioned at this point to analyse the data obtained from the pilot survey and to prepare a report on its suitability as an instrument.
A number of problems with the design of the survey were identified at this stage of the analysis. The ‘satisfaction’ version of the instrument did not list items randomly, but grouped them under the headings decided a priori.
It would be difficult, therefore, to accept the results of a subsequent analysis that derived the same factors from the data obtained. There were also high levels of missing data on this version of the instrument, particularly towards the end, indicating perhaps both that the satisfactory-unsatisfactory scale may be inappropriate for this instrument and that the instrument itself was too long.
The ‘agree’ version of the instrument did not have the same problems, in that the items were randomised and there were not the same amounts of missing data. Principal components analysis derived six subscales, reflecting aspects of supervision, ethos, skills development, understanding of the process, experience of examination, and library services. Unfortunately these last two factors were not well defined, having two and one item loading on them respectively.
In summary, it was found that neither version of the instrument was suitable in its present form, and so using the results from the pilot study two further instruments were devised, to be tested in a second pilot study.
At this stage of the study, 29 Australian and three New Zealand universities participated, involving 1832 postgraduate research students. Again, two forms of the instrument were used, one using an agree—disagree scale and the other a satisfied—dissatisfied scale. Items for both forms of the instrument were randomised and the ‘satisfaction’ version was kept shorter.
Data analysis was carried out using exploratory principal components analysis, item response theory and multi-factor Structural Equation Modelling, where appropriate. Recalling that there were hypothesised to be six factors underlying the postgraduate research experience, analysis was conducted that would examine the instruments for this structure.
For the ‘satisfaction’ instrument it was found that there was a great deal of confounding in the wording of the items, such that interpretation of the factors obtained from the principal components analysis would be very doubtful. There was also a large proportion of missing and ‘not applicable’ responses to these items. The ‘agree’ version of the instrument proved far more robust. There were far fewer problems with missing data, and interpretation of factors was far more straightforward.
Principal components analysis on both instruments confirmed five underlying factors in both instruments. These reflected aspects of:
- Supervision;
- Skills development;
- Intellectual climate;
- Infrastructure; and
- Clarity of goals/expectations.
As well, the ‘agree’ version of the survey provided an additional subscale that gives us some information about students’ perceptions of the thesis examination process. Following this, item response theory analyses successfully identified relatively homogeneous clusters of items on the ‘agree’ questionnaire that were clearly related to the underlying dimensions of the student experience, and multi-factor confirmatory factor analysis confirmed that the measures based on the six clusters of items did indeed identify separate constructs as previously described in the exploratory factor analysis.
The recommendation of this report is that the ‘agree’ version of the PREQ instrument should be used, and the final chapter examines the results obtained from this instrument. These analyses show that the subscales previously derived have good discriminatory power. They highlight a number of differences between students, by gender, level of award, mode of attendance and mode of study, as well as discriminating between similar courses at different universities, and by field of study.
To top
Note: A full copy (822KB)
of this report is available in PDF format
|