|
|
||||||||||||||||||||||||
|
|
|
courses | hecs/oldps
| higher education home | higher
education links The 1999 Postgraduate Research Experience Questionnaire
IntroductionThe Postgraduate Research Experience Questionnaire is designed to provide information on the educational experiences of students in higher research degree courses in Australian institutions. 1.1 BackgroundIn Australia, as in other countries, increased attention on the quality of provision in higher education corresponded with an expansion of participation in higher education. In a situation of high demand for places in higher education, and limited resources with which to provide those places, the utilisation of resources becomes important. During the late 1980s a series of reviews of teaching in various disciplines (in Accounting, Engineering) and government enquiries argued the case for more systematic data gathering about teaching and learning in higher education. Government statements at this time crystallised the view that there was need for greater public accountability in higher education. It was argued that universities needed to be able to make judgements about whether the courses they provide are well taught, whether changes that are made to those courses impact on graduates’ perceptions and whether graduates consider their experience of the courses to be rewarding and satisfying. In this discussion it was suggested that inferences about the quality of courses be facilitated by relative measures, comparisons between universities and faculties, schools and departments and comparisons over time. The Performance Indicators Review Group (PIRG) recommended that an instrument structured initially along the lines of the Course Experience Questionnaire be incorporated in, or administered in conjunction with, the GCCA graduate destination survey. Since 1992, the Graduate Careers Council of Australia (GCCA) has included a Course Experience Questionnaire (CEQ) with its annual survey of graduate destinations: the Graduate Destination Survey. The CEQ is based on a theory that sees students’ perceptions of curriculum, instruction and assessment as key determinants of their approach to learning and the quality of the outcomes of that learning. Through this questionnaire graduates record their opinions about a number of aspects of teaching and learning in the courses that they have completed. This instrument has captured graduates’ perceptions of their higher education experiences, and is generally considered to be both robust and reliable. Twenty four items form five scales: Good Teaching, Clear Goals and Standards, Appropriate Workload, Appropriate Assessment and Generic Skills. In addition it contains a single item concerned with Overall Satisfaction. For a number of years, however, it has been recognised that the CEQ, which was originally developed for bachelor degree graduates, is inappropriate for the growing number of postgraduate research students in Australia. For this reason, in 1996 the Department of Employment, Education, Training and Youth Affairs asked the GCCA to develop a brief instrument in similar form to the CEQ which would gather data concerning the experiences of research higher degree graduates. 1.2 The Postgraduate Research Experience QuestionnaireThe current (1999) version of the Postgraduate Research Experience Questionnaire (PREQ) was developed by the Australian Council for Educational Research (ACER) and the GCCA. It has a number of potential uses: providing a national overview of broad issues; setting a context against which institutions might undertake their own analyses and identifying examples of good practice (ACER, 1999). 1.2.1 DevelopmentThe GCCA was responsible for the initial conceptualisation of the instrument and the development of the individual items. ACER conducted analyses of the data from the first and second trial versions of the PREQ, leading to the selection of the 28 items that form the six sub-scales of the present PREQ. Twenty-eight universities, involving some 2336 postgraduate research students, participated in the first trial. In the second trial 29 Australian (and three New Zealand) universities participated, involving 1832 postgraduate research students. In both trials two forms of the instrument were tested, one using an agree—disagree scale and the other a satisfied—dissatisfied scale. Data analysis was carried out using exploratory principal components analysis, item response theory and multi-factor Structural Equation Modelling, where appropriate. The ‘agree’ version of the instrument proved far more robust. There were far fewer problems with missing data, and interpretation of factors was far more straightforward. Item response theory analyses successfully identified relatively homogeneous clusters of items on the ‘agree’ questionnaire that were clearly related to the underlying dimensions of the student experience. Multi-factor confirmatory factor analysis confirmed that the measures based on the six clusters of items did indeed identify separate constructs as described in the exploratory factor analysis. Readers are referred to the report on the evaluation and validation of the trial PREQ (ACER, 1999) for a full discussion of the development of the instrument and sub-scales. More extended analysis of the PREQ trial data has been reported by Elsworth (1999). 1.2.2 FormatThe PREQ consists of 28 statements about research students’ higher degree experiences. Respondents indicate the extent of their agreement with each statement by selecting a number from one through to five (one standing for ‘Strongly Disagree’ and five for ‘Strongly Agree’). Those who believe that the statement does not apply to them or their experiences may indicate so by selecting ‘Does not Apply’. 1.2.3 The scalesThe purpose of the PREQ is to gather data concerning the experience of research degree (Masters by Research and PhD) graduates with regard to broad aspects of their studies. As with the CEQ, which uses 25 items, there is no attempt to be all-inclusive in terms of the experiences of the target group or in the range of possible experiences. There might be other dimensions that reflect particular circumstances of a field of study, a programme or an institution. Even within the areas addressed by this questionnaire there might be further issues to be probed in more specific studies. The strength of the present approach is that it focuses on dimensions central to the postgraduate experience in most fields of study. The items included in the PREQ and the scales that they form were initially based on a review of the literature and discussions within focus groups of research higher degree students. They were revised on the basis of item performance during the first and second trials (see ACER, 1999 for greater detail of the scale development). The scales that form the current version of the PREQ focus on the following six areas of research higher degree experience:
In addition there is, as in the CEQ, an Overall Satisfaction item. Appendix A contains a copy of the questionnaire. Chapter 2 provides information about the PREQ scales and its structure based on the current data. 1.2.4 Within-programme variabilityThe emphasis of the PREQ is on the overall experience as a postgraduate research student. It is sometimes argued that the experience of postgraduate study is different for each individual, and that, dependent on each supervisor, there may not be systematic patterns within units (departments, schools or faculties) in institutions. The argument is an extension of a general issue about studies of learning environments experienced in bachelor degree programmes (eg using the CEQ) or in schools. It is recognised that for most measures of learning environments there is variation among respondents within courses and institutions. This variation arises both because individuals may have different perceptions of the same situation and because there can be actual variations in the situation that is experienced within a field of study within an institution. Despite this, in university bachelor degree programmes, and in schools, it has been possible to establish shared or common perceptions of programmes and to relate those perceptions to characteristics of the course (Wilson, Lizzio & Ramsden, 1997). Although it can be argued that the postgraduate research student experience might differ between supervisors even within the same department, there are other aspects to consider. Postgraduate research students may learn from each other, and from academic staff other than their own supervisor, depending upon the environment of the department in which they are enrolled. Indeed, the way supervisors interact with postgraduate students may be shaped by formal requirements of the department and by the less formal patterns of work across the department. In other words, the extent of within-programme variability is a matter of interest but it should not be assumed that the experience of postgraduate study is entirely supervisor dependent. Since the PREQ encompasses a number of dimensions it should be possible to explore whether within-programme variability is greater on some dimensions than others. 1.3 The 1999 surveyThe PREQ was sent to approximately 4600 research higher degree graduates, who had completed either a Masters by Research or a PhD in the financial year 1998 to 1999. Responses were received from 2253 postgraduates in 35 Australian universities and the University of Waikato in New Zealand. There were 2225 respondents from Australian universities. The largest number of responses from an institution was 262 and the smallest was four. On average there were 63 respondents per institution. 1.3.1 Response ratesThe overall response rate was 49 per cent. The average response rate for participating Australian institutions was just less than 50 per cent. Three quarters of the institutions had a response rate of 40 per cent or more and one quarter of the institutions had a response rate of 57 per cent or more. The distribution of response rates is shown in Figure 1. Figure 1: Institutional response rates to the PREQ in 1999
The overall response rate is not above the minimum 50 per cent required by the AVCC guidelines for reporting of the CEQ results, but is similar to the response rates obtained by ACER (1999) in the developmental studies of the PREQ. It may be that there are some characteristics particular to this population of graduates that is limiting response rates to the PREQ. Findings from the PREQ 1999, therefore, should be interpreted with caution, due to the less than ideal response rate, as findings reported here may not be representative of the feelings of the entire postgraduate survey population. The issue of potential bias from non-response affects how one deals with the accuracy of the data. The achieved set of responses is not a structured sample from the population and the estimation of errors based on the presumption that it is a sample is not appropriate. If the response rate were 100 per cent, sampling error would not be relevant, although one would still need to allow for measurement error. The question is the extent to which one should allow for the potential bias introduced by non-response. From the response data it is evident that there is little difference in response rate by level of study. For the 3141 PhD graduates who were sent a PREQ form, 1570 were returned, giving a response rate of 50 per cent for PhD graduates, whilst the corresponding response rate for Masters by Research graduates was 45 per cent. As a check on the extent of bias in response, it is possible to compare the distribution of key characteristics among respondents to the PREQ survey in 1999 with postgraduate research students in general in that year. These characteristics were level of study, broad field of study, gender and age. Information relevant to that comparison is contained in Table 1. From those data it can be seen that the distribution of those characteristics among respondents to the PREQ matched the corresponding distribution among postgraduate research students. There was a slight under representation of engineering graduates. There was also an under representation of graduates aged 25 and below but that could reflect a real difference between the graduate population and the student population rather than any bias in response. Overall these data suggest that the respondents to the PREQ in 1999 did not reflect bias in these background characteristics. 1.3.2 Issues arising from the numbers of respondentsThere are other issues that arise from the number of respondents to the survey apart from the potential bias that might arise from differential non-response. Firstly, there are some fields of study in which the numbers are small. Secondly, even in fields of study with larger total numbers of respondents, the numbers from particular institutions may be too small to provide a reliable estimate of the learning environment. The numbers involved necessarily limit the range of analyses that can be conducted on the basis of one annual survey. It may be possible to extend the scope of analyses, if there is a cumulation of data from successive surveys. 1.4 About this reportThis report of the 1999 postgraduate research experience survey is structured around six chapters. This first chapter introduces the background to the report. It outlines the nature of the data on which the analyses in the report are based. The second chapter provides a national overview of the survey results. It provides information about the distribution of item responses, information about summary statistics to characterise items and scales and some information about the structure of the instrument. The third chapter describes some of the associations between PREQ measures and background characteristics of respondents. Chapter 4 probes differences between masters and doctoral graduates and variations among broad fields of study. Differences between masters and doctoral graduates are in the directions expected but differences among fields of study provoke speculation as to sources of difference. Table 1: Characteristics of respondents to PREQ 1999 survey and postgraduate completions in 1998
Chapter 5 contains an exploration of the patterns of difference across institutions. The extent to which there are identifiable differences among institutions is a controversial issue (Marsh, Rowe & Martin, in press). In many senses it is an issue that will become clearer when there have been successive collections of data so that the extent to which patterns are replicated can be assessed. It is an issue that is separate from the properties of the PREQ as an instrument for obtaining graduates’ views of the postgraduate research experience. It would be possible for an instrument to be highly reliable in terms of generating scores reflecting the views of individual respondents but for no differences among institutions to be detected. The final chapter draws together the results in a brief conclusion about the future uses of the PREQ and similar instruments.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
contact
details | search |
archive search | publications
| site map | subscribe Any comments or queries should be sent to: wwweditor@dest.gov.au This page was
last updated on
Tuesday, 26 August 2008
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
