[next chapter] [previous chapter] [contents]
Development of the Questionnaire
Development of the Email List for the Expert Pool
Phase 2-Email Feedback of Phase 1 Results
The project made use of a modified Delphi technique to obtain insight into future developments in information technology and teaching. In essence the Delphi approach is a particular method for collecting, organising, reviewing and revising the opinions about the future, using a panel of individuals who never meet face-to-face (Reid, Pease & Taylor 1990). In its original form the Delphi Technique involves selection of a panel of participants based on certain characteristics which make their opinions valuable to the outcomes of the study. The panel then receive successive questionnaires. These questionaries ordinarily require participants to make rank judgements. The results of the first round are summarised and then fed back to the panel. The panel is then asked to review their responses in light of the additional information provided. This iterative process is continued until consensus is reached among the panel members. The Delphi Technique was considered desirable because of its psychological, educational and financial advantages. Its main psychological advantage is that it allows participants to express opinion without the usual constraint associated with group interactions. Its educational advantage relates to the clarification of problems issues and goals achieved by both researchers and participants through the actual Delphi process. Finally, its financial advantages include avoiding the inconvenience and cost associated with gathering a group of experts in one place for a meeting.
Although we considered using the World Wide Web (WWW) (see for example comments in Appendix A), we decided to use email as the medium for conducting the modified Delphi study. There were several reasons for this decision. First, a WWW questionnaire would have meant relying on the participants to contact us to get a copy of the questionnaire. This would have reduced the response rate. Second, the cost of creating the web based questionnaire was beyond the funds available. Finally, we were concerned that not all potential participants would have appropriate WWW browsers. At the time the study was conducted the safest assumption was that most people would have access to standard text based email, and hence we restricted our study to this medium.
The issue of informed consent in email questionnaires is one that will receive increasing attention. The way in which we dealt with this was to send the questionnaire only to those participants who had agreed to participate on the basis of the information provided in a preliminary email. As the nature of the information collected in this study was not invasive, the issue of confidentiality was less critical, but nevertheless, warranted caution. We investigated the possibility of using an anonymous site on the WEB for submission, but felt that the non-invasive nature of the questionnaire did not justify the necessity of dealing with the technical difficulties and expense inherent in a Web based system.
Development of the Questionnaire
In order to clarify the domain of the project and to determine the focus of the questionnaire, literature and Internet resources were consulted, and interviews were held with informed academics and administrators at the following universities:
Edith Cowan University
Curtin University
Murdoch University
Charles Sturt University
University of Queensland
Griffith University
University of New South Wales
Queensland University of Technology
University of Melbourne
Royal Melbourne Institute of Technology.
Focus group discussions were carried out with staff at Macquarie University, most of whom had an interest in information technology approaches to teaching. Members of the Advisory Group were also consulted. The information sheet in Appendix B, which was given to all those consulted, formed the basis of the questions and discussion. The many relevant and substantive issues raised from this early phase of the research are integrated with the results and discussion in later sections of the report.
Development of the Email List for the Expert Pool
All those interviewed in the initial stages of the project were asked for the names of individuals within their universities that fitted several categories:
Key individuals in universities were identified and telephoned or approached for suggested names and various existing email lists were consulted (Librarians email list and that for senior university computing administrators, staff development units and NCODE participants, ODLAA), and selected presenters at the Ascilite95 IT Conference held in Australia.
An initial list of 439 names was gathered in this manner. These participants were sent the preliminary email information (described below and provided in Appendix B) outlining the study and asking those approached to nominate others whom they felt would be suitable.
A pre-pilot questionnaire was trialed among the four members of the Research Team. This provided an opportunity to decide on the optimum format for the email questionnaire. The four people in the pre-pilot group used different versions of Pegasus mail-for DOS, Windows, and Macintosh. Several issues arose from the pilot study. For example, it became apparent that the normal Likert scale, presented horizontally across the screen, could not be guaranteed to appear appropriately when received by a variety of email systems. An example of a horizontal Likert scale is given below.
| Strongly Disagree | Disagree | Slightly Disagree | Neither Disagree nor Agree | Slightly Agree | Agree | Strongly Agree |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
For this reason, the response options were presented vertically as indicated in the example below. Options were written to ensure that the stem and the response could be seen on one screen.
1 Strongly disagree
2 Disagree
3 Slightly disagree
4 Neither disagree nor agree
5 Slightly agree
6 Agree
7 Strongly agree
In pencil and paper questionnaires, it is unnecessary to repeat the frame for each question since it is visually easy to scan to the top of the page. In an email questionnaire, however, the frame and stem had to be repeated for every question to avoid recipients having to scroll backwards and forwards. This, of course, gave the impression of a very lengthy questionnaire. In practice, the questions could easily have fitted on five printed pages, a typical size for a questionnaire.
The reply requested respondents to place an 'X' next to their chosen option. Examples were given early in the questionnaire to illustrate what was required. With hindsight, square brackets should have been provided to indicate where the 'X' should have been placed. This would have made subsequent data collation considerably easier.
In addition, the requirement for compatibility necessitated the use of text mode for transmission, without any special formatting such as tab characters, bolding, centring or underlining. Even under these conditions, some replies received, primarily from Macintosh computers were often such that line breaks had been removed, necessitating considerable additional editing.
Based on the early interviews, reading and focus group studies, and using the format derived from the pre-pilot study, a complete pilot questionnaire was forwarded to the questionnaire development and advisory group. The pilot study was invaluable in fine-tuning the questionnaire and in identifying such factors as:
The pilot email questionnaire also provided an opportunity for the research team to learn about possible technical problems in managing the volumes of data being returned.
The questionnaire in its final form consisted of three parts (see Appendix C).
PART A related to usage-gauging the likelihood of computer-mediated applications in teaching and learning occurring within the next five years.
PART B related to barriers and forcesgauging which of the barriers and forces described will have an impact on the introduction of new technologies in the next five years.
PART C related to strategic issues-policy, planning, technical and resource issues associated with the role of information technology and computer-mediated communication over the next 3-5 years.
Copies of the questionnaire can be obtained by emailing Bhesketh@bunyip.bhs.mq.edu.au .
Prior to sending out the email questionnaire, all potential participants (N=439) were sent an information sheet (see Appendix B), informing them of the aims of the project, and what their task would be; namely to respond to the three part email questionnaire before the Christmas break, and to a follow up round in January 1996.
Potential participants who had not replied to the information email were sent one reminder, and then dropped off the list. Of the 439 sent this information email (and one reminder) 254 agreed to participate (58%). The three part questionnaire was sent to all those who agreed.
One reminder email was sent to those who had agreed to participate in the study, but had not returned their completed questionnaire. Of the 254 originally sent, 198 responses were received to Part A (75%), 179 to Part B (71%) and 167 to Part C (66%). The response rate to Part C was understandably lower as the questions were aimed at a subset of the group who would have been aware of university wide issues in relation to information technology policy. Of the 198 who replied, 54 were female and 134 were male.
The replies to the questionnaire were analysed using a computer program which separated responses and comments, and put the responses in a format that could be easily entered into a statistical package. Those email responses that were not in a format that could be analysed by the program were entered manually.
Phase 2-Email Feedback of Phase 1 Results
An important feature of the modified Delphi Technique involved providing feedback to participants of their responses, with an invitation to modify these and to comment on modifications. Typically the Delphi technique involves the provision of feedback of both group responses as well as the particular individual response. In this study only group feedback was provided.
It was decided that the easiest way of summarising responses in a meaningful way for the participants was to rank order the items within a section of the questionnaire while also providing the mean response for each item. These rankings are discussed under the results section of the report. Participants were asked to:
A rich source of information was gathered from this phase. These findings and the comments have been integrated into the results and discussion sections.
As in the first study several pilot versions of the feedback format were trialed among the researchers to ensure that formatting was appropriate. The project's Advisory Group formed the main participants in the pilot study for the second phase. They were sent the follow-up questionnaire discussed below, and asked to comment.
A reminder was sent to those who had not responded to the Phase 2 study, indicating that we would assume no response implied comparative agreement with the rank ordering. Of the 232 Phase 2 questionnaires sent, 75 (32.33%) replied. As the changes suggested by those who did reply were minimal, we feel it is comparatively safe to assume that the majority of the non-responders agreed with the rankings that had been sent to them.
In interpreting the results it is important to gain an understanding of the nature of those who responded to the questionnaire.
Table 1 provides a breakdown of response by university. Although the survey participation rate is clearly uneven, a wide range of universities are represented in the sample (traditional universities, those involved in distance education and newer universities).
Table 1: Respondents by Universities
| Universities | Frequency | Percent of Total Response |
| Australian Catholic University | 2 | 1.0 |
| Adelaide University | 4 | 2.0 |
| Australian National University | 6 | 3.0 |
| Bond University | 3 | 1.5 |
| University of Canberra | 3 | 1.5 |
| University of Central Queensland | 12 | 6.1 |
| Charles Stuart University | 5 | 2.5 |
| Curtin University | 9 | 4.5 |
| Deakin University | 8 | 4.0 |
| Edith Cowan University | 5 | 2.5 |
| Flinders University | 2 | 1.0 |
| Griffith University | 7 | 3.5 |
| James Cook University of North Queensland | 4 | 2.0 |
| La Trobe University | 3 | 1.5 |
| Macquarie University | 10 | 5.1 |
| University of Melbourne | 6 | 3.0 |
| Monash University | 17 | 8.6 |
| Murdoch University | 8 | 4.0 |
| University of Newcastle | 3 | 1.5 |
| Northern Territory University | 5 | 2.5 |
| University of Queensland | 5 | 2.5 |
| Queensland University of Technology | 7 | 3.5 |
| Royal Melbourne Institute of Technology | 5 | 2.5 |
| Southern Cross University | 6 | 3.0 |
| Swinbourne University | 1 | 0.5 |
| University of Sydney | 3 | 1.5 |
| University of New England | 3 | 1.5 |
| University of South Australia | 9 | 4.5 |
| University of New South Wales | 3 | 1.5 |
| University of Wollongong | 4 | 2.0 |
| University of Southern Queensland | 6 | 3.0 |
| University of Tasmania | 8 | 4.0 |
| University of Technology, Sydney | 5 | 2.5 |
| University of Western Australia | 3 | 1.5 |
| University of Western Sydney | 1 | 0.5 |
| University missing | 7 | |
| Total | 198 |
Respondents were asked to indicate whether they fitted one or more of several categories used to generate the sample. Table 2 provides the breakdown of responses to this question. Respondents could endorse more than one category so the percentages given are a proportion of the total sample for each question.
Table 2: Information Technology Role of Respondents
| Number | % Taking this Role | |
| 1. Staff with an executive responsibility for setting IT strategies | 87 | 51 |
| 2. Staff with an executive responsibility for IT operations | 50 | 25 |
| 3. Staff with an executive responsibility for educational development and teaching technology strategy | 110 | 55 |
| 4. Academics reflecting a range of views and experience in the use of IT in teaching | 118 | 60 |
| 5. Staff with an executive responsibility for the provision of distance education | 76 | 38 |
Current Position of Respondents
Respondents also indicated the nature of their current position. This information is given in Table 3
Table 3: Status of Respondents
| N | % | |
| Professor | 13 | 6.6 |
| Associate Professor | 14 | 7.1 |
| Academic | 98 | 49.5 |
| Library or IT coordinator | 10 | 5.1 |
| Instructional/Teacher Development | 4 | 2.0 |
| Director of a Division | 30 | 15.2 |
| Head of Unit | 4 | 2.0 |
| PhD Student | 3 | 1.5 |
| Other | 6 | 3.0 |
| Missing | 16 | 8.0 |
About 62% of the respondents classified themselves as academics.
Table 4 provides information about the area or academic department in which the respondents worked.
Table 4: Academic Area or Department
| Frequency | % | |
| Education | 29 | 14.6 |
| Library | 14 | 7.1 |
| Information Technology | 16 | 8.1 |
| Academic Services | 10 | 5.1 |
| Information Services | 7 | 3.5 |
| Distance Education | 16 | 8.1 |
| Teaching/Learning | 12 | 6.1 |
| Interactive Multimedia | 4 | 2.0 |
| Computing | 11 | 5.6 |
| Science | 13 | 6.6 |
| Humanities | 18 | 9.1 |
| Applied areas | 14 | 7.1 |
| Economics/Financial | 6 | 3.0 |
| Administration | 1 | 0.5 |
| Engineering | 3 | 1.5 |
| IT Strategy | 2 | 1.0 |
| Other | 4 | 2.0 |
| Missing | 18 | 9.1 |
| Total | 198 |
Although not representative of the university population, the group does reflect a wide range of areas in universities.