11. INSTITUTIONAL BARRIERS IN UNIVERSITIES TO THE ADOPTION OF CML
J.D. STANFORD
Department of Economics
The University of Queensland
Introduction
While this paper reports the results of the Computer Managed Learning Project in the Department of Economics at the University of Queensland, it takes up some more fundamental issues arising out of the adoption of a new technology in University teaching.
The Project found that students generally like Computer Managed Learning because it gives them greater control over their learning and over the pace at which they learn. As well the Project concluded that Computer Managed Learning improves student learning and is a cost effective means of issuing and marking student tests and maintaining student records.
This paper is concerned with the following conundrum: although Computer Managed Learning has been shown to be a cost effective means of improving student learning and of managing university classes, it has not been adopted widely even in the institutions in which initial projects show the effectiveness of Computer Managed Learning.
The questions this paper addresses is why there are institutional barriers to the adoption and diffusion of educational technology and what these barriers are. The paper goes on to discuss the barriers which have been identified and to discuss how they may be overcome.
The solution to the problem of greater use of Computer Managed Learning in Universities is seen as requiring greater institutional cooperation so that computer systems developed in one institution can be used in others; the ideal arrangement is for a nation-wide system of computer based instruction. The paper goes to consider whether this is feasible.
2. Computer Managed Learning
Although computers are now being used extensively in education, the appropriate role of computers is still to be determined. A particular application of computers to education is Computer Managed Learning ; the distinguishing feature of Computer Managed Learning is that instruction is provided externally to the computer and that the computer is used as a testing and management device for the course.
Computer Managed Learning is a general system which allows many particular applications so that the lecturer is not limited to a restricted series of options imposed by an inflexible computer system but is able to make effective choices about the implementation and management of the course.
The fundamental requirements for the use of Computer Managed Learning are careful course design, elaboration of learning objectives and test construction. These requirements are consistent with the best practices in education.
Under Computer Managed Learning the computer does not have an instructional role in teaching/learning process but the function of the computer is to control, administer and issue testing of the learning process. Management of learning is the process of course design, specification of learning objectives, issue and marking of tests, provision of feedback and remedial support, analysis of module effectiveness, analysis of learning performance, preparation of reports to students and lecturers.
There is an important distinction between Computer Managed Learning and Computer Assisted Instruction. Computer Assisted Instruction is the learning process in which instructional material is stored in the computer and released to a student seated at a computer terminal. Computer Managed Learning is the system where instructional material is available externally to the computer and where the computer plays an important management function. While it is possible for a Computer Managed Learning system to incorporate Computer Assisted Instruction, it is generally not necessary or even desirable that it should.
The Computer Managed Learning system which will be described here uses a commercially available software package produced by CBTS Ltd from Calgary, Canada. This package operates on the VAX range of DEC computers.
The introduction of a Computer Managed Learning system to a university course requires the preliminary steps of:
a. specification of the course objectives;
b. division of the course into modules (or relatively self contained components of the course);
c. allocation of learning objectives to each module;
d. preparation of a study guide for each module;
e. construction of a testbank which includes a large number of questions structured according to module and course objective and the answers to these questions;
f. entering the names of students on the computer.
It must be pointed out that preparation of the system is very time consuming and that one of the major elements in the preparation is writing the items in the testbank and testing the answers. The CBTS package allows the following style of questions in the testbank:
true/false
multiple choice
matching and short answer
completion and spelling
calculation and random generation
assignment
A particularly useful feature of the CBTS package that it allows randomly generated numbers to be used in calculation questions and in other questions involving numerical quantities.
A Computer Managed Learning system provides the following test and management functions:
# generation of unique tests for each student
# marking tests immediately on demand
# provision of an electronic mail facility
# monitoring individual student progress through the course by the lecturer
# recording student marks and other data
# production of various reports on demand to the lecturer
# providing detailed statistical analyses of student performance
# assigning a degree of difficulty to each question
# generation of detailed statistical analysis of the answers to the test bank so that testing procedures can be modified in the light of experience
A Computer Managed Learning system offers significant potential advantages in course management and administration because computers have a comparative advantage in processing repetitive tasks in marking tests and recording data.
A Computer Managed Learning system provides the opportunity for the substitution of capital for labour in university teaching. This option is useful in circumstances where financial stringency exists in universities.
3. Results of Computer Managed Learning
The results of the experience of the Computer Managed Learning system over two years in the first year course EC110 Introductory Economics in the Department of Economics at the University of Queensland concluded that:
a. student performance in the unseen final examination had improved significantly since the introduction of the Computer Managed Learning system;
b. students generally liked Computer Managed Learning because it gave them greater control over the pace of their learning;
c. students learned new skills;
d. in general students regarded Computer Managed Learning as a positive learning experience;
e. Computer Managed Learning was cost effective;
f. Computer Managed Learning improved productivity in two ways:
(i) By allowing academic staff to manage more students with given resources and
(ii) by enabling students to progress through the course more quickly and at a higher level of achievement;
g. the costs of establishing Computer Managed Learning are relatively high where it is necessary to purchase hardware and software and to allow for the cost of staff time for courseware development- University accounting systems do not accurately record costs on a project basis but it has been estimated that the first year costs of establishing a Computer Managed Learning system are about $100,000;
h. Computer Managed Learning offers significant cost advantages in later years of the system life - it has been estimated that the average cost per student in an introductory class of 600 or more with an established system are of the order of $125;
i. a major cost in establishing Computer Managed Learning is the use of skilled and dedicated staff time - in the computing area general systems management skills are required as well as particular knowledge of the software package, while in the academic area course development skills and item writing skills are required - as well in the establishment stage many bugs are encountered and the project initiators have to devote much time to trouble shooting;
j. in later years the introduction of new courses to the Computer Managed Learning system or developments of the original course can be undertaken using less skilled staff since the procedures can be made a matter of routine;
k. the initial experience with a Computer Managed Learning system strongly suggested that organisational changes are necessary if the full benefits of Computer Managed Learning are to be reaped.
4. Barriers to the Introduction and Use of Computer Managed Learning
As the results reported above indicate, the costs of setting up a Computer Managed Learning system are high. Under current institutional practice in Australian tertiary institutions these are likely to be borne by the individual academic rather than the institution. There are of course benefits to the individual academic resulting from educational innovation, but the institutional incentive system typically overlooks the academic contribution to the institutions goals. This constitutes a significant barrier to the introduction and extension of the use of Computer Managed Learning in university courses.
Educational institutions are currently not organised to take full advantage of the benefits of the new computer technology. Moreover it appears that institutions are slow to change to reap these benefits. One example of institutional rigidity given in the 1987 Report which is not meant to be critical of the institution but shows how institutions find it difficult to come to grips with the new technology.
At the background of this point is the recognition that many innovations in Universities rely on the enthusiasm of individual academics and that these innovations are confined to one course in one institution so that when the academic concerned moves on to other activities the innovation dies. The history of university teaching shows many examples of innovations which have enjoyed a brief life of success and enthusiasm but have failed to diffuse throughout the university system and faded relatively quickly into obscurity.
We can classify barriers to the introduction of the introduction and use of Computer Managed Learning systems under the following categories:
i. Teaching/learning considerations
ii. Incentive questions
iii. Quasi-rents
iv. Organisational Questions
i. Teaching/learning considerations
One of the things we know clearly in economics education is that we have no effective learning theory; perhaps all we known for sure about learning is that not all students learn in the same way. Consequently it is difficult to analyse in theoretical terms the contribution of a new teaching technique to learning. Further the empirical evaluation of a new technique is made difficult by this lack of a theory and also by the absence of a standard method of analysis. The position is further compounded by the difficulty of obtaining data especially comparable pre-test and post-test data.
Universities have no formal or effective procedure for project/function evaluation of new and innovative techniques so the determination of whether a new technique is successful is subject only to ad hoc assessment which is usually highly influenced by subjective factors and personalities. Some of these factors are considered later in the paper.
ii. Incentive questions
It has often been remarked that the incentive system in universities is odd; academics are paid basically for their involvement in teaching but their academic and professional success is evaluated by their involvement in research and publication. The quest for promotion and higher appointments suggests that academics should concentrate on research and teaching rather than teaching. This is especially true in relation to Computer Managed Learning where as we have seen the establishment costs are high.
One view of university teaching would emphasise that teaching is not about learning ie ensuring that students acquire particular sets of objective knowledge or skills but about "the gatekeeper role". Under this view universities are seen as credentialing authorities whose role it is to screen students into categories so that rewards may be conferred upon the successful and denied to the failures.
This view would also point to "the hidden agenda" in teaching so that part of the education process is for students to find the real agenda which has to do with input criteria rather than learning knowledge. If this view of universities teaching is appropriate, then the actual techniques used in teaching are largely arbitrary chosen and there is little reason to change them unless the hidden agenda is changed.
iii. Quasi-rents
In any large organisation the existence of quasi-rents is thought to be common. This is likely to be so in academic departments where long serving members have implicit property rights in courses and where course structures and teaching techniques are changed only marginally and infrequently.
Innovations in teaching techniques may be seen as threats to these quasi-rents. Although there are benefits flowing from the adoption of new techniques, the distribution of these benefits does not necessarily coincide with the distribution of pre-existing quasi-rents. Similarly the distribution of costs of innovation are unlikely to match the distribution of benefits and we have found previously that in one documented case this is clearly so.
Under conditions such as these it is difficult to predict the outcome of any attempt at innovation although it would be surprising if the forces of inertia were not strong.
iv. Organisational Questions
Whether Computer Managed Learning is adopted in university courses depends on the nature of decision making in the organisation as well as the authority structure residing in the organisation. As well the flexibility in decision making and in the operation of the organisation is important.
Of critical importance is whether decision making about teaching techniques is the level at which decisions are taken. In the usual departmental structure where decisions are taken at relatively low levels in the organisation the decisions are likely to be influenced by the factors previously discussed so that decisions taken at this level may favour "one-shot" innovation but are not favourable to sustained organisational change.
Decisions taken at a higher level we would think would favour institutionalisation of new teaching techniques but may emphasise standard approaches rather than educational innovation.
If decisions are taken by technical based personnel this may create a conflict with academic personnel who fear domination of the teaching process by technology for its own sake.
Budgetary questions are important; especially in relation to the level of decision making and the degree of flexibility under which funds may be switched from recurrent purposes to capital requirements.
These comments remain highly speculative and require further work in a context of a fully specified theory of organisational behaviour. However the tentative hypotheses we draw from this discussion are that:
1. Universities provide the circumstances for innovation in teaching techniques but
2. they lack the organisational structure to institutionalise Computer Managed Learning; consequently we would expect to observe that Computer Managed Learning is used in institutions with a more formal organisational authority than universities rather than universities.
3. Computer Managed Learning is less likely to be adopted in universities if it is seen solely as a teaching activity.
5. A Proposed Solution
The solution to these problems lies in greater institutional cooperation so that testbanks or systems developed in one institution can be used in others; the ideal arrangement is for a nation-wide system of computer based instruction to be developed. To some extent this may happen spontaneously as innovators in scattered institutions develop networks and cooperation.
However there is a need for further research to determine the full extent of institutional rigidities and the full economies of the system.
The success to date of the Computer Managed Learning project has come from the insistence that the project is fully documented in a manner identical to research projects and that the process of analysis and evaluation of the project are as important as its implementation and description. As well there is the recognition that such projects will continue in the medium to longer term only if they are institutionalised in some way. Universities are particularly good at encouraging innovation but are not successful in providing the organisational and management structure for the continuation of innovations.
The proposal is that a national project for the development of Computer Managed Learning in universities be established through a consortium of Universities to provide a network for economics courses. The specific activities to be undertaken by this national project are:
a. preparation of question-banks for use in Computer Managed Learning ; these banks would be available for use by other institutions, subject to modification for their own particular circumstances, on the condition that credit was given to the national project, that modified questions were pooled and that data on student performance and question-bank performance was given to the national project for further analysis and research.
b. preparation of systems outlines and procedures for standard economics courses which could be adopted for use by other institutions on conditions similar to those in a above;
c. preparation of a manual for the introduction of Computer Managed Learning in university courses;
d. encouragement of academics from institutions in the consortium to work on the national project in terms of a, b and c and to work on the research aspects of the project.
e. analysis of the question-banks in the light of their performance at the different institutions which would allow the national project to assign a degree of difficult to each question and ultimately produce a validated national test of university economics.
f. research into student performance in economics over a number of institutions;
g. research into the cost-effectiveness of Computer Managed Learning by using pooled data from the institutions participating in Computer Managed Learning activities. The research would be able to employ a case study approach and as well a broader approach which would generalise from the range of experience;
h. research into the diffusion of innovative practices based on the new computer technology again by case studies and a more generalised approach;
i. research in the economics of Computer Managed Learning which would use all the information and data obtained to provide a complete view.
The question is whether a national project of this type is viable; certainly there are substantial barriers to inter-institutional cooperation which dwarf those in any one institution. However circumstances have changed in Australian universities which are now encouraged to adopt a national approach.
The proposal would not succeed if were seen as a teaching project but the proposal incorporates a strong research element which accord with the current incentive structure in Universities.
Results of the CML Project have been reported in:
Jon D. Stanford and Howard P. Cook, Report on the Introduction of CML in a Large First year Economics Course, Department of Economics, University of Queensland, 1987.
Jon D Stanford, "Computer Managed Learning and Student Performance in First Year Economics," Paper given at the 16th Conference of Economists, Surfers Paradise, August 1987
Jon D. Stanford and Howard P. Cook, "Computer Managed Learning - Its application to increase student performance using formative self-assessment," in J. Barrett and J. Hedberg (eds), Using computers intelligently in Tertiary Education, Sydney, ASCILITE, 1987.
Jon D Stanford, "Using Computer Technology to Improve Student Performance in Economics," Vol 2, Contributed Papers, Proceedings of the 14th International Conference: Improving University Teaching, Umea, Sweden, University of Umea and the University of Maryland College, June 1988, 408-417.
Jon D. Stanford, "Computer Managed Learning and Distance Education," Paper given at the ASPESA Workshop, Rockhampton, July, 1988.
Jon D. Stanford, Submission in respect of Inquiry into New Technology in Education, House of Representatives Standing Committee on Technology, Education and Employment.
[Return to Top] [Previous Part] [Next Part] [Return to Index]