I
P
Mike Vitale and Kim Johnston
| |
E I P |
4: Management Processes: Evaluating IT
Investments Mike Vitale and Kim Johnston |
| [next chapter] [previous chapter] [contents] |
Introduction
Evaluation of Information Technology Investments
Findings
CASMAC and Inter-university Collaboration
Management Issues in Evaluating Collaborative IT Investments
The goal of the information technology evaluation portion of the project was to investigate and report on the management methodologies used by universities to review and evaluate current and proposed investments in information technology, and on the results of applying those methodologies. It was anticipated that universities would use a variety of methodologies, depending on the size, purpose, and other attributes of the investment, on the university's information technology strategy, and possibly on other factors as well. This view was based on the results of research in the private sector (for example, see Weill and Lucas 1992).
In fact the project revealed relatively little formal evaluation of any kind, either before investments were made or afterward. Except for the most straightforward investments, which are analysed on the basis of discounted cash flow, the universities studied tend to rely on informal and intuitive approaches to the review and evaluation of current and proposed investments in information technology. Across the group of universities, academic and administrative staff alike are nevertheless confident that spending on IT has, overall, led to improved quality of teaching and learning and increased efficiency of operation. Some believe that the nature of educational use of IT precludes quantitative analysis of most projects in the university environment, while others think that a more rigorous approach would be both viable and useful.
In most cases, some evaluation criteria are specified for the internal allocation of discretionary funds. This is consistent with seeding initiatives supporting research where evaluation of the proposal occurs first and a successful outcome usually leads to further funding. Also, increased compliance through internal and external audit, especially for annual reports, has increased the need for post-implementation reviews of IT systems development, particularly for administrative systems. It has also led to the formulation of performance indicators developed from corporate plans, that assist governing bodies in assessing the performance of the university. But in general, little formal evaluation of IT investments is currently conducted.
This chapter first briefly summarises some theoretical background on IT evaluation, then describes the team's findings. We discuss the difficulties of evaluating cost-benefit of both academic and administrative IT, provide some examples of current practice, and discuss the collaborative CASMAC initiative.
Evaluation of Information Technology Investments
As expenditure on information technology has grown, so has interest in evaluating proposed IT investments before they are made and reviewing the results of those investments afterwards. In recent years a number of books and a spate of articles have described various frameworks for assessing the strategic and economic value of information technology. To some degree these publications have been motivated by continuing scepticism among senior managers about the return on their organisations' investments in information technology. A continuing flow of stories in the press about expensive IT project failures (See 'Managing risk in large scale projects' below) has fueled this scepticism in some and, in others, led to the belief that there is little that can be done in advance to estimate the potential returns from an IT investment. In fact, as described below, many universities are receiving substantial benefits from information technology, and some are moving towards reliable processes for estimating these benefits in advance.
Since the value received by organisations from the use of information technology is multi-dimensional, it is not surprising that the published frameworks span a wide range, from quantitative approaches focused solely on demonstrable financial results to qualitative approaches that include multiple perspectives. The challenge for universities in an era of ever-increasing emphasis on the use of information technology and ever-growing financial constraints is to select the evaluation framework(s) most appropriate to their mission, strategy, and resources, rather than to search for a single 'right' technique or to abandon the concept of evaluation altogether.
In the final chapter of their 600-page book, which is filled with descriptions of approaches, methods, and techniques for evaluating information technology, Banker, Kauffman, and Mahmood (1993) offer guidelines that they believe 'should be applied whenever a new investment is under consideration' (pp. 600-602):
objective versus subjective
qualitative versus quantitative
single or multiple period, single or multiple site
historical baseline versus future projections; and
Although the guidelines are stated in the terms used by private firms, nothing about them except terminology would appear different from what might be recommended to a university. Stakeholders would include the governing body, academic and general staff, current and prospective students, government officials, taxpayers, and so on. In some cases, the locus of value - the key areas in which stakeholders will be most affected - will fit neatly inside a department, school or faculty. Often, however, the value of a given investment may cut across the entire university or even extend outside it. This kind of extended value will be recognised only by analysts who are able and willing to consider a broad set of possible stakeholders and a range of levels of analysis.
Examples of the latter occur when a new system improves automation of processes in central administrative systems, such as streamlining procurement through direct on-line entry. Although they return some benefits to the centre in that functional area, no benefits are delivered to the IT unit, while considerable benefits are realised by the end user of the system through improved productivity. Traditionally, these gains have been identified through a more general statement of 'efficiency improvements through automation' as part of an enterprise bargaining agreement.
Although objective data is to be preferred when available, the cost of collecting it can be high, and in many cases subjective data is an adequate substitute. Finally, targets should be set for the benefits expected from the investment, and a follow-up study should be done to see the proposed level of benefits are actually achieved.
The relative lack of importance that most of the universities studied place on the quantitative analysis of IT investments, even in the area of administrative computing, is suggested by the fact that few appear to collect detailed information about the actual costs of information technology, much less try to quantify the generally more elusive benefits. Naturally there is an overall budget for the central IT organisation, and usually budgets for at least the major projects that this central organisation undertakes. In addition to the central budget for core services, a number of universities have a discretionary IT budget that is usually allocated by an advisory committee which supports both strategic and innovative agendas. It does not however appear that actual project costs are routinely compared with project budgets, nor are actual project outcomes regularly compared with the results promised when the projects were undertaken. At the University of New South Wales, for example, while steering committees for administrative systems projects did periodically assess and cost system specifications and modifications against the original budget, post-implementation reviews of outcomes have not typically been conducted. At another university, proposals for administrative systems are subjected to scrutiny by a steering committee, albeit without routine follow-up to see if promised benefits have been achieved; proposals for systems to support teaching and learning are typically not accompanied by a cost-benefit analysis, or even by a careful statement of anticipated benefits. While a certain amount of research is clearly appropriate, particularly with regard to teaching and learning, it is not obvious that all such efforts should be treated as one-off experiments.
None of the universities studied has more than a rudimentary chargeback system for administrative IT, although basic chargeback was often in place for a number of support units and network related areas. (Here we define 'chargeback system' as a mechanism by which internal users of organisational resources are routinely charged for the use of those resources. The internal telephone billing system used by many universities is a common example of a chargeback system.) For example, the University of New South Wales and University of Queensland both charge for network connections, while the University of Wollongong includes the cost of network connection in the price of computers from their computer shop. Some universities have no chargeback whatsoever for IT services. In the for-profit sector, chargeback is widely believed to have the potential to encourage informed decision-making with regard to IT investment and use, and recent developments in information technology have intensified efforts to use chargeback as a management tool.
The emergence of an external market for IT services (generally referred to as 'outsourcing') has created an option for organisations who want to focus management attention on strategic priorities or who question the value they receive from an internal IT unit. Despite the difficulty of comparing internal and external prices, IT charges are one of the few means available for assessing the financial impacts of outsourcing IT. Universities operating without a chargeback system, then, lack both an important gauge of IT efficiency and a valuable benchmark for investigating and negotiating outsourcing contracts.
Most universities do, of course, have chargeback for some services, including telephone calls, Internet usage, IT training, and production of videos and other material. These systems appear to be well understood and widely accepted. The introduction of such systems has usually been driven by a desire to manage costs in these areas. Delays in introducing them are typically due to the absence of adequate software or physical infrastructure to support automatic logging of transaction activity. Their existence supports the argument that properly designed and implemented chargeback systems have a place at universities.
Chargeback systems will be particularly important in the increasingly common situation in which a university department accepts commercial work as well as serving the university. Examples of such departments include desktop publishing, audio-visual production and photography, and their numbers are sure to grow as government operating grants are reduced. In the absence of an internal chargeback system, these departments may be inclined to focus on doing 'real' (outside) work, for which they get paid in real dollars, instead of inside work, for which they get paid nothing. An elaborate control system would be required to be certain that an adequate balance was maintained. A chargeback system is probably less expensive and has the additional benefit of increasing the understanding of the full costs of running university departments.
Cost/Benefit Analysis in Teaching and Learning
With regard to the use of information technology for teaching and learning, the universities studied used even less formal methods for reviewing proposed investments, setting priorities and reviewing results. It was difficult to find an example of cost/benefit analysis being applied to a proposed investment in academic IT, although in principle the cost/benefit approach could be used effectively if care were taken to identify all the stakeholders that might be affected. Many projects supporting academic IT are innovative in nature and are therefore treated more like the seeding of research activity. Cost/benefit approaches are beginning to emerge in some areas, especially in the support of commercial multimedia teaching material, or programs for network delivery. Examples can be found at IPACE, an institute of Unisearch, the commercial arm of the University of New South Wales, which has prepared a number of multimedia language products that would not have proceeded without a sound business case.
Costing of IT-based flexible educational initiatives is notoriously difficult due to the inadequacy of existing accounting mechanisms in apportioning costs in the development and support of teaching. An example of some of the issues faced in this area can be observed in those universities collaborating with the Open Learning Australia (OLA) initiative. The universities providing courses to OLA claim they are losing money on the venture. Rather than providing existing units at marginal cost, the universities found that they were in many cases required to collaborate on developing new courses, at substantial cost. If this type of investment is seen as part of a long term investment in developing growth options for the future, then an evaluation of return on investment in the short term may be inappropriate.
With multimedia development involving very intensive up-front investment of resources, costs are prohibitive unless defrayed over large numbers of students. Furthermore, evaluating both the costs and the benefits of a range of IT-based support services that are required in different educational programs becomes problematic if standard 'student contact hours' is the measure used for funding.
In the absence of information about project costs, and in an environment in which the beneficiaries of IT services are not made to pay for them, it can well be asked how IT allocation decisions are made. At the universities studied, the typical situation with regard to administrative computing projects involves the allocation by committee of a fixed budget across competing projects, with choices among projects being made on the basis of logical sequencing, perceived importance, presumed benefit, or other criteria. Allocation of resources for ongoing maintenance of administrative applications tended to follow historical precedents. Typically, resources for academic IT were given out by a committee, which generally neither published its decision criteria in advance nor revealed them after its decisions had been made known. Clearly the selection among proposed projects would appropriately be multi-dimensional, leading to complex and, in part, subjective decisions; unwillingness to admit such subjectivity, or inability to defend it, may be part of the reason for the lack of transparency in the resource allocation process.
Charging for IT-focused Staff Development
There was little evidence that institutions were charging for IT-focused staff development as a matter of principle, although some charging occurs in almost all institutions. Often basic training in the use of software packages is charged for or outsourced. At The University of Melbourne, the Information Technology Services unit, which provides a range of PC package courses to staff, charges for staff who do not take up a reserved place in a course, and for courses considered to be outside the strategic goals of the Service. At the University of New South Wales, some PC package training is provided internally on a user-pays basis, but most staff use commercial services for this type of training. Similar arrangements are in place at the University of South Australia.
Some instances of charging for services provided by central educational technology or teaching and learning units were encountered, but these policies were often implemented in a discretionary manner in order not to stifle local initiatives or impair the quality of products. Examples of partial internal user-pays charging in this area were encountered at several universities.
The issue of charging for internal IT services to support educational developments is posing problems. Universities, particularly those not extensively involved in distance education, are not accustomed to paying for the development of curriculum and teaching materials. These aspects, including quality, are typically seen as the private responsibility of teaching staff. Consistent with this, teaching materials are rarely considered to be university property.
In contrast, the use of IT-based teaching resources and processes demands much higher 'production values' than is the case for lecture notes and paper handouts - more staff and staff time in development, and the use of a range of production services. In some institutions the cost of teaching with information technologies is starting to be recognised, in others it is not. A key element in the process of recognising the true costs of teaching with technology based resources and processes, and therefore of funding it appropriately, is to shift the issue from the private concern of teachers to the concern of the institution. This shift has cultural, industrial relations and legal implications, all of which will have to be addressed.
Controlling Investments in IT-based Teaching
Institutions are increasingly attempting to control and coordinate the development of teaching resources and processes which employ information technology. Andrews and Isaacs (1995) see an element of control as preferable to allowing the proliferation of uncoordinated 'lone ranger' development. Many of the institutions surveyed had implemented measures, or were in the process of doing so, to bring the goals of teaching resource development projects into line with faculty or university goals, and to focus staff development support onto projects which support these broader directions. The Head of one central teaching and learning unit sees the issue of control as a quality issue:
The university is determined that university priorities will override individual and faculty priorities. So while [our] priorities are driven by faculty priorities, the money comes directly to us so the faculties do not get a say in it. Some are worried that if money is devolved to faculties, [our unit] will lose its ability to maintain quality control.
But control and coordination is about managing shrinking budgets as well as maintaining quality, as one Vice-Chancellor of a 'new' university states clearly:
Running with the innovators rather than running with need is a problem. In [one faculty] we have spent huge amounts of money helping somebody produce a very expensive piece of technology for a class of 12 people. That was fine in the early innovation stage, but it has become ludicrous now...
I am getting this message across to people by not funding them basically. We have done a number of things, a series of documents, which are about attempting to control what is going on in the university through the [central teaching and learning unit]. [This unit] is an attempt to say these high cost learning technologies are university business and not some individual's play thing and we are trying to change the culture and bring Deans and Heads into planning subject development from the top down....
The difference between this university and many others is that I believe the quality of teaching and learning is actually university business. It is not individually owned. I want the structures to reflect that.
In a similar attempt to set the direction for development, Deakin University has just transferred $200,000 from a budget for print based materials to a fund for developing World-Wide Web based materials.
At the University of Canberra internal funding tends to go to teams now rather than to individuals, thus controlling 'lone rangers'. These teams are mixed groups of academics and administrative staff, and in this way administrative staff get direct access to University Development Grant funds.
While we have been critical of the lack of systematic evaluation of IT investments, we should emphasise that substantial benefits are being realised. A good example is the implementation of a new administrative system in the Baillieu Library at the University of Melbourne. Here the existing inefficiencies were well documented and costed, and, according to the Librarian, the new system was designed from the beginning to achieve rationalisations in staff roles and work processes:
The system we used to have was not efficient and we knew what that was costing us because of our quality processes. We were spending a quarter of a million dollars annually duplicating Uni processes. We got staff and customers together to develop the specification. We looked at the new system for efficiency (how much duplication we could eliminate and this was costed) and at how easy it was to use because we wanted to free staff up to work on other things.
It took us many hours and many people. In our technical services area (cataloguing, processing etc.) a lot of those operations can now be done in one sequence rather than like an assembly line. We will reduce staff as a result. We did this because we knew we could improve effectiveness first, efficiency second. We were duplicating so much of the University's admin system. The main benefit of the new system is that staff and students don't need training to use it, so it is saving us in training and in being able to redeploy staff. We are putting functional and discipline based staff together into teams.
Another system recently developed and implemented at The University of Melbourne - the student administration system - has, according to the Academic Registrar, brought both cost savings and time efficiencies by eliminating duplication between the central administration and the faculties (see Chapter 2).
In the realm of 'softer' benefits, top administrators at Deakin credit their ability to recast 'distance education' to the judicious use of information technology. The distance learner at Deakin is no longer a lone scholar with a self-contained package of information; he or she is now engaged in a process that combines individual inquiry with discourse with teacher and other learners. Deakin's determination to develop a distinctive style of distance education in which IT is used as a means both of learning (an outgrowth of the early movement to computer-mediated learning) and as a means of communication (a distant descendent of the one-way broadcasts of the School of the Air) had led the university to substantial success in commercial sector (Deakin Australia) as well as in the traditional student market.
Thus we should not confuse the lack of evaluation with a lack of realised benefits. Indeed, a series of relatively small scale IT projects such as these can be very effective in driving performance up and costs down across the institution, while controlling the risk. In doing so, they are essentially re-engineering business processes to strip out inefficiencies and improve customer service.
Resource allocation decisions are an important means of communicating and reinforcing strategy, whether for an organisation as a whole or for information technology specifically. It would appear on the surface that, while some processes are being put in place, the universities studied are not making full use of the opportunity to openly select for funding those IT projects whose proposed outcomes support the strategy of the organisation, and to reward individuals who are able to achieve those outcomes. It is not required that the selection process rely solely on financial outcomes or even on quantifiable factors; as described above, there is scope to include subjective and qualitative factors as well. Regardless, the goal of communicating and reinforcing strategy will best be served by a transparent allocation process that is widely publicised in advance and explicitly used to describe allocation decisions.
At most of the universities studied, members of both academic and general staff, when asked about the evaluation and review of proposed IT investments, commented that an improved process was needed. Many seemed to be relying on the administrative centre of their university to develop, promote and enforce a new approach. Among the reasons cited for the institutions' failure to date to create such a process were:
In the context of most Australian universities, one explanation for the lack of a process for evaluation and review of IT investments could be that there are few proposals for administrative IT projects at any one time, and few resources to apply, thus few real choices to make. An elaborate process may not seem to be required in order to select the projects for a given year. Teaching and learning proposals are generally handled by committees, whose membership may change from year to year and which may have little administrative staff support for the development and enforcement of procedures. However, a more detailed look at the proposed IT investments could have advantages beyond the resource allocation stage. By requiring that a business plan and analysis accompany each proposal for a major investment in IT, either for administrative use or to support teaching and learning, a university could increase its confidence that the proposals are in support of its strategy and, perhaps more importantly, could follow up on the projects selected for funding to see if the promised benefits have in fact been achieved. Without this follow up, the university loses an important opportunity to learn about, and improve, its IT skills.
Similarly, among the historical explanations for the absence of a chargeback system for IT in the university context may be that the amount of money spent on IT was once small relative to the presumed expense of operating such a system, while in a non-profit environment there may appear to be little need for precise allocation of costs of any kind. However, at most universities IT expenses have been rising rapidly and now represent a significant budget item, while a growing cost consciousness, and the opportunity to find alternative suppliers, has led to a desire to understand the costs of running each part of the organisation. Modern chargeback systems are relatively inexpensive to acquire and operate, and chargeback must be seen in a context much broader than simply accounting. A good chargeback system can promote cost control and the effective use of IT resources across the university while motivating and monitoring the performance of the IT unit.
CASMAC and Inter-university Collaboration
For many IT investment decisions in universities, the question is not 'should we invest' but 'where and how to invest'. It is often not a choice of whether to develop, upgrade, or replace a system, but which system will efficiently deliver the required benefits, and how should it be obtained and installed. For such a system, which could be termed an operational or competitive necessity, 'return on investment' is not usually a very relevant measure. Where the system is fundamental to running the organisation and delivering services, there may be no choice but to invest. In many cases, and especially in the public sector, it is difficult to quantify all of the costs and benefits (Stacey 1993). This may cause some universities to neglect the cost-benefit analysis of proposed IT projects. Some of these will be operational or competitive necessities, which will not give you any competitive advantage, but will keep you in the game and thus are necessary for survival.
Many university administrative IT projects would fall into this category. In attempting to address the need for such systems, most universities have collaborated in the CASMAC initiative. Collaborative development has been proposed as a way to share the costs of developing standardised IT systems for areas such as finance, human resources, student administration and management information. Below we describe CASMAC and discuss some of the management issues it raises. We do so because of the scale and scope of the initiative, the apparent delays that have been experienced, and our belief that universities need to pay close attention to managing the costs, benefits and risks associated with such large IT projects.
While we have not conducted a formal review of the collaborative inter-university CASMAC initiative, it was mentioned by a number of interviewees, and we have consulted other reviews (e.g. Australian Vice-Chancellors' Committee September 1996).
A national initiative to assist Australian universities to replace their existing administrative computer systems commenced in 1990, after a group of Australian university administrators became aware of the UK universities' Management and Administrative Computing (MAC) project. As a result of a Department of Employment, Education and Training (DEET) funded feasibility study, it was agreed to proceed with the preparation of a high level specification covering the core functional requirements for management and administrative computing for Australian universities. This specification was referred to as the Core Australian Specification for Management and Administrative Computing and was assigned the acronym 'CASMAC'.
The CASMAC initiative is based on the premise that there is a high degree of commonality in the core functionality needed to support the administrative and management functions of universities, and that there are benefits to be obtained by specifying these requirements and sharing the development and ongoing costs of the resulting systems (Australian Vice-Chancellors' Committee 1996).
It was anticipated that the core specifications would cover up to 80 per cent of an individual university's management and administrative computing requirements. Each university is expected to refine the core specification to meet its own requirements.
The term 'CASMAC' currently refers to three distinct components of the initiative. Again, citing the Australian Vice-Chancellors' Committee report, these three uses of the term refer to:
While the term 'CASMAC' technically refers to only the first of the above components, its use has become synonymous with all aspects of the project. It should, however, be recognised that whereas the Australian Vice-Chancellors' Committee has responsibility for the first component, the responsibility for the other two components lies with the collaborative groups (UniPower, UniOn and Natural) and with individual universities.
(Australian Vice-Chancellors' Committee 1996)
The objectives for CASMAC include cost-effective collaborative development and acquisition of administrative systems to meet the operational and information requirements of the universities, and especially to enable a standardised response to the universities' statutory requirements and external reporting responsibilities to government.
The collaborative focus of CASMAC was therefore intended to provide a framework for institutional development, towards which universities could migrate in the knowledge that systems obtained from other universities, or possibly developed in conjunction with other universities, would have minimum integration problems.
In mid-1993, consortiums of universities were formed to undertake the joint development, acquisition and/or exchange of CASMAC compliant software. The three CASMAC Collaborative Groups are UniPower, UniOn and Natural.
The UniPower Group is a consortium of 19 universities that, together with its commercial partner CHA, is using the PowerHouse software environment in its development and/or acquisition of application software.
The UniOn Group is a consortium of 12 universities that is using the Oracle software environment in its development and/or acquisition of application software.
The Natural Group is a consortium of 2 (originally 3) universities, The Australian National University, and The University of New South Wales (Macquarie University was also initially a member). These universities have a Memorandum of Understanding for the joint development, acquisition and exchange of software based on the Natural fourth generation language.
The four universities that are not members of either the UniPower, UniOn or Natural Groups are The University of Melbourne (which has developed CASMAC-compliant systems in-house using the Ingres relational database management system and development tools), Swinburne University of Technology (which has implemented Aspect's ASCOL software), The Australian Catholic University and Bond University which are evaluating software packages as well as developing systems.
In November 1991, Version 1 of CASMAC was distributed to university representatives on Application Reference Groups for comment. CASMAC Version 2.1 was published in October 1992. The specification contained a Management Summary, as well as high level specifications for Student, Human Resources, Finance, Physical Resources, Research and Consultancy, and Executive Reporting applications.
In April 1994, the CASMAC Steering Committee recommended to the Australian Vice-Chancellors' Committee that the next version of the CASMAC specification (Version 3) include essential functionality identified by the collaborative groups; include new and revised requirements imposed by external agencies; broaden and deepen the specification of the core functional requirements; and include a comprehensive glossary and a data dictionary of data elements required by external agencies.
Universities provided input to the revision process through the Application Specification Review Groups which were established for each application system to review the revisions undertaken by the CASMAC Project Manager. Following a period for comment, Version 3 of the Student System specification is expected to be completed by the first quarter of 1997. Version 3 of the Human Resources specification will be finalised by the second quarter of 1997. No further work has been carried out to prepare Version 3 of the Finance, Physical Resources, and Research and Consultancy specifications. It has been estimated that it will take 4-5 months to complete Version 3 of each of the applications. There is agreement that some of the process (business analysis of the requirements, synthesis of the input from user groups, etc.) should be outsourced to external consultants or individual universities.
The data from the core CASMAC systems will underpin a university's executive information system. UniPower has taken the view that an EIS can be built from sound core databases using data warehousing techniques and the standard inquiry tools which have been provided as part of the agreement with Cognos. UniOn has taken a similar view, but has a slightly more structured approach, involving the purchase of the Holos EIS package and the development of a common approach to the establishment and maintenance of a data warehouse. The Executive Reporting specification will be reviewed after Version 3 of the other core system specifications have been completed. (Australian Vice-Chancellors' Committee 1996).
While the implementation of systems is the responsibility of universities, consortia such as UniOn and UniPower support this process through facilitating sharing of experiences between universities and collaborative development of system enhancements and training materials.
Management Issues in Evaluating Collaborative IT Investments
The CASMAC initiative aims to develop common, and to some extent compatible, administrative systems for universities, which will provide the core systems requirements for university administration as well as standardised reporting for government agencies. While progress is being made, we wish to raise several issues that may have a bearing on the effectiveness of CASMAC. We do so only to flag them as management issues for consideration rather than offering definitive findings or solutions.
The description of CASMAC draws attention to the issues facing universities as they attempt to collaborate in the design and delivery of large scale IT systems within an environment of both resource constraints and government pressures for standardisation. In view of the complexity of the project, and its relevance to nearly all universities, it is worth discussing CASMAC in the light of what we know generally about:
By highlighting these general issues, we hope to flag some areas where university administrators could focus their evaluation of such collaborative projects as CASMAC. Specifically we highlight the problems that can arise for collaborative projects due to differences between organisational configurations, the ownership problem when systems design and specification are divorced from implementation, and the level of technical and business risk associated with IT projects. We examine these management issues below.
Organisational Configurations and Collaborative Systems Development
Recent research into collaborative development of systems between organisations has highlighted the difficulties that can arise from even small differences between organisational configurations. An organisational configuration is the gestalt of strategy, structure, management processes, technology and roles and skills that together form an organisation. All these interdependent organisational elements should be congruent or in fit (see Chapter 1). Changes in one element, say technology, can have consequences for this organisational fit. Sharma and Yetton (1996) have found, for example, that collaborative systems development efforts between area health services often foundered on very minor differences between the management processes and structures of the organisations. These differences meant that adoption of commonly developed designs met resistance within individual organisations if they meant changing established processes and modes of operation, and especially if it meant inter-departmental change was required.
While not insurmountable, such resistance should not just be classed as an 'implementation problem'. Rather, these issues need to be addressed as an integral component of the design phase of the systems. This is difficult if not impossible to achieve within the constraints of a collaborative systems project between organisations because this type of project inevitably seeks solutions that are standardised across organisations. Or, as CASMAC does, by choosing to side-step the issue of inter-organisational differences and leave systems adaptation and implementation up to individual universities, it may also potentially leave the latter with a significant source of problems. Universities, however, can benefit by taking the opportunity to re-engineer their administrative processes and practices in line with best practice, prior to redesigning their systems. Some universities are cooperating in such reviews of best practice and are using these as the basis for redesign efforts.
It would be a mistake, in our opinion, for university executives to assume that CASMAC will solve the problem of administrative systems for universities. It may be that by separating out and focusing on high level design specifications, CASMAC has also to some extent separated ownership of the anticipated positive consequences of the design from the unanticipated (often costly) consequences of implementing the proposed IT changes. We discuss this further below.
Ownership, Accountability and Organisational Integration of IT
One of the sources of difficulty in assessing the cost-benefit of IT is that the costs and the benefits are often 'owned' by different parties. This has two dimensions - horizontal and vertical. Frequently, the IT division wears the costs of an IT project, while those units using the system obtain the benefits. This creates a horizontal asymmetry in the allocation of costs and benefits. Similarly, ownership of costs and benefits can be vertically asymmetric across organisational levels. The predicted benefits are espoused by and associated with high level project sponsors, who are often senior managers and outside consultants. The unanticipated actual costs of systems implementation, however, are borne by the middle and lower managers tasked with putting the system in place (Yetton 1997).
In this case, collaborative IT projects across universities such as those developed through the CASMAC initiative face the additional complexity of the high level specifications being developed independently of each institution's specific and often unique organisational and management structures and processes. In a sense, this exacerbates the separation of ownership of the 'benefits' of the grand plan from the 'costs' of its actual implementation.
It was always acknowledged that universities would have to undertake additional work to customise the specification to reflect any requirements that were more detailed or more institution-specific than the core specification. CASMAC was prepared from a logical viewpoint and was not intended to represent the physical organisation of a university. There was no chronological sequence to any of the processes defined and no attempt to deal with system security and audit requirements or to describe how functions and sub-functions might be implemented.
(Australian Vice-Chancellors' Committee 1996; emphasis added)
In addition, this separation of the project from individual universities seems to have led to a distancing from, and lack of ownership by, university line management, even though the development process involved considerable consultation, and software development teams usually included application specialists from within the universities. The interviewers often heard both university IT managers and senior administrators say they were not focused on the administrative systems of the institution because they were 'waiting for CASMAC'. It did not have senior management attention, although some did express concern about how long it was taking to get UniOn or UniPower systems in place. These delays meant that 'interim systems' were sometimes being developed or purchased by some universities to meet the immediate needs of the institution.
Competition between universities has grown since CASMAC began in 1991, and this is leading to increasing strategic differentiation (as discussed in Chapter 1). Perhaps this changing strategic focus within universities has contributed to the lack of attention to and 'ownership' of the CASMAC initiative by individual universities.
A related issue is that the benefits of IT systems are often obtained from their tight integration with the management processes and operations of the organisation, where IT becomes embedded in and core to the business. Obtaining this will be more difficult with CASMAC. Firstly, there is a major difference between developing IT systems to provide standardised data for external agencies and developing systems which add value to the core operations of the university - especially delivery of education. CASMAC attempts to achieve both of these objectives, and thus faces additional complexity. Secondly, it may be that in the future, administrative systems will need to be seamlessly integrated with these core operations; for example, to enable devolved access by on-line university teachers to all relevant student information pertaining to a particular unit or course, to facilitate enrolment, assessment, teacher-student interaction, and so on. Separate, outside development of common systems is less likely to facilitate such trends.
Alternatively, where the IT systems are not seen as core, then outsourcing is a viable option. Does CASMAC add any value for an institution beyond that of outsourcing its standardised administrative IT functions? And does CASMAC have a higher or lower risk than the outsourcing option? It is true that universities participating in CASMAC have obtained access to DEETYA resources that otherwise may not have been available, although this has been relatively small when compared to the overall cost of the project. It may, however, partly explain why the degree of management scrutiny of CASMAC projects within universities, seems to be less than that typically found when negotiating and managing outsourcing contracts. This has the potential to increase risk. Before further considering the IT outsourcing question, it is worth briefly reviewing what we know about managing risk in IT projects.
Managing Risk in Large Scale Projects
Put very simply, the success of an IT investment has three key related components - the costs, the benefits, and the risks. These include not only technical but also business and organisational costs, benefits and risks. While these factors should be subject to evaluation before, during and after the project, in practice most evaluation of IT investments in universities and elsewhere occurs prior to the decision to invest, and focuses on an analysis of the estimated costs and benefits of the proposed IT project.
What tends to be neglected or underestimated in these analyses are the technical and business risks associated with a project. This is in spite of regular reports of large scale IT project failures - for example, London Stock Exchange's TAURUS project, American Airlines' CONFIRM system, Westpac's CS90 system - which together have lost hundreds of millions of dollars. Despite these examples, the risk of project failure, and the implications of this failure for organisational performance, are often not adequately assessed by senior managers.
IT project failure is not limited to large headline grabbing projects. Recent research which surveyed over 8000 IT projects across a large range of organisations in the USA found that 31 per cent of all projects were abandoned before completion. Furthermore another 53 per cent had run over budget, over time, and/or had impaired functionality. Only 16 per cent of projects were completed on time, to budget and delivered what was expected. Furthermore, average cost overrun across all projects was 75 per cent, there was an average schedule overrun of 102 per cent, while on average, projects delivered only 67 per cent of specified functionality (Johnson 1995, cited by Sauer 1996). While we have seen no comparable Australian evidence, we have no reason to expect that the figures would be very different here. These sobering statistics should be enough to flag to senior management of universities that the evaluation and management of risk in IT investments is a subject requiring serious attention.
In a sense, these figures tell only part of the story. It is not just the direct investment of resources in an IT project that is at risk. Of potentially even greater importance are the risks to the business itself of IT failure. For example, software problems with the sophisticated automated baggage handling system at the new Denver International Airport were responsible for delaying the opening of the airport for 16 months. The interest costs associated with this delay exceeded $500 million. When the French railway SNCF developed a new ticketing system, the problems associated with its introduction caused a significant and lengthy reduction in sales. In another example, problems with a new computer system at British Gas recently led to a doubling of customer complaints, large scale recruitment of staff to manage the increased workload, and loss of revenue because of problems with billing (Sauer 1996). Thus IT projects can generate significant business risk. In universities, where IT is becoming more critical to both the delivery and administration of educational services, management of these risks will become increasingly important. Senior managers therefore need to ask 'what is the risk to our ability to effectively function if these current or planned IT systems fail?'. Recent problems with the UAC (Universities Admission Centre) voice-activated system designed to streamline registration of school leavers' university preferences, is a more simple yet typical example of the disruption to basic operations that can occur when a new system fails to meet expectations.
Although the above stories and statistics might reinforce the tendency of those who are sceptical of the value of IT to oppose any IT investments as too risky, this is not our intention. Indeed, our point is that because IT is important, we need to pay a lot of attention to managing its costs and risks in order to realise its benefits. A number of universities are recognising this and appointing their IT general manager or CIO to the senior executive group. While risk is not easily controllable because uncertainty of outcomes will continue to be present, efforts at risk containment can be made. These might include selective outsourcing, purchasing off the shelf systems, and breaking developments into smaller, more manageable chunks. Below we briefly discuss one of these options-selective outsourcing.
For a number of years, organisations have selectively used IT outsourcing or contracting services to meet variability in workload requirements, or to undertake major systems developments. Industry has also used outsourcing to mitigate technical risk and uncertainty, as well as to access emerging technology. Outsourcing presents an opportunity to access resources outside the organisation. This is achieved either by using a vendor to provide a discrete service, taking over work previously done in-house, or by combining resources from one's own organisation with the vendor's resources to add value to, or to fast track, a project.
Selective outsourcing enables the ability to flexibly respond to changing requirements, especially for different and scarce skill sets. In particular, it avoids being locked in to long term and inflexible staffing arrangements. The ability to engage in strictly defined performance contracts on projects can also reduce risk and costs.
Many public sector organisations in Australia have aggressively pursued outsourcing, most notably in the whole-of-government agreement between South Australia and Electronic Data Systems (EDS). Universities, however, appear to have used outsourcing only sparingly. As an example, the University of New South Wales has outsourced its computer shop, software development for financial and human resources systems projects, and some of its audio-visual services. This University, as well as the University of Melbourne, have also outsourced cabling and hardware maintenance in its Communications Unit, as well as software development for several communications systems. We believe that there may be substantial opportunities to reduce expenses and improve service, particularly in the administrative area, through selective outsourcing of IT activities currently being performed by internal university departments.
University policies requiring a formal evaluation of each major administrative IT project before it starts, and carrying out a follow-up evaluation after it ends, could bring benefits beyond initial resource allocation. The preparation of a business plan at the outset could require a more careful and widespread analysis of both the costs and the impacts of a proposed project, and an evaluation afterwards could improve a university's understanding of the effectiveness and efficiency of its approach to obtaining and implementing information systems. Furthermore, once an IT project is approved, adherence to and support of formal project management methodologies will assist in reducing risk.
Unlike administrative computing, universities appear to have many possible academic IT projects contending for funds, so a more formal approach to evaluation could in fact improve the ratio of benefits achieved to costs expended. In addition, evaluation criteria announced in advance, applied consistently to select among the proposed investments, and then used after the fact to review the results of those investments, could direct attention towards desired outcomes and improve the credibility of the entire process. For example, it would be important to evaluate investments in academic IT for their pedagogical value and evidence of an improved learning outcome. This is increasingly a focus of university Teaching and Learning Committees.
It is therefore recommended that universities should explore possible frameworks for IT investment evaluation, adopt or develop a consistent approach, possibly involving multiple criteria, towards such evaluation, and apply this approach to proposed IT investments for administration, teaching, and learning. To the extent that universities are unaware of the frameworks already in use, DEETYA could facilitate the improvement of IT evaluation practice by sponsoring a series of workshops to expose, explore, and discuss the possibilities. DEETYA could also provide financial rewards and other incentives for improved evaluation performance.
Universities could create an action plan to develop and implement chargeback systems to move them towards higher levels of accountability and control of their internal IT resources, while providing the basis for consideration of obtaining some or all of those resources from outsourcers. It is recommended that DEETYA facilitate the creation of such plans by sponsoring, either separately or as part of the workshops described above, workshops on the benefits, uses, and alternative forms of IT chargeback.
Finally, individual universities should continue to closely manage and evaluate their investment in the CASMAC initiative, because of the costs and risks typically associated with implementing large collaborative IT projects. Furthermore, it is strongly recommended that universities evaluate the option of selective outsourcing of IT services.