Benchmarking
A manual for Australian universities
K R McKinnon
S H Walker
D Davis
February 2000
Higher Education Division
___________________________________________________________
© Commonwealth of Australia 1999
ISBN
0 642 23971 1
ISBN
0 642 23972 X
(Online version at www.detya.gov.au/highered/)
DETYA No. 6441HERC00A
This work is copyright. Apart from any use as
permitted under the Copyright Act
1968, no part may be reproduced by any process without permission
from AusInfo. Requests and inquiries concerning reproduction and rights
should be addressed to the Manager, Legislative Services, AusInfo, GPO
Box 84, Canberra ACT 2601.
The views expressed in this report do not necessarily
reflect the views of the Department of Education, Training and Youth
Affairs.
___________________________________________________________
What is the purpose of this Manual? Who is it for?
It has three potential uses. It provides senior staff with tools to
ascertain performance trends in the university and to initiate
continuous self-improvement activities. Second, it is sufficiently well
developed for use by groups of universities wishing to compare
performance on all or some of the areas covered. Third, some of the
benchmarks can be used by universities now to ascertain their
competitive position relative to others.
The objective of the project has been to identify the most important
aspects of contemporary university life in changing times and to find
ways of benchmarking them. It assumes excellence and value adding are
goals sought by all universities. It assumes also that those aspects of
excellence and value adding that can be easily quantified are not the
only ones worth taking into account; the drivers of future performance
are often qualitative.
The sixty-seven benchmarks included here may still be thought too
many for regular use in all universities. Chapter 12 addresses the
difficulties and possibilities of using a smaller core sub-set.
No single university, however large, can encompass all knowledge.
Every university has to make choices. It is demanding to be world class
in even a few academic fields.
Each university has to prioritise the use of its resources and use
them to best effect. Knowing whether it is succeeding in its aims is
another more demanding level of difficulty.
The key consequential question is how university leaders will know
where their institutions stand and how they can be improved.
The quality of universities cannot be ascertained by the bottom line
measures that apply to commercial firms, or even by the yardsticks that
might apply in large governmental organisations.
How do they differ?
The term, university,
embraces a broad collection of institutions around the world. In
Australia the common features are that they are all internally
self-governing, including establishing the standards of graduating
students.
They are all non-profit institutions, governed by state or federal
legislation. They all teach at the tertiary level, undertake at least
some research, and provide services to the community. Within that broad
commonality, however, constrained by their history and funding sources,
they have adopted widely diversified missions.
They do not seek simply to communicate a standard body of knowledge
in each course. The knowledge conveyed in courses at the tertiary level
is of a different type. The courses and the teaching are informed by the
latest research, which often challenges or even overturns previously
accepted knowledge. The ‘knowledge’ is leading edge—provisional.
Students are expected to learn to create their own intellectual maps and
to search out information for themselves.
They enrol as students and staff seek employment in universities for
a variety of reasons, including intellectual curiosity and a search for
truth. Some students, but certainly not all, seek a vocational
qualification as a primary objective.
University commitment to research varies. Some are research
intensive: others are not. Some emphasise individual research activity
while others seek to create teams in pursuit of pre-eminence in a
limited number of research fields. Some concentrate on basic research
through maximising Australian Research Council and National Health and
Medical Research Council grants, while others concentrate on applied
research and source most of their research funds from industry.
Some management thinking urges on universities the desirability of
commercial approaches to management. That type of thinking suggests that
a first step should be the declaration by the university governing body
of a clear mission and strategic directions and its imposition on the
university. The processes within the university would then be confined
to working out the programme details to realise the mission.
It would be grossly over-optimistic to think such a process would
work. Furthermore the suggestion misunderstands the nature of a
university. Most academics regard themselves as partners rather than
employees. They hold fiercely to their independence and right to be
involved in major decisions. They believe in collegiality; they can be
led but not coerced. Good university leadership, therefore, has to bring
together the aspirations and thinking of the whole university community.
It is in the nature of the quest for new knowledge that no external
authority can say with certainty that the approaches chosen by
particular individuals or a particular university are irrefutably wrong.
In most aspects universities are thus to be measured by criteria
other than profit or return on assets.
Nevertheless, benchmarking is as essential in universities as it is
in other spheres. They need reference points for good practice and for
ways of improving their functioning.
Australian universities have not been at all keen on the university
league tables purporting to show rankings of excellence published by
media, such as those published by The
Times in the UK, or Asiaweek
in the Asia-Pacific area (Appendix 1 lists the methodology used). They
feel that the diversity of missions and their different locations and
ages make comparisons and league tables invidious and unfair.
While accepting that there is a good basis for those objections,
equally invidious is the counter-approach of self-identified sub-groups
of universities directly or indirectly claiming merit by association.
This Manual is based on the belief that the best way to counter poorly
informed assessments of quality of the kind that come with either
approach is to identify measures that will give a valid and balanced
picture of the parameters that distinguish good universities.
The notion of balance in the phrase ‘valid and balanced’ requires
elaboration. Quality, for instance, is not a static, uni-dimensional
phenomenon. Reputations lag. There are always universities living on
past glories unsupported by current performance, and universities,
particularly young universities, whose performance is well ahead of
their current standing.
They are, moreover, complex institutions. To keep relevant they must
respond successfully to the massive changes now challenging them.
Benchmarking thus needs not only to identify successes to date but also
vital signs of adaptation to the future. A university’s dynamism is as
important as its current achievements, indeed probably a better guide to
its future performance. The best universities are those that combine
high achievements with extensive evidence of dynamism and rapid rates of
adaptation to new challenges.
But how are the latter features best measured? If it is true that an
institution cannot be sure that it is changing in particular dimensions
unless it can measure that change, identification of appropriate
performance measures (metrics) becomes of crucial importance.
All too often outputs (or outcomes) measuring the success of past
activities have been the only performance measures used. While such lagging
indicators provide useful information there is also a need for leading
indicators, that is, measures of the drivers of future performance, and learning
indicators, measures of the rate of change of performance. There are
valid ways of measuring dynamism and innovation. As change must be in
particular directions if it is to be effective, there needs to be direct
links between all performance measures and the strategic plan of the
organisation.
A conceptually relevant approach incorporating the above thinking
that has been found effective in commercial and industrial companies is
the Balanced Scorecard
approach of Kaplan and Norton
and the consulting firm of Arthur D. Little Inc. This approach can be
used either in the translation of vision and strategy into key
objectives and associated benchmarks, or, alternatively, in monitoring
how well particular performance outcomes, performance drivers, and rates
of change are helping the organisation to achieve its strategic
objectives.
The fully developed matrix of benchmarks is intended to provide
executive staff with comparative data of past success, the information
needed for improvement, and a realistic appreciation of how well the
organisation is moving towards its goals. In the process it should also
help clarify distinctions between what are simply measurable outputs and
important outcomes. A diagrammatic representation of the matrix of
measurements used in the balanced scorecard approach is shown below.
|
|
Lagging
(outcomes)
|
Leading
(performance
drivers)
|
Learning
(rate of change)
|
|
Financial
|
|
|
|
|
Customer/Student
|
|
|
|
|
Internal Process
|
|
|
|
|
People/Culture
|
|
|
|
This approach applied to universities, for the financial
perspective will ask what financial outcomes are required to achieve
success. As there are several stakeholders; funding bodies, the public,
and the university community, success will be measured via several
financial aspects. The customer perspective will consider how the organisation should
appear to the customers (students) to be considered successful. The
customer perspective is primarily of concern to students and those who
help them make decisions. The internal
business process perspective will look at what products and services
need to be developed and delivered to achieve the customer goals. In
universities this perspective will ask for course development processes
and benchmarks that demonstrate efficiency and excellence. The people/culture
perspective will address the nature of the organisational culture
required to deliver the products and services of the university.
For greatest effect in managing performance, a suite of cross
supporting metrics directly linked to the strategy of the university,
addressing past performance, the drivers of the future and rates of
improvement will be required. The optimal suite will have aspects unique
to a particular university, given the diversity of strategies, state of
development and mix of activities that characterise Australian
institutions. Particular benchmarks can certainly be used to benchmark
the performance of universities with other universities. Indeed this is
a primary purpose of this manual. While such comparisons are part of any
good internal management system, this Version 2 Manual is not at the
stage where it could be used to derive a ‘league table’ or in the
allocation of resources across the sector.
Clearly a balanced approach is needed in the benchmarking of
universities, which is the reason is permeates subsequent chapters. The
focus it puts on the signs of dynamism and rates of adaptation is
important. These features are of special importance in all universities
in the current period of rapid change, especially because of the long
lead times for the production of some types of graduates and some types
of research outcomes.
The balanced scorecard is not, however, a complete answer. One
shortcoming is that the matrix illustrated above relates to profit
making enterprises emphasising profit, whereas universities have
multiple and often ambiguous goals, differentially valued by the many
constituencies within them. That complexity has to be kept in mind. It
is not possible or desirable to fit every desirable benchmark uniquely
within each cell of the matrix.
Chapter 12 addresses in more detail possible vital signs of the
health of universities. It explores both the fit between the benchmarks
in the Manual and the Balanced Scorecard concept, and the question of
whether universities could sensibly use a smaller core set of the
benchmarks for rapid checks of the vital signs.
In summary, this Version 2 Manual makes a considerable effort to
incorporate benchmarking metrics using the lagging, leading and learning
framework. From the benchmarks included here a suite of benchmarks can
be identified that will both suit the circumstances of any particular
university, and allow it to assess objectively its achievements and its
progress.
