courses | hecs/oldps | higher education home | higher education links
publications | issues | research  | scholarships | statistics | universities

Validating Scholarship on University Teaching:

Constructing a national scheme for external peer review of ICT-based teaching and learning resources

Executive Summary

• The conduct of the project
• Issues addressed
  • ICT-based teaching and learning resources
  • The ‘scholarship-in-teaching’
  • Peer review
  • The conduct of reviews
  • A basis for a trial of the scheme
  • Institutionalisation of the scheme

There is a ‘window of opportunity’ for the establishment of a scheme such as that discussed in this report. Australian universities, like their counterparts in most OECD countries, are rapidly expanding their use of ICTs. The focus of the current project is on the pedagogical applications of these technologies, specifically the development of ICT-based teaching and learning resources. The need for such a scheme is evident and known. And this project has developed strategies and resources that provide a basis for its trial.

The aim of this project was to develop conceptual and procedural bases for a national scheme for independent and expert peer-review of ICT-based teaching resources. It is the first stage of a larger three-stage project born out of:

• a concern that teaching resources based on the use of ICTs were being developed without any external quality verification and feedback process;

• a need for ICT-based resource developers to be recognised and rewarded for their achievements;

• a need for the collaboration and dissemination of knowledge regarding the design, nature and availability of ICT-based resources; and

• a need for the work of academics in this area to be recognised as scholarly.

The intentions of the scheme are:

• to provide a basis for encouraging staff to engage in the adaptation and creation of ICT-based teaching resources;

• to allow for evaluation of the quality of those resources;

• to provide a basis for recognising and rewarding the achievements of those involved in the development of high quality resources;

• to encourage the use of existing resources through provision of readily accessible and credible information on their features and value;

• to contribute to the development of a culture which recognises and values the ‘scholarship of teaching’ that informs the design and use of ICT-based resources;

• to contribute to the development of a culture of research into the design and use of ICT-based resources, through the development of shared language and a critical attitude to their design and use; and

• to contribute to the improved use of existing resources through a process of adaptation and re-use by developing strategies for sharing the outcomes of those reviews, and through making resources more readily discoverable by using metadata standards to describe them.

top

The conduct of the project

The Project was completed in a little more than three months—July to September, 2000. Information relevant to conceptual and procedural bases for the scheme was gathered through three primary strategies:

• a literature review;

• invitations to members of the national and international External Reference Group to comment on particular issues; and

• the conduct of a Project Symposium attended by fifty Australian peers who commented on initial drafts of the core documents underlying the proposed scheme.

The Project Symposium was conducted on 22 August 2000. Prior to that event four documents, developed as a result of the first two strategies, were forwarded to participants: a conceptual overview of relevant issues; a Review Proforma; an explanation of the categories listed on that proforma; and a paper on issues related to the procedures for a review process. Those documents became the focus of discussion during the Symposium, and were re-drafted on the basis of that feedback. Additional work was also undertaken to develop other chapters included in this report, particularly in relation to institutional issues.

top

Issues addressed

ICT-based teaching and learning resources

These resources are artefacts. Their material nature makes external peer review possible. Yet they tend not to be neatly bounded artefacts. They can draw on pre-existing learning objects, and be quite distributed in nature. Thus the scheme will allow for peer review of resources that vary in their scope and nature, from relatively discrete e-simulations to entire online degree programs.

The ‘scholarship-in-teaching’

The scheme depends upon the development of an explicit understanding of the ‘scholarship of teaching’ involved in the development of ICT-based teaching resources. Our sense is that most examples of the scholarship of teaching are really discussions about teaching and learning processes, mostly in the form of conference and journal papers. We refer to this publication-focused aspect of the scholarship of teaching as ‘scholarship-about-teaching’. Our interest in this work is with those aspects of the scholarship of teaching that directly inform the decisions teachers make. That is, we are interested in those aspects of the scholarship of teaching that both inform the design of pedagogical practices and are evident in resources constructed on the basis of that design. We refer to these aspects as ‘scholarship-in-teaching’. The key elements of this scholarship shape the criteria for review included in the Review Proforma.

top

Peer review

Peer review has been a central part of the quality assessment and decision-making processes in higher education for many years. While most academics express few concerns about peer scrutiny of research activities, they tend to be skeptical of any process of peer review involving teaching. The pressure for peer review of teaching and learning reflects an emerging reality—the activities of teaching and learning are now claimed as public territory. The development of ICT-based teaching resources offers an opportunity for academic peers to validate the consistency between their work and the scholarship of teaching. Through the implementation of this scheme academics will be supported in the shared development and use of a scholarship-in-teaching focused on the design, construction and use of ICT-based teaching resources. The intention is to promote this scholarship, and to add a significant element of credibility to claims about the value of those resources. In turn, this will assist those involved in their creation to seek appropriate recognition and reward for their contributions.

top

The conduct of reviews

The report suggests that the process of peer review should be controlled by the academic community, through the central role of journal Editors in the management of the review process. This possibility was explored at the Symposium, and strongly endorsed by participants.

Peer review of ICT-based teaching resources (and supporting documentation) of the type suggested in this Project is novel. For this reason we have provided a relatively detailed set of recommendations as a means to inform the work of the trial stage of this overall project. We have also outlined the options available to participants in the review process and provided commented on the advantages and disadvantages of each of these.

Recommendations for the conduct of the review process are:

1.1 Control of the review process:

• the review process be controlled and administered by journal and conference proceedings editors;

• those editors advertise the scheme and their journal-specific review protocols within their journals;

• the scheme be advertised more widely through use of the commercial media and relevant email lists, particularly lists likely to involve potential users of the scheme;

• professional development strategies and resources be prepared to introduce the scheme to editors, and to clearly explain its operation, identifying issues that are non-negotiable and those issues that will need to be negotiated with other stakeholders; and

• these strategies and resources be reviewed during the trial, with a particular concern to ensure that they will effectively introduce the scheme to editors who have had no prior contact with it.

1.2 Support for creators of resources:

• professional development strategies and resource be prepared to introduce the scheme to resource creators, and to clearly explain its operation, identifying issues that are non-negotiable, issues that are journal-specific, and issues that will be negotiated with a journal Editor;

• these resources be prepared in ways that allow for their use via a Web-site or as part of a workshop activity; and

• these strategies and resources be reviewed to improve their effectiveness during the trial.

1.3 Support for reviewers:

• professional development strategies and resources be developed to introduce potential reviewers to the purpose and procedures of the scheme, and to provide initial training in the team-based conduct of reviews following the Review Proforma;

• these strategies and resources be developed as an urgent priority during that trial; and

• these strategies and resources be reviewed to improve their effectiveness during the trial.

1.4 Access to resources and accompanying documentation:

• the creator/s exercise, in collaboration with the relevant Editor/s, control of access to their resources during any review process.

1.5 Publication of information about resources that have been reviewed:

• only information on resources that have been ‘recommended’ should be made public;

• the author/s of the resources should provide a statement on the intentions, nature and limitations of the resources, including their availability and compliance with relevant metadata standards, for publication;

• the Editor of the journal should provide a statement on the strengths and limitations of the resource, based on the feedback provided by the review team;

• the author/s should have an opportunity to suggest changes in that statement before it is made public; and

• the author/s and Editor should indicate in any publication concerning the resource that the ‘recommendation’ of the resource is a qualified recommendation, based on the intentions, nature and limitations of the resources provided by the authors prior to their review.

1.6 The conduct of the review process:

• options for the conduct of a review, including the use of collaborating panels of peers, be explored in any trial of this scheme; and

• editors be invited to nominate their preferred strategy, and collaboration with potential users and reviewers be negotiated in terms of those preferences.

1.7 Providing feedback to the creator/s of resources:

• options for providing feedback to the creators of resources under review, including the use of closed e-forums, be explored in any trial of this scheme; and

• editors be invited to nominate their preferred strategy, and collaboration with potential users and reviewers be negotiated in terms of those preferences.

top

A basis for a trial of the scheme

In order for the scheme to be trialed some basic components need to be provided. This work is provided in the form of two documents: the Review Proforma and Interpreting the Review Proforma. The Review Proforma is a two-page checklist and comment sheet intended to guide the resource review and facilitate the provision of feedback to the Editor and/or creator/s. The first page provides all the criteria for the review, as well as tick boxes for ratings of each criterion and/or sub-criterion. The second page offers room for the reviewer to comment on the resource, and a set of choices to make in terms of their final recommendation.

The Interpreting the Review Proforma is a four-page statement designed to assist those who use the Review Proforma. It provides a succinct yet informative interpretation of the overall layout and content of the Review Proforma. That ‘layout’ involves two sets of criteria. The first, design considerations, includes five major foci for review: student needs; relevant theory and research; the learning context; the resources available to support the construction of this resource; and, evaluation strategies and intentions. The second set, addressing the construction of the resource, includes: the approach to learning opportunities; the approach to assessment and evaluation; the use of ICTs; and the overall consistency and coherence of the construction.

It is expected that a reviewer will take into account all available information when reviewing the resource, and deciding whether to ‘recommend’ it to peers. Thus, the review will be informed by three documents: an overview of the resource; a statement on the design consideration; and the resource creator/s reflections on the resource.

top

Institutionalisation of the scheme

The report addresses the policies and practices necessary to ensure that the scheme is fully integrated with the broader policies and practices of the higher education sector and its institutions. This reflects a concern to ensure that the scheme generates sufficient benefit for all involved to maintain their commitment to it, and provide the resources necessary for its continued evolution. This is essential in a system characterised more by change than by continuity.

The following recommendations are proposed for the continuation of this work:

Stage 1

1. That the outcomes of Stage 1 be widely disseminated by DETYA to the stakeholder groups identified in Chapter 8.

2. That opportunities to discuss the report of Stage 1 and its outcomes with those stakeholder groups be pursued by all members of the Core Team and External Advisory Group.

Trial phase

3. That trial implementation of the scheme be undertaken with the financial support of DETYA and under the direction of the Core Team involved in this project, with that trial involving:

• use of the Review Proforma and its explanation provided in Chapter 7;

• the three disciplines of business management, engineering and nursing and related journals; and,

• three universities, to include Griffith University and two other universities to be selected on the basis of their willingness to be involved and the nature of their process for academic review and promotion.

4. That these trials be widely advertised by the Core Team.

5. That the trial be supported through the development by the Core Team of staff development strategies and resources for those involved in the implementation process involving both journal reviews and academic review and promotion, as discussed in Chapter 6.

6. That the trial actively compare the alternative strategies for conducting the review process and sharing its outcomes discussed in Chapter 6.

7. That the trial focus on and develop strategies to ensure that appropriate metadata standards and usage are agreed and implemented for all resources reviewed, in collaboration with EdNA and other interested groups.

8. That the trial be conducted over at least a 12-month period to ensure that a full cycle of institutional procedures is able to engage with its use.

9. That the scheme be actively monitored by the Core Team throughout this period, and agreed improvements be continuously incorporated.

10. That a report on the trial be prepared by the Core Team and widely circulated by DETYA to the stakeholder groups identified in Chapter 8.

11. That opportunities to disseminate the intentions, practices and outcomes of this phase be actively pursued by all members of the Core Team and External Advisory Group throughout its conduct.

12. That opportunities for collaboration with additional discipline/journals, and/or Australian universities, and/or countries be encouraged yet managed so that this trial is completed and the further development and expansion of the scheme supported.

Full Report

Return to the Top of the Page

contact details  |  search  |  archive search  |  publications  |  site map  | subscribe
career information | education network australia (EdNA)  
australian education international | prime minister's web site 

Any comments or queries should be sent to: wwweditor@dest.gov.au

This page was last updated on Tuesday, 26 August 2008
Department of Education, Science and Training
Copyright © Commonwealth of Australia
DETST Web Site Privacy Statement
Disclaimer