Developing and Evaluating Criteria to Help Reviewers of Biomedical Informatics Manuscripts
- Elske Ammenwerth, PhD,
- Astrid C Wolff, PhD,
- Petra Knaup, PhD,
- Hanno Ulmer, PhD,
- Stefan Skonetzki, MSc,
- Jan H van Bemmel, PhD,
- Alexa T McCray, PhD,
- Reinhold Haux, PhD,
- Casimir Kulikowski, PhD
- Affiliations of the authors: Research Group Assessment of Health Information Systems, University for Health Informatics and Technology Tyrol (UMIT), Innsbruck, Austria (EA); Department of Medical Informatics, University of Heidelberg, Germany (ACW, PK, SS); Institute for Biostatistics and Documentation, University of Innsbruck, Austria (HU); Erasmus University, Rotterdam, The Netherlands (JHV); National Library of Medicine, Bethesda, Maryland (ATM); Institute for Health Information Systems, University for Health Informatics and Technology Tyrol, Innsbruck (UMIT), Austria (RH); Department of Computer Science, Rutgers, The State University of New Jersey, New Brunswick, New Jersey (CK), USA
- Correspondence and reprints: Asst.-Prof. Dr. Elske Ammenwerth, Research Group Assessment of Health Information Systems, University for Health Informatics and Technology Tyrol (UMIT), Innrain 98, 6020 Innsbruck, Austria; e-mail: <elske.ammenwerth{at}umit.at>.
- Received 21 November 2001
- Accepted 6 May 2003
Abstract
Peer-reviewed publication of scientific research results represents the most important means of their communication. The authors have annually reviewed a large heterogeneous set of papers to produce the International Medical Informatics Association (IMIA) Yearbook of Medical Informatics. To support an objective and high-quality review process, the authors attempted to provide reviewers with a set of refined quality criteria, comprised of 80 general criteria and an additional 60 criteria for specific types of manuscripts. Authors conducted a randomized controlled trial, with 18 reviewers, to evaluate application of the refined criteria on review outcomes. Whereas the trial found that reviewers applying the criteria graded papers more strictly (lower overall scores), and that junior reviewers appreciated the availability of the criteria, there was no overall change in the interrater variability in reviewing the manuscripts. The authors describe their experience as a “case report” and provide a reference to the refined quality review criteria without claiming that the criteria represent a validated instrument for quantitative quality measurement.
Footnotes
-
The authors are Editors and Managing Editors of the IMIA Yearbook 2001. The authors thank Martina Hutter, Steven Huesing, Thomas Kleinöder, and the Schattauer Publishing Company for their support in publishing the IMIA Yearbook. They also want to thank the fellow Managing Editors A. Bohne, K. Ganser, C. Maier, A. Michel, V. Mludek, and R. Singer for their discussions and comments on this paper as well as the 16 reviewers (S. Abel, B. Baumgarten, A. Bess, B. Brigl, T. Bürkle, E. Finkeissen, S. Garde, E. Lang, F. Leiner, F. Phillip, J. Pilz, U. Prokosch, M. Schwabedissen, R. Weber, T. Wendt, T. Wetter) for their participation in the study. Thanks also to Frieda Kaiser for her support and the anonymous reviewers for their fruitful comments on an earlier version of this paper.
