rss
JAMIA 2002;9:1-15 doi:10.1136/jamia.2002.0090001
  • The Practice of Informatics
  • Review Paper

Reference Standards, Judges, and Comparison Subjects

Roles for Experts in Evaluating System Performance

  1. George Hripcsak,
  2. Adam Wilcox
  1. Affiliation of the authors: Columbia University, New York, New York
  1. Correspondence and reprint requests: George Hripcsak, MD, MS, Department of Medical Informatics, Columbia University, 622 West 168th Street, VC5, New York, NY 10032; e-mail: <hripcsak{at}columbia.edu>
  • Received 24 May 2001
  • Accepted 9 October 2001

Abstract

Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.

Footnotes

  • This work was supported by grants R01 LM06910 and R01 LM06274 from the National Library of Medicine and grant NY01-002153889 from Pfizer, Inc.

  • ↵* See Willems .,4 Figure 3.

This Article

  1. Abstract
  2. Full text
  3. PDF

Responses

  1. Submit a response
  2. No responses published

Access policy for JAMIA

All content published in JAMIA is deposited with PubMedCentral by the publisher but with varying embargo times. Authors/funders may pay an Unlocked fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication. Research funded by government and other recognised agencies is deposited with a 12 month embargo. All other content is deposited with a 36 month embargo.

The Journal of the American Medical Informatics Association is published for the American Medical Informatics Association by BMJ Publishing Group Ltd.