Agreement, the F-Measure, and Reliability in Information Retrieval
- Correspondence and reprints: George Hripcsak, MD, MS, Department of Medical Informatics, Columbia University, 622 West 168th Street, VC5, New York, NY 10032; e-mail: <hripcsak{at}columbia.edu>
- Received 3 November 2004
- Accepted 4 January 2005
Abstract
Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the κ statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that κ approaches these measures as the number of negative cases grows large. Positive specific agreement—or the equivalent F-measure—may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.
Footnotes
-
This work was funded by National Library of Medicine grants R01 LM06910 “Discovering and Applying Knowledge in Clinical Databases” and N01 LM07079 training grant.
