rss
J Am Med Inform Assoc 12:296-298 doi:10.1197/jamia.M1733
  • The Practice of Informatics
  • Technical Brief

Agreement, the F-Measure, and Reliability in Information Retrieval

  1. George Hripcsak,
  2. Adam S Rothschild
  1. Affiliation of the authors: Department of Biomedical Informatics, Columbia University, New York, NY
  1. Correspondence and reprints: George Hripcsak, MD, MS, Department of Medical Informatics, Columbia University, 622 West 168th Street, VC5, New York, NY 10032; e-mail: <hripcsak{at}columbia.edu>
  • Received 3 November 2004
  • Accepted 4 January 2005

Abstract

Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the κ statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that κ approaches these measures as the number of negative cases grows large. Positive specific agreement—or the equivalent F-measure—may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.

Footnotes

  • This work was funded by National Library of Medicine grants R01 LM06910 “Discovering and Applying Knowledge in Clinical Databases” and N01 LM07079 training grant.

Free Sample

This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JAMIA.
View free sample issue >>

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Open Access fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.

Navigate This Article