rss
JAMIA 2006;13:488-496 doi:10.1197/jamia.M2082
  • Focus on Automated Categorization Technique
  • Model Formulation

Advancing Biomedical Image Retrieval: Development and Analysis of a Test Collection

  1. William R Hersh,
  2. Henning Müller,
  3. Jeffery R Jensen,
  4. Jianji Yang,
  5. Paul N Gorman,
  6. Patrick Ruch
  1. Affiliations of the authors: Department of Medical Informatics & Clinical Epidemiology (WRH, JRJ, JY, PNG), Oregon Health & Science University, Portland, OR; Medical Informatics Service (HM, PR), University & Hospitals of Geneva, Geneva, Switzerland
  1. Correspondence and reprints: William R. Hersh, MD, Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd., BICC, Portland, OR 97239: e-mail: <hersh{at}ohsu.edu>
  • Received 12 February 2006
  • Accepted 12 June 2006

Abstract

Objective Develop and analyze results from an image retrieval test collection.

Methods After participating research groups obtained and assessed results from their systems in the image retrieval task of Cross-Language Evaluation Forum, we assessed the results for common themes and trends. In addition to overall performance, results were analyzed on the basis of topic categories (those most amenable to visual, textual, or mixed approaches) and run categories (those employing queries entered by automated or manual means as well as those using visual, textual, or mixed indexing and retrieval methods). We also assessed results on the different topics and compared the impact of duplicate relevance judgments.

Results A total of 13 research groups participated. Analysis was limited to the best run submitted by each group in each run category. The best results were obtained by systems that combined visual and textual methods. There was substantial variation in performance across topics. Systems employing textual methods were more resilient to visually oriented topics than those using visual methods were to textually oriented topics. The primary performance measure of mean average precision (MAP) was not necessarily associated with other measures, including those possibly more pertinent to real users, such as precision at 10 or 30 images.

Conclusions We developed a test collection amenable to assessing visual and textual methods for image retrieval. Future work must focus on how varying topic and run types affect retrieval performance. Users’ studies also are necessary to determine the best measures for evaluating the efficacy of image retrieval systems.

Footnotes

  • Instructions for obtaining the data described in this paper can be obtained from the ImageCLEFmed Web site (http://ir.ohsu.edu/image/).

This Article

  1. Abstract
  2. Full text
  3. PDF
  4. Appendix

Responses

  1. Submit a response
  2. No responses published

Access policy for JAMIA

All content published in JAMIA is deposited with PubMedCentral by the publisher but with varying embargo times. Authors/funders may pay an Unlocked fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication. Research funded by government and other recognised agencies is deposited with a 12 month embargo. All other content is deposited with a 36 month embargo.

AMIA members log in here to access the full text of JAMIA.

Register for free content

Individuals may register for a free 30 day online trial to all content.

The Journal of the American Medical Informatics Association is published for the American Medical Informatics Association by BMJ Publishing Group Ltd.