Information Retrieval Performance of Probabilistically Generated, Problem-Specific Computerized Provider Order Entry Pick-Lists: A Pilot Study
- Affiliation of the authors: Division of Health Sciences Informatics, Johns Hopkins University School of Medicine, Baltimore, MD
- Correspondence and reprints: Adam S. Rothschild, MD, Department of Biomedical Informatics, Columbia University, Vanderbilt Clinic 5th Floor, 622 West 168th Street, New York, NY 10032; e-mail: <adam.rothschild{at}dbmi.columbia.edu>
- Received 11 August 2004
- Accepted 24 January 2005
Abstract
Objective The aim of this study was to preliminarily determine the feasibility of probabilistically generating problem-specific computerized provider order entry (CPOE) pick-lists from a database of explicitly linked orders and problems from actual clinical cases.
Design In a pilot retrospective validation, physicians reviewed internal medicine cases consisting of the admission history and physical examination and orders placed using CPOE during the first 24 hours after admission. They created coded problem lists and linked orders from individual cases to the problem for which they were most indicated. Problem-specific order pick-lists were generated by including a given order in a pick-list if the probability of linkage of order and problem (PLOP) equaled or exceeded a specified threshold. PLOP for a given linked order-problem pair was computed as its prevalence among the other cases in the experiment with the given problem. The orders that the reviewer linked to a given problem instance served as the reference standard to evaluate its system-generated pick-list.
Measurements Recall, precision, and length of the pick-lists.
Results Average recall reached a maximum of .67 with a precision of .17 and pick-list length of 31.22 at a PLOP threshold of 0. Average precision reached a maximum of .73 with a recall of .09 and pick-list length of .42 at a PLOP threshold of .9. Recall varied inversely with precision in classic information retrieval behavior.
Conclusion We preliminarily conclude that it is feasible to generate problem-specific CPOE pick-lists probabilistically from a database of explicitly linked orders and problems. Further research is necessary to determine the usefulness of this approach in real-world settings.
Footnotes
-
Dr. Rothschild is currently with the Department of Biomedical Informatics, Columbia University, New York, NY.
-
Presented in part at the 2004 National Library of Medicine Training Program Directors' Meeting in Indianapolis, IN in a plenary session and MEDINFO 2004 in San Francisco in poster form.
-
Supported by the National Library of Medicine training grant 5T15LM007452 (ASR).
-
The authors thank the following physicians for serving as reviewers: David Camitta, Richard Dressler, Cupid Gascon, Mark Laflamme, Arun Mathews, Zeba Mathews, Greg Prokopowicz, Danny Rosenthal, Shannon Sims, and Chuck Tuchinda. The authors thank Bill Hersh, George Hripcsak, and Gil Kuperman for their expert advice on analysis and manuscript preparation.
