Clinical Decision Velocity is Increased when Meta-search Filters Enhance an Evidence Retrieval System
- aCentre for Health Informatics, University of New South Wales, Australia
- bHealth Informatics Research & Evaluation Unit, Faculty of Health Sciences, University of Sydney, Sydney, Australia
- Correspondence: Dr. Enrico Coiera, University of New South Wales, Centre for Health Informatics, UNSW, NSW 2052, Australia (Email: <e.coiera{at}unsw.edu.au>)
- Received 20 February 2008
- Accepted 8 June 2008
Abstract
Objective To test whether the use of an evidence retrieval system that uses clinically targeted meta-search filters can enhance the rate at which clinicians make correct decisions, reduce the effort involved in locating evidence, and provide an intuitive match between clinical tasks and search filters.
Design A laboratory experiment under controlled conditions asked 75 clinicians to answer eight randomly sequenced clinical questions, using one of two randomly assigned search engines. The first search engine Quick Clinical (QC) was equipped with meta-search filters (the combined use of meta-search and search filters) designed to answer typical clinical questions e.g., treatment, diagnosis, and the second ‘library model’ system (LM) offered free access to an identical evidence set with no filter support.
Measurements Changes in clinical decision making were measured by the proportion of correct post-search answers provided to questions, the time taken to answer questions, and the number of searches and links to documents followed in a search session. The intuitive match between meta-search filters and clinical tasks was measured by the proportion and distribution of filters selected for individual clinical questions.
Results Clinicians in the two groups performed equally well pre-search. Post search answers improved overall by 21%, with 52.2% of answers correct with QC and 54.7% with LM (χ2 = 0.33, df = 1, p > 0.05). Users of QC obtained a significantly greater percentage of their correct answers within the first two minutes of searching compared to LM users (QC 58.2%; LM 32.9%; χ2 = 19.203, df = 1, p < 0.001). There was a statistical difference for QC and LM survival curves, which plotted overall time to answer questions, irrespective of answer (Wilcoxon, p = 0.019) and for the average time to provide a correct answer (Wilcoxon, p = 0.006). The QC system users conducted significantly fewer searches per scenario (m = 3.0 SD = 1.15 versus m = 5.5 SD1.97, t = 6.63, df = 72, p = 0.0001). Clinicians using the QC system followed fewer document links than did those who used LM (respectively 3.9 links SD = 1.20 versus 4.7 links SD = 1.79, t = 2.13, df = 72, p = 0.0368). In 6 of the 8 questions, two meta-search filters accounted for 89% or more of clinicians' first choice, suggesting the choice of filter intuitively matched the clinical decision task at hand.
Conclusions Meta-search filters result in clinicians arriving at answers more quickly than unconstrained searches across information sources, and appear to increase the rate with which correct decisions are made. In time restricted clinical settings meta-search filters may thus improve overall decision accuracy, as fewer searches that could otherwise lead to a correct answer are abandoned. Meta-search filters appear to be intuitive to use, suggesting that the simplicity of the user model would fit very well into clinical settings.
Footnotes
-
The design and implementation of Quick Clinical was supported by funding from Australian Research Council SPIRT Grant C00107730, NHMRC Development Grant 300591, and Merck Sharpe and Dohme (Australasia). Evaluation work was in part supported by ARC Discovery grant DP0452359 and the National Institute of Clinical Studies. The authors thank the clinicians who gave their time to take part in the study; and the clinicians who contributed to the development of the scenarios: Karolyn Vaughan, Karen Lintern, Dr. Madlen Gazarian, Dr Vitali Sinchenko and Dr Barbara Booth. Staff from the Centre for Health Informatics assisted with setting up and running the experiment including Nerida Creswick, Keri Bell and Annie Lau. Michelle Wensley from New South Wales Health assisted with recruitment of clinicians. The QC development team included Ken Nguyen, Martin Walther, Hugh Garsden, Victor Vickland and Luis Chuquipiondo.
-
Conflict of Interest: Quick Clinical was developed by researchers at the Centre for Health Informatics at the University of New South Wales, and the university and some of the authors could benefit from commercial exploitation of QC or its technologies.
