Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs' Computerized Patient Record System
- Ching-Ping Lina,
- Thomas H Paynea,b,
- W Paul Nicholc,
- Patricia J Hoeyc,
- Curtis L Andersond,
- John H Gennaria
- aBiomedical and Health Informatics, University of Washington, Seattle, WA
- bUW Medicine Information Technology Services, University of Washington, Seattle, WA
- cVA Puget Sound Health Care System, Seattle, WA
- dSalt Lake City CIO Field Office, Dept. of Veterans Affairs, Salt Lake City, UT
- Correspondence: Ching-Ping Lin, Biomedical and Health Informatics, Department of Medical Education and Biomedical Informatics, University of Washington, Box 357240, 1959 NE Pacific Street, HSB I-264, Seattle, WA 98195-7240; (e-mail: <ping{at}u.washington.edu>)
- Received 28 March 2008
- Accepted 13 June 2008
Abstract
Objective To measure critical order check override rates in VA Puget Sound Health Care System's computerized practitioner order entry (CPOE) system and to compare 2006 results to a similar 2001 study.
Design Analysis of ordering and order check data gathered by a post-hoc logging program. Use of Pearson's chi-square contingency table test comparing results from this study and the earlier study.
Measurements Factors measured were total number of orders, frequency of order check types, frequency of order check overrides by order check type and comparisons of these results with previous results.
Results A total of 37,040 orders generated 908 (2.5%) critical order checks. Drug-drug critical alert override rate was 74/85 (87%) in 2006 compared to 95/108 (88%) in 2001 (X2=0.04, df=1, p=0.85). The drug-allergy override rate was 341/420 (81%) compared to 72/105 (69%) in 2001 (X2=7.97, df=1, p=0.005). In 2001, 0.25% (105/42,621) orders generated a drug-allergy order check compared to 1.13% (420/37,040) in 2006 (X2=238.45, df=1, p<0.0001).
Conclusion Override rates of critical drug-drug and drug-allergy order checks remain high at VA Puget Sound Health Care System including significant increases in drug-allergy order checks. We recommend that monitoring override rates be regular practice in clinical computing systems and conclude that qualitative research should be carried out to better understand how physicians interact with decision support at the point of ordering.
Footnotes
-
This research was funded in part by National Library of Medicine Training Grant #T15 LM007442. The authors thank the anonymous reviewers for a number of suggestions that improved this paper. The authors also thank Terry L. Roth, MS and Michael N. Frost for their technical assistance.
-
↵* A common term for automatic drug-drug or drug-allergy warnings is “alert.” However, this term can refer to many types of messages generated such as reminders or monitoring alerts. In our study, we only examine alerts generated during the ordering process which we refer to as “order checks.”
-
↵† We must distinguish between “cancelled” and “discontinued” orders. Orders that are “cancelled” are acted upon by the ordering clinician before they are signed and processed. Orders that have been signed and processed but then stopped are considered “discontinued.” Discontinued orders are not deleted from the archived database and remain as part of the patient's permanent medical record. Our study was only concerned with orders cancelled prior to signing.
-
↵†† In 2001, we counted order checks rather than distinct orders. Thus, there is the possibility that the 2001 order override rate was actually slightly higher, due to multiple critical drug-drug order checks per order.
