Evaluation of a Cardiac Diagnostic Program in a Typical Clinical Setting
- Affiliations of the authors: Informatics Group, Children's Hospital, Boston, Massachusetts (HSFF); Clinical Decision Making Group, MIT Laboratory for Computer Science, Cambridge, Massachusetts (HSFF, WJL); Division of Cardiology, Tufts-New England Medical Center, Boston, Massachusetts (SN), USA
- Correspondence and reprints: Hamish S. F. Fraser, MBCHB, MSc, The Program in Infectious Disease and Social Change, Harvard Medical School, 641 Huntington Avenue, Boston, MA 02115; e-mail: <hamish{at}medg.lcs.mit.edu>
- Received 8 July 2002
- Accepted 23 February 2003
Abstract
Context The Heart Disease Program (HDP) is a novel computerized diagnosis program incorporating a computer model of cardiovascular physiology. Physicians can enter standard clinical data and receive a differential diagnosis with explanations.
Objective To evaluate the diagnostic performance of the HDP and its usability by physicians in a typical clinical setting.
Design A prospective observational study of the HDP in use by physicians in departments of medicine and cardiology of a teaching hospital. Data came from 114 patients with a broad range of cardiac disorders, entered by six physicians.
Measurements Sensitivity, specificity, and positive predictive value (PPV). Comprehensiveness: the proportion of final diagnoses suggested by the HDP or physicians for each case. Relevance: the proportion of HDP or physicians' diagnoses that are correct. Area under the receiver operating characterist (ROC) curve (AUC) for the HDP and the physicians. Performance was compared with a final diagnosis based on follow-up and further investigations.
Results Compared with the final diagnoses, the HDP had a higher sensitivity (53.0% vs. 34.8%) and significantly higher comprehensiveness (57.2% vs. 39.5%, p < 0.0001) than the physicians. Physicians' PPV and relevance (56.2%, 56.0%) were higher than the HDP (25.4%, 28.1%). Combining the diagnoses of the physicians and the HDPs, sensitivity was 61.3% and comprehensiveness was 65.7%. These findings were significant in the two collection cohorts and for subanalysis of the most serious diagnoses. The AUCs were similar for the HDP and the physicians.
Conclusions The heart disease program has the potential to improve the differential diagnoses of physicians in a typical clinical setting.
Footnotes
-
Parts of this material have been presented at the AMIA Fall symposiums in 199743, 199835, 199944, and 200032.
-
This work was supported by the National Heart, Lung, and Blood Institute Grant R01-HL33041. The study was approved by the New England Medical Center Human Investigation Review Board. The authors thank the cardiology and General Medical Associates (GMA) departments at New England Medical Center for permission to use patient data; members of the medical resident and GMA staff for case entry; and John Wong, Peter Szolovits, James Stahl, and Laura Smeaton for their advice.
