Automated Detection of Adverse Events Using Natural Language Processing of Discharge Summaries
- Affiliations of the authors: Department of Biomedical Informatics, Columbia University (GBM, GH); and Medical Informatics Services, NewYork-Presbyterian Hospital (GH), New York, NY
- Correspondence and reprints: George Hripcsak, MD, MS, Department of Biomedical Informatics, Columbia University, 622 West 168th Street, Vanderbilt Clinic, 5th Floor, New York, NY 10032; e-mail: <hripcsak{at}columbia.edu>
- Received 19 January 2005
- Accepted 20 March 2005
Abstract
Objective To determine whether natural language processing (NLP) can effectively detect adverse events defined in the New York Patient Occurrence Reporting and Tracking System (NYPORTS) using discharge summaries.
Design An adverse event detection system for discharge summaries using the NLP system MedLEE was constructed to identify 45 NYPORTS event types. The system was first applied to a random sample of 1,000 manually reviewed charts. The system then processed all inpatient cases with electronic discharge summaries for two years. All system-identified events were reviewed, and performance was compared with traditional reporting.
Measurements System sensitivity, specificity, and predictive value, with manual review serving as the gold standard.
Results The system correctly identified 16 of 65 events in 1,000 charts. Of 57,452 total electronic discharge summaries, the system identified 1,590 events in 1,461 cases, and manual review verified 704 events in 652 cases, resulting in an overall sensitivity of 0.28 (95% confidence interval [CI]: 0.17–0.42), specificity of 0.985 (CI: 0.984–0.986), and positive predictive value of 0.45 (CI: 0.42–0.47) for detecting cases with events and an average specificity of 0.9996 (CI: 0.9996–0.9997) per event type. Traditional event reporting detected 322 events during the period (sensitivity 0.09), of which the system identified 110 as well as 594 additional events missed by traditional methods.
Conclusion NLP is an effective technique for detecting a broad range of adverse events in text documents and outperformed traditional and previous automated adverse event detection methods.
Footnotes
-
Supported by grants from the Agency for Healthcare Research and Quality (R18 HS11806) “Mining Complex Clinical Data for Patient Safety Research” and National Library of Medicine (R01 LM06910) “Discovering and Applying Knowledge in Clinical Databases.” Dr. Melton was supported by the National Library of Medicine Training Grant (5T15LM007079-12).
-
The authors thank Carol Friedman for the use of the natural language processor MedLEE (National Library of Medicine grant support R01 LM06274 and R01 LM07659), Sue West for her assistance with institutional NYPORTS reporting, and Karina Tulipano for serving as a case reviewer.
