Table 2
Results and Barriers to Implementation of Studies Evaluating an Adverse Event Monitor Using a Gold Standard
| Study | Description of Monitor | Study Results | False Positives and/or False Negatives | Barriers to Implementation |
| Nosocomial infections | ||||
| Rocha et al.62 | An expert system using boolean logic to detect hospital-acquired infectons in newborns. The system is by positive micro-biology results (data-driven) and at specific periods of time (time-driven) to search for signals. | The computer activated 605 times, 514 times by culture results and 91 times by CSF analysis. The sensitivity of the tool was 85% and speci-ficity 93%. Compared with an expert reviewer's judge-ment, the tool had a kappa statistic of 0.62. | There were 32 false posi-tives (7%) and 11 false negatives (16%). | The detection system would require a highly integrated and sophisticated HIS to operate. No data were pro-vided regarding the time necessary to maintain the system. |
| Evans et al.55 | A series of computer pro-grams that translate the patients' microbiology test results into a hierarchical database. Data are then compared with a computer-ized knowledge base devel-oped to identify patients with hospital-acquired infections or receiving inappropriate antibiotic therapy. The system is time-driven, and alerts are transported to the infectious disease service for confirma-tion and investigation. | Either or both computerized detection or traditional methods identified 217 patients. 155 patients were determined to have had a nosocomial infection. The computer identified 182 cases, of which 140 were confirmed (77%). Out of all the confirmed cases (150) the computer identified 90% while traditional methods detected 76%. | 23% (42/182) of the alerts were false positives. The rate of false positives was the same as manual review. Contamination was responsible for many of the false positives. | HIS without a high level of integration might not be able to support the rule base. Infection control practitioners (traditional method) spent 130 hours on infection surveil-lance and 8 hours on collect-ing materials. Only 8.6 hours were necessary to prepare similar reports using com-puterized screening results plus an additional 15 minutes for verification in each patient, resulting in a total of 45.5 hours of surveillance time. |
| Hirschhorn et al.65 | A computer program that captures the duration and timing of postoperative anti-biotic exposure and the ICD-9-CM coded discharge diagnosis. This information was used to screen for possible nosocomial infections. | The overall incidence of infection was 9%. Eight per-cent of all patients had a coded diagnosis for infection. Exposure to greater than 2 days of antibiotics had a sensitivity of 81%, a speci-ficity of 95%, and a PPV of 61% to detect infections. The coded diagnosis had a sensitivity of 65%, a specificity of 97%, and a PPV of 74%. A combination of screens had a sensitivity of 59% and a PPV of 94%. | Based on manual review, 5% of pharmacy records were misclassified with 18% of patients being incorrectly labeled as having received greater than 2 days of antibiotics. Discharge codes missed 35% of the infections. | The monitor would not require a highly integrated HIS and would be easier to implement. No information was described regarding the level of work necessary to maintain the system. |
| Adverse drug events | ||||
| Honigman et al.17 | A computerized tool that reviewed elctronically stored records using four search strategies: ICD-9-CM codes, allergy rules, a computer event monitor, and auto-mated chart review using free-text searches. After the search was performed the data were narrowed and queried to identify incidents. | The monitor detected an esti-mated 864 (95% CI, 750-978) ADEs in 15,655 patients. For the composite tool the sensi-tivity was 58% (95% CI, 18-98), specificity 88% (95% CI, 87-88), PPV 7.5% (95% CI, 6.5-8.5), and NPV 99.2% (95% CI, 95.5-99.98). | For the composite tool the false-positive rate was 42% (637/1501) and the false-negative rate was 12% (10,619/87,013). | The monitor requires a highly integrated HIS to implement. ICD-9-E codes were not used frequently at the study insti-tution. Only a small lexicon had been developed for free-text searches. The study did not mention the amount of time that would be necessary to maintain the monitor. |
| Levy et al.72 | A data-driven monitor where automated laboratory signals (alerts) were gen-erated when a specific labora-tory value reached a pre-defined criteria. A list of alerts was generated on a daily basis and presented to staff physicians. | 32% (64/199) patients had an ADR. There were 295 alerts generated involving 69% of all admissions. Of all ADRs, 61% (43/71) were detected by the automated signals. The sensitivity of the system was 62% with a specificity of 42%. 18% (52/295) of alerts represented an ADR | Overall 82% (243/295) of the alerts were false positives. | Authors mention an “easy implementation” but imple-mentation is not described; however, the high false-posi-tive rate would add to the overall work required to maintain the system. The time necessary to maintain the system is not described. |
| Jha et al.11 | A computerized event monitor detecting events using individual signals and boolean combinations of signals involving medica-tion orders and laboratory results. The computer gen-erates a list of alerts that are reviewed to determine if further evaluation is needed. | 617 ADEs were identified during the study period. The computer monitor iden-tified 2,620 alerts of which 10% (275) were ADEs. The PPV of the event monitor was 16% over the first 8 weeks of the study but increased to 23% over the second 8 weeks after some rule modification. | The false-positive rate over the entire study period was 83%. | In hospitals without this sophisticated a IS, it might be challenging to implement the monitor. The monitor was unable to access microbiology results. To maintain the system required 1-2 hours of programming time a month and 11 person-hours a week to evaluate alerts. |
| Adverse events | ||||
| Weingart et al.75 | A computer program that searched for ICD-9-CM codes that could represent a medical or surgical com-plication. Screened positive discharge abstracts were discharge abstracts were initially reviewed by nurse reviewers, and if a quality problem was believed to have occurred, the physician reviewers then reviewed the chart. | There were 563 surgical and 268 medical cases flagged by the monitor. Judges con-firmed alerts in 68% of the surgical and 27% of the medical flagged cases. 30% of the surgical and 16% of the medical cases identified by the screening tool had quality problems associated with them. | 73% of the medical alerts and 32% of the surgical alerts were flagged with-out an actual complica-tion. 2.1% of the medical and surgical controls had quality problems associ-ated with them but were not flagged by the program. | The monitor would be rela-tively easy to implement; however, the low PPV of the tool for medical charts raises concerns about the accuracy of ICD-9-CM codes and threatens the usefulness of the tool in medical patients. The kappa scores were low for interrater reliability (0.22) con-cerning quality problems. No data were presented about the time necessary to maintain the monitor. |
| Bates et al.76 | The study evaluated five electronically available bill-ing codes as signals to detect AE. Medical records under-went initial manual screen-ing followed by implicit physician review. | There were 341 AEs detected in the study group. The use of all 5 screens would detect 173 adverse events in 885 admissions. The sensitivity and specificity of this strategy were 47% and 74% with a PPV of 20%. Eliminat-ing one poorly performing screen (the least specific) would detect 88 AEs in 289 charts with a sensitivity of 24% and specificity of 93% and a PPV of 30% | The first strategy resulted in 712 false-positive screens out of 885 alerts (80%). The second strategy resulted in 201 false-positive screens out of 289 alerts (70%). | The monitor utilized readily available electronically stored billing data for signals, making the tool more general-izable for most institutions. Electronic screening cost $3 per admission reviewed and $57 per adverse event $57 per adverse event detected compared with $13 per admission and $116 per adverse event detected when all charts were reviewed manually. |
-
NI = nosocomial infection; AE = adverse event; ADE = adverse drug event; ADR = adverse drug reaction; PPV = positive predictive value; HIS = hospital information system.
