The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review
- aUMIT–University for Health Sciences, Medical Informatics and Technology Tyrol, Institute for Health Information Systems, Tyrol, Austria
- bDepartment of Public Health, Medical Decision Making and HTA, Tyrol, Austria
- cAlfried Krupp von Bohlen und Halbach Foundation, Chair for Medical Management, University of Duisburg-Essen, Essen, Germany
- dInstitute for Technology Assessment and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA
- eProgram in Health Decision Science, Department of Health Policy and Management, Harvard School of Public Health, Boston, MA
- Correspondence: Elske Ammenwerth, Professor, Institute for Health Information Systems, UMIT—University for Health Sciences, Medical Informatics and Technology Tyrol, Eduard Wallnöfer Zentrum 1, A-6060 Hall in Tyrol, Austria Email: < >
- Received 19 November 2007
- Accepted 6 June 2008
The objective of this systematic review is to analyse the relative risk reduction on medication error and adverse drug events (ADE) by computerized physician order entry systems (CPOE). We included controlled field studies and pretest-posttest studies, evaluating all types of CPOE systems, drugs and clinical settings. We present the results in evidence tables, calculate the risk ratio with 95% confidence interval and perform subgroup analyses for categorical factors, such as the level of care, patient group, type of drug, type of system, functionality of the system, comparison group type, study design, and the method for detecting errors. Of the 25 studies that analysed the effects on the medication error rate, 23 showed a significant relative risk reduction of 13% to 99%. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction of 35% to 98%. Four of the seven studies that analysed the effect on ADEs showed a significant relative risk reduction of 30% to 84%. Reporting quality and study quality was often insufficient to exclude major sources of bias. Studies on home-grown systems, studies comparing electronic prescribing to handwriting prescribing, and studies using manual chart review to detect errors seem to show a higher relative risk reduction than other studies. Concluding, it seems that electronic prescribing can reduce the risk for medication errors and ADE. However, studies differ substantially in their setting, design, quality, and results. To further improve the evidence-base of health informatics, more randomized controlled trials (RCTs) are needed, especially to cover a wider range of clinical and geographic settings. In addition, reporting quality of health informatics evaluation studies has to be substantially improved.
Medical errors are an important factor that influences the quality of patient care. According to Barach et al., nearly 100,000 individuals per year in the US die of preventable medical errors.1
Medication errors have been identified as a major type of medical errors. The Council of Europe2 and the British Department of Health3 defined medication errors as “any preventable event that may cause or lead to inappropriate medication use or patient harm … .” The Institute of Medicine reports that a hospital patient can expect on average to be subjected to more than one medication error per day.4 5 Medication errors can lead to adverse drug events (ADEs) that are defined as “any response to a drug that is noxious and unintended… .”3 A report from the Institute of Medicine that was published in 1999 stated that annually in the US 7,000 deaths can be associated with medication errors.6
In light of these figures, it is not surprising that the British Department of Health recommends the wider use of electronic prescribing to reduce the risk of medication errors.3 Electronic prescribing is supported by Computerized Physician Order Entry (CPOE) systems. The term CPOE refers to a variety of computer-based systems for ordering medications, which share the common features of automating the medication ordering process.7 The CPOE systems can range from systems that only provide a list of possible medications that the physician can then choose from, to systems providing varying levels of decision support, including checks of drug-drug interactions, drug-allergy contraindications, or checks of prescriptions concerning the patient's recent laboratory results. All those checks lead to alerts and reminders given to the ordering physician in case problems are detected.
In the scientific literature, many studies investigate the effects of electronic prescribing on the quality of patient care. Many studies showed a reduction in medication errors (such as Bates et al., 1999),8 while others described negative effects such as the facilitation of medication error risks9 or an increase in mortality.10 The available reviews on this topic seem to not come to an unambiguous conclusion on the overall effects of electronic prescribing, concluding that published evidence is often insufficient.11 12 13 14 15 16 None of the published reviews has—to our knowledge—tried to quantitatively summarize the available evidence on the effects, and to systematically investigate the factors contributing to those effects.
Therefore, we performed a systematic and quantitative review to determine the effect of electronic prescribing on the risk of medication errors and adverse drug events (ADEs), and analyzed factors for those effects. This review should inform the health informatician of the effects that can be expected of electronic prescribing, and on the factors that influence the success of such systems. We see this review as a contribution to support evidence-based health informatics.17
Identification of Trials
We searched MEDLINE (1966 to April 2006) and EMBASE (1976 to April 2006) for studies on electronic prescribing that evaluated its effect on medication errors and ADEs. We combined MeSH terms, such as “Medical Order Entry Systems,” “prescriptions, drug,” “drug therapy, computer-assisted” with general search terms such as “order entry,” “CPOE,” “POE,” “order communication,” “prescription system,” “drug prescription,” “prescribing,” “ordering,” and “computerized reminders.” Those search terms were combined with terms that searched for evaluation studies that were taken from an earlier literature survey.18 The complete query is available upon request.
We also searched the Cochrane Database of Systematic Reviews and examined the reference lists of the relevant reviews in order to find further studies. Finally, we performed a hand search of the Journal of the American Medical Informatics Association (1994–2006), of the International Journal of Medical Informatics (1997–2006), and of the Methods of Information in Medicine (1990–2006). We completed this by hand-searching the references of retrieved study papers. We did not restrict the search to any single language.
Inclusion Criteria and the Selection of Studies
Intervention: We included studies wherein the intervention was electronic prescribing. As an intervention, we included all the computer-based application systems to order drugs that are used at the point of care. We included electronic prescribing systems independent of the level of decision support that they provided (e.g., with or without alerts on drug-drug interaction), and for all types of drugs. We excluded the studies that used electronic prescribing only for ordering diagnostic tests or therapeutic procedures.
Control: We only included studies wherein either an electronic prescribing system was compared with handwritten ordering, or where an electronic prescribing system with a more sophisticated functionality (e.g., a drug-drug interaction alert) was compared with another less sophisticated system.
Population and setting: We included studies where physicians were the primary users of the electronic prescribing system. We excluded studies where other groups (e.g., nurses, pharmacists) were the primary users. We included all clinical settings such as outpatient care, inpatient care, and intensive care. We included all patient groups.
Study design: We included randomized controlled trials, non-randomized controlled trials, before-after trials, as well as time-series analysis with multiple measurements. We only included field studies and excluded all lab studies and simulation studies.
Outcomes: We included studies that evaluated the effect of electronic prescribing on medication errors, potential ADEs, and ADEs. We defined medication error as all errors in the process of ordering, transcribing, dispensing, administering, and monitoring medication. This included an inappropriate drug, dosing, frequency, route, or timing (when related to patient safety), involving problems such as illegible or unsigned orders, and problems related to drug-allergy, drug-drug, drug-lab, and other interactions. Potential ADEs were defined as a medication error with significant potential to harm a patient that may or may not actually reach a patient. Adverse drug events (ADE) were defined as patient injuries resulting from drug use. We excluded studies where medication errors or ADEs were not the primary focus of the study, and studies that were still ongoing. We excluded papers if groups were definitely not comparable. If the data reporting was unclear, we contacted the authors and requested further information.
Data Extraction and Study Quality Assessment
We extracted data from the text, tables, and graphs of the original publications. Two reviewers (EA and CM) examined the data and reached consensus after discussion. In addition, one reviewer (PSI) independently reviewed all the extracted data. All of the cases of discordant data were resolved by discussion.
To detect medication error rates and ADE rates, we used the definition that is provided in each paper (please see Table 3, available online at www.jamia.org, for the definition of medication error and ADE for each respective paper).
When no absolute numbers were provided for medication error or ADE, we calculated these numbers based on the given data (e.g., if the frequency of ADEs was only given per 1.000 patient-days, the absolute ADE number could be calculated from the number of patient-days). To determine the study size, we used the number of orders as an observation unit of the analysis. If the number of orders was not available, the number of patients or patient-days was used. If multiple data were reported in a study (e.g., in time-series analysis), we used the data of the last reported measurement.
We classified the functionality of the CPOE system either as
no decision support: selection of drugs from a list, information on available doses and on costs, access to drug monographs, no further decision-support;
limited decision support: evidence-based patient-specific recommendation of a drug, dosing, frequency etc.; or
advanced decision support: at least some drug-allergy, drug-drug interaction, drug-lab, or other patient-specific alerts.
All results were reported in systematic evidence tables. The study quality was assessed by using a checklist (please see Table 1, available online at www.jamia.org), which was developed based on a 16-item assessment tool by the German Scientific Working Group Technology Assessment for Health Care that was applied independently by two reviewers (PSI, EA). Differences in judgment were then solved by discussion.
For each study, the risk ratio (RR) with its 95% confidence interval (CI) was calculated by comparing medication error rates, potential ADE rates, and ADE rates between the intervention and comparison group. If available, the number of orders was used as the denominator. Otherwise, the number of patient-days or the number of patients was used as the denominator.
We used a graphical approach based on forest plots to perform sub-group analysis and arranged studies by increasing risk ratios within subgroups. Subgroups were a priori defined as potentially relevant such as the clinical setting (inpatient, outpatient, or intensive care), patient group, type of drugs, type of system (home-grown or commercial), functionality (no, limited, or advanced decision support), study design, and method to detect errors.
All the analyses were performed with the software package STATA 9.2 (StataCorp, College Station, Texas, USA).
Overall, we identified 172 evaluation studies and 15 systematic reviews. From the 172 evaluation studies, 27 studies met all the inclusion criteria and were, therefore, included in a detailed review.8 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 For details, see Figure 1, available online at www.jamia.org. We obtained additional data from eleven authors.
Characteristics of the Trials
Table 2 shows the general characteristics of all the analysed 27 studies.
The majority of studies were conducted in the United States and in normal inpatient care units. Most studies were unspecific concerning the included patients and concerning the ordered drug. Half of the studies evaluated a commercial system, and half evaluated a home-grown, self-developed system. All of the systems offered a display of basic patient data (name, age), a list of drugs from which a provider can choose, and a list of potential doses for a chosen drug. Half of the studies used a system with advanced decision support, the others used systems with no or limited decision support. In most trials, electronic prescribing was compared to handwritten ordering.
Table 3, available online at www.jamia.org, shows the further study characteristics of the 27 studies, showing that most of them were before-after studies (including time-series analysis), with only two studies being randomized controlled trials.
Of those 27 studies, 15 only evaluated medication errors, two were only on ADE, and 10 reported both on medication errors and ADEs. Table 4, available online at www.jamia.org, shows the evidence table for outcomes.
Effect on Medication Errors
Twenty-five studies reported on the risk of medication errors (Figure 2). Twenty-three of these studies showed a significant relative risk reduction, with a risk ratio between 0.01 and 0.87. This indicates a relative risk reduction for medication errors of 13% to 99%. One study reported a statistically significant increase of 26% for the risk of medication errors (Spencer 2005).42 In this study, there was no connection between the CPOE system and the pharmacy system, so that orders had to be entered twice, in turn leading to transcription errors in the intervention group. In addition, the study authors only analysed those errors that were reported by the staff members themselves (voluntary reporting system)—this constitutes a potential detection bias. The one study (Mitchell 2004)35 showing inconclusive results (RR = 1.02, 95%-CI: 0.88; 1.19) was one of the smallest studies in this review, with only 320 orders analysed.
Effect on Potential Adverse Drug Events
Nine studies reported on the risk of potential ADE (Figure 3). Six of these studies showed significant relative risk reduction, with a risk ratio between 0.02 and 0.65. This indicates a relative risk reduction for potential ADEs of 35% to 98%.
Two studies showed inconclusive effects (Bizovi 2002 and Gandhi 2005).20 28 Both of these studies were rather small studies. The remaining also very small study by Mitchell 2004 showed a significant increase in the risk for potential ADEs.35
Effect on Adverse Drug Events
Seven studies reported the risk of ADE (Figure 4). In one of them (Shulman 2005),41 no events occurred in the intervention or in the comparison group and, therefore, this study was excluded from further analysis.
Four of the six remaining studies showed a significant relative risk reduction for ADEs, with a risk ratio between 0.16 and 0.70. This indicates a relative risk reduction for ADEs of 30% to 84%. One study (Mullett 2001)36 showed a not statistically significant relative risk reduction of 13%, and the remaining study (Bates 1998)19 showed a small, not statistically significant increase of 9%.
A graphical sub-group analysis was only performed for those 25 studies that focused on medication errors.
The sub-group analysis comparing medication error reduction by commercial systems versus home-grown systems indicated a higher relative risk reduction by home-grown systems (Figure 5).
The sub-group analysis for those seven studies that compared advanced electronic prescribing with limited electronic prescribing and those 18 studies that compared electronic prescribing with handwritten ordering showed a higher relative risk reduction for the comparison made with handwritten ordering (Figure 6).
The 14 studies with an advanced decision-support reported a higher relative risk reduction than the 11 studies with limited or no decision-support (Figure 7). Most studies with limited support, however, were compared to computer-based ordering, whereas most studies with advanced decision-support or without decision-support were compared to handwritten ordering.
The sub-group analysis comparing different methods to detect errors showed higher relative risk reductions with the manual chart review of prescriptions (11 studies) than with the automatic database analysis of prescriptions (6 studies) (please see Figure 8, available online at www.jamia.org).
The other graphical sub-group analysis did not indicate differences between groups: The risk ratios between groups seem to be similar for the level of care (normal care versus intensive care), patient groups included (elderly, children, unspecific), type of drugs included (specific or unspecific), or study design (before-after trial, controlled trial, RCT).
Study and Reporting Quality
The reporting quality was poor for most of the studies. The details are shown in Table 5, available online at www.jamia.org. For example, many studies did not clearly specify the inclusion or exclusion criteria of the participating institutions or patients, did not report on the baseline characteristics of participants or as to whether the comparison and intervention groups were treated similarly (with the exception of the intervention), or did not comment on missing values or drop-outs.
In addition, only two studies were randomized trials, and few studies had comparison groups recruited over the same period. Only in half of the studies, the outcome seemed to be measured validly and reliably, however, the measurement was mostly not blinded. Nearly all the studies used adequate hypothesis tests and reported on the statistical precision of the main outcomes, but only less than half of the studies attempted to adjust for confounding or clustering. Most studies were single centre studies, so that the generalisability of the results to other centres, especially of a different type, is unclear.
Thwenty-three of the 25 studies that analysed the effects on the medication error rate showed a significant relative risk reduction of 13% to 99% for medication errors. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction for potential ADEs of 35% to 98%. Four of the seven studies that analysed the effect of electronic prescribing on ADEs showed a significant relative risk reduction for ADEs of 30% to 84%. These findings, that are in line with results from other reviews, indicate that electronic prescribing can substantially reduce the risk for medication errors, potential ADEs, and ADEs.12 13 16 45 46 47 The low number of studies focussing on ADE may be due to the fact that medication errors can be detected more easily than ADEs.16
From a clinical perspective, the observed effect sizes (e.g., up to 99% reduction of medication errors, up to 84% reduction of ADEs) seems to be rather large. However, the medication errors and ADEs are just surrogate outcomes, and are not necessarily directly related to changes in the patient-relevant medical outcomes.48 The actual improvements in medical outcomes (e.g., reduction in mortality rates or hospitalisation days) have not yet been sufficiently analysed by quantitative, controlled trials.15 45 48 One exception is the study by Han et al.10 that observed an increase in mortality after the introduction of a CPOE system (these heavily discussed findings could not be reproduced later on by others such as Del Baccaro et al.49). Altogether, more systematic trials looking at patient-relevant medical outcomes should be conducted in the future. Furthermore, more studies should also be conducted to analyse the costs for any benefit that may be obtained. For this review, we concentrated on medication errors and ADEs, as a sufficient amount of published evidence is available.
Results of the Sub-group Analysis
The included studies showed substantial heterogeneity (see their detailed description in Table 2 as well as in Table 3, available online at www.jamia.org). This reflects the diversity of electronic prescribing systems and their respective use.
The sub-group analysis indicated that the level of care, as well as the included patient groups and drugs did not affect the overall risk ratios. The graphical presentation, however, indicated that home-grown systems show a higher relative risk reduction than commercial systems. This is not surprising, as home-grown systems can be modified and adapted easier and more quickly to the local needs. Rigby50 calls this “alpha sites” and argues that those sites are atypical concerning larger technical, emotional, and financial support.
Also, studies comparing handwritten prescribing with electronic prescribing seem to show a higher relative risk reduction than those studies comparing groups with a different level of electronic ordering. This is also not surprising, as certain error types such as illegible orders, that are often present in handwritten prescribing, are completely removed by electronic prescribing.
Electronic prescribing systems with advanced decision support seemed to show a higher relative risk reduction compared to those with limited or no decision support. It is here that the patient-specific alerts seem to best support the quality of prescribing.
Finally, different methods for detecting medication errors seem to show a different relative risk reduction. Using a manual chart review, relative risk reduction seems to be highest. However, chart review can hardly be blinded, and reviewers may be biased towards CPOE.
Overall, the reporting quality and study quality of the included studies were not always satisfactory.
One major problem was that many studies did not provide sufficient information in order to adequately assess the comparability of the intervention and comparison groups (e.g., by not reporting baseline characteristics, by not reporting about adjunct initiatives such as training sessions or about drop-outs etc.). This made it difficult to analyse as to whether the differences found between the study groups really originated from electronic prescribing, or from other factors. Subsequently, about two-thirds of the studies did not attempt to adjust for possible confounding factors. Then, many studies also used designs of lower level of evidence such as before-after trials, here it is unclear whether any context such as staffing or workflow of the study departments may have changed over time, in turn influencing the observed effects. The same is true for any non-randomized allocation of clinicians and/or patient to study groups. This all affects the validity of the analysed studies. We dealt with this by describing in detail the individual study quality (see Table 5, available online at www.jamia.org), and by excluding those studies that seem to have non-comparable groups.51 52
There were other limitations of the included studies: Four studies analysed the data from voluntary error reporting systems (e.g., nurses and physicians voluntary reported on errors), which may underestimate the real number of errors and ADE in those studies (detection bias). Other studies retrospectively analysed routinely collected information on medication errors and ADEs. Here, it is partly unclear as to how they assessed the quality of this routine data. Overall, only a few studies provided information on the objectivity and reliability of the instruments that were used to (manually or automatically) detect errors.
Summarizing, the reporting and study quality were not always adequate. This must be taken into account when interpreting the results of the studies. Efforts should be made to improve the reporting quality and study design and analyses of evaluation studies. Publication guidelines that are comparable to CONSORT53 or STROBE54 should be developed, taking into account the often quasi-experimental design of the health informatics evaluation studies. One example for such publication guidelines for health informatics evaluation studies are the STARE-HI guidelines (Standards for Reporting on Evaluation Studies in Health Informatics) that will shortly be published.55
Limitations of the Review
Our review may present some limitations.
First, the published trials on electronic prescribing differ substantially in their respective terminology. We used many different Keywords to search for trials; searched in several databases as well as for non-English papers; and applied hand-searching. Nevertheless, we may have overlooked some individual studies. In addition, there is the possibility of publication bias, as we have only included published studies.
Second, the definition of the measured outcome variable differed between the studies. For example, some studies included the legibility of orders in the definition of medication, and others did not. This can affect the measured medication error rate.
Third, when several end-points were reported (e.g., in a time-series analysis), we only included the final endpoint. This may overestimate the positive effects of electronic prescribing, as the intermediate results partly show less (or even negative) effects during an early introduction period. In addition, our analysis relied on the assumption that one order could have maximal one error. However, this was not explicitly reported by several studies.
Fourth, our categories that were used to stratify the sub-groups might have been too crude. For example, the study design was very heterogeneous and the diversity might not have been represented adequately by the categories we chose. A multivariate sub-group analysis controlling for the multiple factors simultaneously was not performed because of the limited number of studies. Then, further factors such as the organisation of medication processes in the various settings, age and computer knowledge of users, management support, or the quality and usability of the CPOE system may further affect the effects. Information on those items was not included in most studies and could, therefore, not be used for sub-group analysis.
Fifth, the rather large percentage of home-grown systems in our review may have biased this review towards more positive results. Moreover, 8 of the studies included came from just two sites: the Brigham and Women's Hospital, Boston, MA with the BICS system, and the LDS Hospital, Salt Lake City, Utah, with the HELP systems. Therefore, we cannot exclude a selection bias, with the observed effects not being representative for general routine clinical settings. Some studies on BICS and HELP may not be completely independent of each other (we therefore excluded Evans et al., 1994),56 but when they used different outcome criteria or addressed different patient populations, we decided to include them.
Sixth, we were unable to use meta-analysis to pool the effect sizes, given the large heterogeneity among the studies. Meta-analysis would help to quantify the overall effect, and has already been performed e.g., for computer-assisted management of anticoagulant therapy57 or for clinical reminder systems for preventive care,58 but needs a sufficient number of homogeneous studies. We therefore conducted a graphical sub-group analysis—even this, however, should be viewed with care, given the tremendous variability in study design and in the methodological rigor of the included studies.
Seventh, most of the evaluated systems have only been implemented shortly before or during the study, and sometimes only as pilots in selected areas. This may have led to erroneous estimations concerning the long-term effects of electronic prescribing. More long-term studies seem to be needed.
Finally, researchers such as Berger15 note that electronic prescribing systems could increase the number of medication errors and ADEs when not appropriately designed and implemented. There is, in fact, increasing evidence from (mostly qualitative) research pointing to risks with electronic prescribing by CPOE systems. Koppel et al.,9 e.g., described an increase in medication errors risks by CPOE. Campbell et al.59 identified categories of unintended consequences of CPOE. This (qualitative) evidence is, however, not (yet) reflected in the results of quantitative controlled evaluation studies and therefore also not in our review. The discussion of the reasons for this (such as a possible publication bias) is outside the scope of the present paper. For a more detailed discussion see Ammenwerth et al.60
Implication and Conclusion
After having been envisioned by Morris Collen in 1970 (“Physicians should enter their medical orders directly into the computer”),61 electronic prescribing systems have now been in routine use for approximately 15 years. Our review is, to our knowledge, the first that quantitatively calculates and compares the risk ratios for each evaluation study. We also conducted a sub-group analysis, hereby analysing factors that influence the effects of electronic prescribing; and we analysed and discussed carefully the heterogeneous and often weak quality of published studies.
Our results indicate that electronic prescribing seems to be a useful intervention for reducing the risk of medication errors and ADEs. We found that there is especially good evidence for a positive effect of electronic prescribing offering advanced decision support functionality in hospital settings. Less evidence is available for systems outside the U.S., for primary care settings, and for specific commercial systems. Therefore, more studies are necessary.
The study quality was often weak, with many before-after trials and an unclear comparability of the study groups. More randomized controlled trials from more sites are needed to further improve the evidence and to identify the setting that those systems are most useful in. Researchers planning comparable studies are advised to focus early on the issue of study quality and study validity, to avoid as many sources of bias as possible (see also Brender, 200262). A larger number of high-quality studies for certain settings and patient groups would also allow for the use of meta-analysis to pool the effect sizes. In addition, the reporting quality of health informatics evaluation studies has to be substantially improved; here, the initiatives for publication guidelines such as STARE-HI55 are urgently needed. All this should help to increase our knowledge on electronic prescribing and globally increase the evidence-base of health informatics.17
Systematic reviews of quantitative trials can indicate which effect can be obtained from an intervention. They do not show, however, how to implement electronic prescribing. It seems obvious that the quality of the implementation process has an impact on the success of a CPOE implementation.63 Insufficient implementation planning, or systems that are not integrated into the general information systems, may lead to negative effects on the process and even the outcome of care.64 65 66 67 Negative effects of electronic ordering were identified only in two studies in our review, even when increasing evidence from qualitative research points to possible adverse effects of electronic prescribing. Each implementation should be carefully monitored concerning an increase in medication errors and ADEs, and more qualitative research should be conducted to assess the reasons for the respective success or failure, and to guide further successful implementation processes.
The authors thank all reviewers for their helpful comments on earlier versions of this paper.