A long-term follow-up evaluation of electronic health record prescribing safety
- Erika L Abramson1,2,3,4,5,
- Sameer Malhotra2,4,
- S Nena Osorio1,3,4,
- Alison Edwards1,2,4,5,
- Adam Cheriff3,5,6,
- Curtis Cole3,5,6,
- Rainu Kaushal1,2,3,4,5,6
- 1Department of Pediatrics, Weill Cornell Medical College, New York, New York, USA
- 2Department of Public Health, Weill Cornell Medical College, New York, New York, USA
- 3New York-Presbyterian Hospital, New York, New York, USA
- 4Health Information Technology Evaluation Collaborative (HITEC), New York, New York, USA
- 5Center for Healthcare Informatics and Policy (CHiP), New York, New York, USA
- 6Department of Medicine, Weill Cornell Medical College, New York, New York, USA
- Correspondence to Dr Erika L Abramson, Departments of Pediatrics and Public Health, Weill Medical College of Cornell University, 530 East 70th Street, Room M-610A, New York, NY 10065, USA;
- Received 12 November 2012
- Revised 13 March 2013
- Accepted 17 March 2013
- Published Online First 11 April 2013
Objective To be eligible for incentives through the Electronic Health Record (EHR) Incentive Program, many providers using older or locally developed EHRs will be transitioning to new, commercial EHRs. We previously evaluated prescribing errors made by providers in the first year following transition from a locally developed EHR with minimal prescribing clinical decision support (CDS) to a commercial EHR with robust CDS. Following system refinements, we conducted this study to assess the rates and types of errors 2 years after transition and determine the evolution of errors.
Materials and methods We conducted a mixed methods cross-sectional case study of 16 physicians at an academic-affiliated ambulatory clinic from April to June 2010. We utilized standardized prescription and chart review to identify errors. Fourteen providers also participated in interviews.
Results We analyzed 1905 prescriptions. The overall prescribing error rate was 3.8 per 100 prescriptions (95% CI 2.8 to 5.1). Error rates were significantly lower 2 years after transition (p<0.001 compared to pre-implementation, 12 weeks and 1 year after transition). Rates of near misses remained unchanged. Providers positively appreciated most system refinements, particularly reduced alert firing.
Discussion Our study suggests that over time and with system refinements, use of a commercial EHR with advanced CDS can lead to low prescribing error rates, although more serious errors may require targeted interventions to eliminate them. Reducing alert firing frequency appears particularly important. Our results provide support for federal efforts promoting meaningful use of EHRs.
Conclusions Ongoing error monitoring can allow CDS to be optimally tailored and help achieve maximal safety benefits.
Clinical Trials Registration ClinicalTrials.gov, Identifier: NCT00603070.
Background and significance
The federal government is investing unprecedented funding to promote the meaningful use of electronic health records (EHRs).1 Electronic order entry of medications is an important tool in these efforts and is included as a core requirement. In the outpatient setting, focus on electronic prescribing appears critical given the high rate of prescribing errors and adverse drug events (ADEs), as well as the frequency with which medications are prescribed.2–5
As a result of federal incentives, widespread use of EHRs for prescribing is expected. Indeed, the percentage of e-prescriptions written annually has been steadily increasing.6 Most providers will be newly adopting EHRs, predominantly commercial systems because they are easily accessible and readily available. However, some providers and institutions will be transitioning from locally developed to commercial EHRs. These migrations will occur despite the unique customization of locally developed systems, to meet the standards necessary for incentives and to take advantage of vendor development and technical support services.7 ,8 Additionally, among providers already using a commercial EHR, many will have to transition to newer system versions to meet meaningful use requirements.
Implementation of new systems is challenging and there has been little research on the effect of transitioning between EHRs on prescribing safety.9 We previously published the first quantitative study to our knowledge examining the effect on prescribing errors of transitioning between a locally developed EHR with minimal clinical decision support (CDS) for prescribing to a commercial EHR with robust CDS.9 We found that implementation of the commercial EHR led to a significant and progressive decrease in overall rates of prescribing errors over 1 year, largely by reducing one specific error type (inappropriate abbreviation use). However, when inappropriate abbreviation errors were excluded, errors rates were significantly higher 12 weeks after transition and were no different at 1 year than at baseline, despite the new system's additional CDS.
While these results confirmed our a priori hypothesis that there are important safety threats at the beginning when transitioning between systems, we had anticipated that the new system's robust CDS would lead to more significant reductions in errors 1 year after transition. CDS has been shown to be effective in the inpatient setting for reducing prescribing errors; however, much less research has been done on commercial systems in the outpatient setting.10–13 Understanding the effects of these systems in actual use will be essential to mitigate safety threats, guide user training, and direct development and refinement of EHR user interfaces and embedded CDS.
Following completion of our original study, including a companion qualitative study, the information systems team made a number of refinements to the prescribing application of the EHR.14 We then conducted a follow-up study to determine the effects of these refinements and long-term use of a commercial EHR for prescribing. Our objective was to evaluate the rates and types of prescribing errors 2 years after transition and to examine the evolution of errors. We also interviewed providers to determine how their experiences and perceptions changed with time.
Materials and methods
Study design and subjects
Our original prospective study compared the rates and types of prescribing errors made by ambulatory care providers at baseline, 12 weeks and 1 year after transition. For this follow-up study, we reviewed prescriptions written by providers over a 2-week period on the commercial EHR, 2 years after transition. Study subjects were faculty providers at an academic-affiliated, urban, hospital-based adult internal medicine outpatient clinic. All faculty working 75% time or more and at least two clinic sessions per week were included. We obtained at least 75 prescriptions on 25 patients per provider, extending data collection beyond 2 weeks if necessary. We limited prescription review to three randomly selected prescriptions per patient to minimize error clustering. In addition, we conducted one-on-one semi-structured interviews to determine how physicians’ perspectives evolved in the context of system refinements and prolonged system use. While we are preparing the entire set of qualitative results as a separate manuscript, we present here qualitative results pertaining to provider perceptions of safety given their utility in informing our quantitative results. We received Institutional Review Board approval from Weill Cornell Medical College and consent from providers.
We classified errors according to Institute of Medicine definitions.15 Our classification scheme included rule violations, prescribing errors with low potential to cause harm (referred to as ‘prescribing errors’), near misses, and ADEs. Rule violations were departures from strict prescribing standards that were unlikely to result in harm (such as failure to write ‘po’ for oral medications). As in other published studies, these were not counted as errors.2 ,4 ,9 ,11 ,12 An example of a prescribing error was omitting the quantity to be dispensed for an anti-lipid medication. Near misses were errors with high potential for harm that were either intercepted or reached the patient but did not cause harm, such as prescribing amoxicillin to a penicillin-allergic patient who took the medication but suffered no reaction. Actual ADEs were injuries from a medication, some of which were associated with an error and considered preventable.
Background and setting
Transition from the older to the newer system
In previously published manuscripts, we described the transition between systems that occurred in April 2008.9 ,14 Use of the new system was mandatory. The information systems team conducted a large scale, intensive effort to transition providers, including transferring medication data between systems, providing mandatory user training, and minimizing schedules in the month after the new system was implemented.
The older system was a locally developed, PC-based EHR implemented in 1993 and developed by one of the providers at the study site. The only CDS was default formulations (the most commonly used formulation would automatically pre-populate when the medication name was entered) and provision of generic alternatives. The medication database was managed by the developer but allowed free-text ‘work-arounds.’ For additional details, please see previously published manuscripts.9 ,14
In addition to the features of the older system, the newer system has additional CDS including provision of default dosages and alerts for allergies and drug–drug interactions. Providers can create preference lists (lists of frequently used orders) and order sets (pre-populated groups of medications). The medication database and CDS logic are derived from a third party supplier.
As part of ongoing efforts to improve the e-prescribing application, a number of refinements were made approximately 1 year after transition. This included limiting alert firing across alert types so only the highest severity alerts fired, a change that primarily targeted drug–drug interactions since drug–allergy interactions were all considered severe. Other refinements included removing route abbreviations from drug descriptions, which was confusing to providers (ie, ‘Amoxicillin 500 mg OR Tab’ became ‘Amoxicillin 500 mg Tab’) and adding tall man lettering to certain medications (capitalizing letters within drug names to highlight primary dissimilarities with look-alike drug names). In addition, as a result of regulatory requirements, the dispense field, which previously was a free-text box, was changed such that prescribers have to separately enter the quantity and unit to be dispensed, which appear only in drop-down menu format. Providers were informed of these interventions via email. Minor updates to the medication database and CDS were also routinely made by the third-party supplier and incorporated on an ongoing basis.
Prescription data collection and review
Electronic prescriptions were extracted from the EHR's database for a 2-week period 2 years after transition. Prescriptions written by residents were excluded.
One experienced nurse reviewer evaluated all prescriptions. This nurse had previously been trained by RK to apply extensively utilized and standardized methodology.2 ,9 ,16–19 Methodology included error classification and identification of ADE trigger drugs.16 The same review process was used for this paper as in the previous paper, although the nurse reviewer was different. However, the two nurses jointly reviewed a set of 25 prescriptions and had 100% inter-rater agreement. Notably, our methodology is primarily designed to detect rule violations and prescribing errors, not near misses and preventable ADEs, which are best detected by chart reviews for every prescription or patient surveys.
The research nurse performed ambulatory chart reviews to confirm suspected near misses or when drugs that are often used to treat an ADE were prescribed. Two physicians then independently reviewed the events determined to be actual near misses or ADEs by the research nurse. Confirmed events were rated on preventability using a five-point Likert scale and attribution using the Naranjo algorithm.20 Severity of ADEs was rated using a four-point Likert scale. Inter-rater agreement for the presence of prescribing errors and near misses was 0.96 and 0.93, respectively, indicating excellent agreement.
For data management and descriptive statistics, we used SAS V.9.2 (SAS Institute, Cary, North Carolina, USA). This included comparisons of patient characteristics across all four time points using χ2 tests for categorical variables and analysis of variance for continuous variables. We considered findings significant at the 0.05 level. We used Poisson regression to estimate error rates at each time point while adjusting for patient characteristics (age, insurance, and gender). We compared rates at 2 years to rates at the other three time points. We calculated error rates per 100 prescriptions and 95% CIs for all rates using linear combinations of the Poisson coefficients, with patient characteristics set to their means and adjusting the SEs for covariance among the coefficients. Only providers with at least 50 prescriptions on 25 patients were included.
By means of generalized estimating equations, we adjusted for clustering, using provider as the unit of analysis. We calculated the intra-class correlations for each error at the four time points using the analysis of variance estimator. We adjusted for clustering as there were several errors above 0.05 (see online supplementary appendix table S1).21 We assumed an independent correlation structure between providers for the Poisson regression. To reduce the amount of patient clustering, we limited the number of prescriptions to three per patient during data collection. The primary Poisson model was analyzed using Stata V.10 (StataCorp, College Station, Texas, USA). Lastly, we conducted a sensitivity analysis by analyzing data for the 10 providers who contributed data to all four time points.
A separate analysis was also conducted, using the methodology described above, excluding inappropriate abbreviation errors from the overall error count. As explained in a previously published manuscript, some versions of the locally developed system automatically converted inappropriate abbreviations (those on the Joint Commission on Accreditation of Healthcare Organizations list of ‘Do Not Use’ abbreviations due to their high potential to cause errors) into acceptable abbreviations on printed prescriptions.9 ,22 However, we could not tell whether this conversion had occurred based on electronic downloads.
Qualitative data collection
We used analogous methodology to the qualitative study we performed with providers 1 year after transition except a second round of field observations was not performed. For a full description of the methodology, please refer to our previously published work.14 Briefly, we conducted semi-structured interviews exploring provider prescribing experiences, focusing specifically on how these experiences have evolved over time. We asked questions related to system refinements and followed up on themes that emerged during the first round of data collection. Interviews, which lasted between 25 and 60 min, were audio-recorded and transcribed. Participants received a $200 cash incentive.
Data analysis was carried out in an iterative manner guided by a grounded theory approach using Atlas.Ti qualitative software. Analysis was done by a multi-disciplinary research team. Expert qualitative researchers provided oversight to ensure methodological rigor. We employed well-established techniques to assess the credibility of findings, including conducting member checking from research participants.
Twenty-four providers in total participated in the study, including 16 at 2 years (table 1). Fourteen of the 16 providers (88%) participated in the interviews.
Ten providers had prescribing data from all four time points. Provider numbers varied due to changes in clinical effort, maternity leave, sabbaticals, and changes in employment.
We reviewed prescriptions for 3151 patients overall, including 920 patients at 2 years. There was no significant difference in gender across time periods, but age and insurance status varied significantly.
Rates of errors
We reviewed 1905 prescriptions 2 years after transition and 6131 prescriptions overall (table 2).
Error rates were significantly lower 2 years after transition compared to all other time points (3.8 errors per 100 prescriptions, p<0.001 compared to each time point). This was true for error rates including and excluding inappropriate abbreviation errors (1.7 and 2.0 errors per 100 prescriptions, respectively). Notably, the provider-level variation in errors rates, as reflected by the intra-class correlations, decreased progressively from baseline to 2 years after implementation. Rule violation rates were no different compared to 1 year and baseline, but were slightly higher than rates at 12 weeks (9.4 vs 6.9 rule violations per 100 prescriptions, p=0.04). Near miss rates were low (1.8 per 100 prescriptions) and not significantly different across time points. No preventable ADEs were detected.
In a sensitivity analysis conducted on the 10 providers with data in all four time periods, the above trends were unchanged for prescribing errors. There was also no longer a significant increase in rule violation at 2 years compared to 12 weeks after transition (7.6 vs 8.1 per 100 prescriptions, p=0.70) and we found a significant reduction in near misses from baseline to 2 years (2.4 vs 1.2 per 100 prescriptions, p=0.01).
Types of errors
Consistent with all other time periods, the most common type of prescribing error at 2 years was inappropriate abbreviation errors, although the number of inappropriate abbreviation and direction errors had declined substantially (table 3).
There were no amount to be dispensed errors at 2 years. Frequency, dose, and length of treatment errors had increased, although the overall number of each of these errors was small.
The most common types of near misses at 2 years were frequency and dose errors, accounting for 44.7% and 26.3% of all near misses, respectively. In contrast, direction errors were previously the most common mistakes. Examples are listed in table 4.
Class of medications involved in errors
The most common classes of medications involved in prescribing errors across all time points were anti-hypertensives (n=131, 10.7%), diabetic oral agents (n=129, 10.5%), cholesterol medications (n=95, 7.7%), and diuretics (n=81, 6.6%). At year 2 specifically, vitamins (n=9, 12.7%), inhaled bronchodilators (n=5, 7.0%), antihistamines (n=4, 5.6%), and anti-hypertensives (n=4, 5.6%) were most frequently involved. There was greater heterogeneity across all time periods regarding near misses. Selective serotonin receptor agonists, anti-malarials, and phosphodiesterase inhibitors were frequently involved. For a complete list of the drug class mix, see online supplementary appendix table S2.
Providers generally felt that prescribing safety improved over time, in large part due to increased comfort and familiarity with the commercial EHR: ‘the more you use it, the easier it is to use.’ Decreased rates of alert firing were noticeable to some providers and considered a positive change that improved safety. At 1 year after transition, 5820 alerts fired for 15 507 orders, resulting in an alert/medication rate of 0.375. The alert/medication rate was down to 0.297 (4547 alerts/15 305 medications ordered) at 2 years.
Although not the focus of this study, the change in alert severity firing resulted in a decrease in override rates of 6.3% (from 81.4% at 1 year after transition to 75.1% by 2 years after transition). As one provider commented, ‘I think that fact that they have decreased the type of alerts is huge. I think you start thinking that this is not going to be important, and then the one time that it is important you ignore it.’ Overall, however, providers still felt that most alerts were not useful and continued to emphasize the need to decrease alert sensitivity: ‘[The system] should just give warnings of a higher degree.’ Some providers even recommended that individual physicians be allowed to determine what types of alerts should fire when they write prescriptions.
With regard to other system changes, providers were very aware of the changes to the dispense field but felt these changes slowed down their efficiency by adding clicks to an already ‘click-heavy’ system. No providers discussed potential safety-related benefits from this change. Providers also supported the addition of tall man lettering to aid in medication selection.
We found a significant decrease in prescribing errors in the second year following the transition from a locally developed EHR with minimal CDS to a commercial EHR with robust CDS. Overall prescribing error rates at 2 years were very low. This study is the first long-term follow-up study to our knowledge to track rates and types of prescribing errors following transition between EHRs.
Our results have several implications. First, they suggest that iterative system refinements can help maximize safety benefits. Customization to achieve clinician-user buy-in and the need for continuous modifications are principles that have been described in a consensus statement on successful electronic order entry implementation.23 ,24 Conducting long-term analyses is critical, however, to understand the impact of iterative system changes. These changes, often used to address perceived problems, could unintentionally lead to worsening errors or decreased provider satisfaction—even if that change ultimately is beneficial.
Looking specifically at the refinements made to the EHR, two in particular may have directly contributed to the reduction in errors. A major area of work by the information systems team has been limiting alert firing as alert fatigue leading to routine overriding is a well-described problem.25–28 Limiting alerts helps to reduce alert fatigue and ensure that the most critical alerts are viewed. In a qualitative study we conducted 1 year after transition, most physicians felt the benefit of useful alerts was lost among the noise of irrelevant alerts.14 In our follow-up interviews, multiple providers positively appreciated the reduction in alerts. It is worth noting that a relatively small change in the alert firing rate—from 38% to 30% of prescriptions—was noticeable to many providers and decreased the override rate, suggesting that even a small change might help tip the balance in favor of physician tolerance and alert effectiveness. Additionally, although not explicitly described by providers, the implementation of a structured discrete dispense field likely also contributed to the elimination of ‘amount to be dispensed’ errors, of which there were 17 at 1 year and none at 2 years. The structured menu options required for prescription completion helped ensure that providers chose an amount and correct unit for dispensing.
Using our methodology, we cannot say whether there was any direct impact from the other system modifications. Tall man lettering is designed to prevent providers from inadvertently choosing ‘look-alike, sound-alike’ medications. Without reviewing cancelled prescription orders or medical event reports, we cannot say whether a provider avoided making such a mistake. In addition, removal of ‘OR’ from drug descriptions was done to help eliminate provider confusion. As there was only a single route error which occurred at 2 years, it is also difficult to say whether this intervention had any direct impact on preventing potential prescribing errors.
Identifying the most common errors can allow for specific error targeting through refinement of CDS and education of providers to improve prescribing practices. In comparing the most frequent errors made over time, inappropriate abbreviation errors remained the most common, although the overall number of these errors decreased sharply between years 1 and 2. In the commercial EHR, two separate alerts targeted this error. Several studies support the use of targeted alerts to improve specific prescribing practices.29–31 However, there is no hard stop alert, or automatic conversion of inappropriate abbreviations, to eliminate this error altogether. Directions errors also decreased considerably between years 1 and 2. Although we can only speculate, it may be that more providers began using and sharing ‘preference lists’ of medications, thereby standardizing direction fields and improving clarity. In qualitative interviews, providers reported increased use of preference lists primarily to improve efficiency; however, a secondary consequence may have been safety benefits. Additionally, the vendor-supplied updates regularly incorporated into the system may have improved direction clarity for certain medications. For example, as one provider commented, ‘I would say that the default dosing, to me, feels more accurate.’
Our results also suggest that safety benefits may require time to be realized. The commercial EHR was very effective at the beginning in targeting inappropriate abbreviation errors. However, it was only at 2 years that error rates, excluding inappropriate abbreviations, were lower than at baseline. Appropriately managing expectations concerning the time it may take to achieve demonstrable safety results will be important to avoid negativity from those hoping for early, robust safety improvements. Research suggests that unmet expectations can negatively impact provider satisfaction with a new EHR system.32
Also importantly, the overall rates of prescribing errors we found are quite low. A recent study demonstrated that 11.7% of outpatient electronic prescriptions contained at least one error.33 In the Nanji study, the most frequent error was omitted information (specifically duration or dose). As in our study, use of inappropriate abbreviations was also quite common. Other studies have also found higher error rates.2 ,12 ,34 ,35 Our error rates at year 2 may be lower for several reasons, including the fact that the commercial EHR has been extensively tailored to meet user needs or because our study was conducted long after implementation, when users had grown accustomed to system nuances. Regardless, it seems clear that a commercial EHR with e-prescribing can significantly reduce ambulatory prescribing errors, despite other studies with negative results.2 ,36
Notably, although the errors targeted by our methodology have lower potential for patient harm, these errors lead to tremendous inefficiencies for providers, patients, and pharmacies and frequently result in clinically significant delays for patients in receiving medications.37–39 For example, a study evaluating pharmacy callbacks found that callbacks for ‘acute’ medications, defined as medications where administration delays could lead to worsening of a medical condition or cause prolonged pain, were not resolved on the same day 34% of the time.37 A more recent study focusing exclusively on e-prescriptions from 68 pharmacies found that pharmacists had to intervene on prescriptions 3.8% of the time, resulting in 10% of prescriptions not being dispensed.39
While the decrease in prescribing errors over 2 years is encouraging, the rates among the near misses we did capture remained unchanged from baseline. Although our methodology is not best suited to identify near misses and ADEs, the fact that we still detected a near miss rate of 1.8 per 100 prescriptions suggests that different strategies may be needed to target near misses. Given the potential for significant harm from these errors, organizations may want to consider using ongoing error surveillance to drive targeted system changes. For example, as the most common type of near miss we detected at year 2 was frequency errors, there may need to be hard stop alerts for prescriptions with incorrect or omitted frequencies. For direction errors, as another example, developing default instructions for particular medications may help to target these mistakes.
Our research has several limitations. Our study was conducted in one site, limiting generalizability. Providers used only one commercial system, although it is a widely utilized commercial EHR. Providers were aware of the study's purpose and may have been extra cautious when writing prescriptions, making our rates underestimates. We do not have quantitative data on the specific alerts removed during the system refinements made between year 1 and 2 and so cannot connect these changes to the specific outcome changes in table 3. We had a different research nurse performing the year 2 reviews, and thus she might have made different judgments from the previous nurse. Lastly, our methodology is best suited to detect prescribing errors, not near misses or ADEs. Therefore there are likely events not captured given that other studies have found that the rate of preventable ADEs among outpatients is 4%.2
Understanding the safety effects following transition to a commercial EHR for e-prescribing will be critically important given the tremendous federal investment in the EHR Incentive Program. Our results suggest that over time prescribing errors can be reduced to very low rates through use of a commercial EHR with robust CDS. Iterative refinements, including a focus on limiting alert firing, appear to be important for maximizing safety benefits, although more targeted interventions may be necessary to reduce the most serious errors. Given the constant evolution of EHR systems, organizations should carefully monitor errors over time so that potential safety threats are managed early and prescribing errors continue to be eliminated.
We would like to thank Katherine Zigmont, RN for her help in conducting nurse reviews of the prescriptions, and Elizabeth Pfoh, MPH for her administrative assistance.
Contributors ELA was the lead co-investigator and was involved in all aspects of the research study, including research design, participant recruitment, data analysis and interpretation, and manuscript writing. SM and SNO assisted in study design, participant recruitment, data interpretation and manuscript editing. AE was involved in study design, data analysis, and manuscript editing. AC and CC assisted in study design and data collection, and edited the manuscript. RK was principal investigator and oversaw all aspects of the research, including study design, participant recruitment, data collection and analysis, and manuscript preparation.
Funding This project was supported by the Agency for Healthcare Research and Quality (R18HS017029), Rockville, Maryland, USA. The funding agency had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Competing interests None.
Ethics approval Weill Cornell Medical College Institutional Review Board approved this study.
Provenance and peer review Not commissioned; externally peer reviewed.
Conference presentation Findings from this manuscript were previously presented as a poster at the American Medical Informatics Association Annual Symposium in Washington, DC, October 24, 2011.