rss
J Am Med Inform Assoc 14:415-423 doi:10.1197/jamia.M2373
  • Original Investigation
  • Research Paper

The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry

  1. Joan S Ash,
  2. Dean F Sittig,
  3. Eric G Poon,
  4. Kenneth Guappone,
  5. Emily Campbell,
  6. Richard H Dykstra
  1. Affiliations of the authors: Oregon Health & Science University, Portland, OR; Northwest Permanente, Portland, OR; Brigham and Women’s Hospital and Harvard Medical School, Boston, MA
  1. Correspondence and reprints: Joan S. Ash, Ph.D., Department of Medica(l Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd., Portland, OR 97201-3098; e-mail: <ash{at}ohsu.edu>
  • Received 10 January 2007
  • Accepted 10 April 2007

Abstract

Background Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research.

Objective To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals.

Design, Setting, and Participants Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE.

Measurements Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE.

Results We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites.

Conclusions The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one’s perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

Introduction

Implementation of computerized provider order entry (CPOE), defined as a computer-based system that allows a physician or other ordering authority to directly enter medical orders, can be an effective mechanism for improving patient safety and practitioner performance.1 2 3 4 5 6 7 8 9 10 11 12 13 14 There can be unintended consequences as well, however,15 16 17 18 19 20 21 22 23 24 and it is likely that fear of adverse unintended consequences is impeding the diffusion of CPOE throughout hospitals in the U.S.25 26 27 28 29

The unintended consequences of CPOE are especially intriguing because they can surprise implementers. Adverse unintended consequences unfortunately can become impediments to health care quality. The terms “unintended consequences” and “unanticipated consequences” are not synonymous. The “unintended” implies lack of purposeful action or causation, while the “unanticipated” means an inability to forecast what eventually occurred. Either kind of consequence can be adverse or beneficial. Unanticipated beneficial consequences are actually happy surprises. Unanticipated, unintended adverse consequences capture news headlines and are often what people imagine when they hear the term “unintended consequences.” An important goal of the current work is to identify the types of unintended consequences, so that they can be monitored and managed: once unintended consequences are predicted or detected, their management can knowingly involve tradeoffs. An example CPOE-related tradeoff is the degree to which the extra time physicians must exert to use CPOE (an undesirable consequence), is offset by the increased information physicians find available at the point of care via CPOE—making patient visits more effective.30 The ability to maintain control over consequences may give hospital decision makers more confidence when making the determination to implement CPOE.

Recent descriptions in the literature of the unintended consequences of patient care information systems include a taxonomy of kinds of errors,31 a list of kinds of errors related to CPOE,18 a hierarchical model of unintended consequences of CPOE,30 and a more general summary of the unintended consequences of patient care information systems.19 In addition, the authors of the current study previously described in detail a typology of unintended adverse consequences of CPOE.32 However, none of the previous studies provided a comprehensive, semi-quantitative description of the extent and importance of unintended consequences in hospitals across the nation. We therefore transformed our descriptions of types of unintended consequences into survey questions to discover how widespread and important the consequences are. We tested and, using short telephone interviews, administered the resulting instrument to representatives of 561 U.S. hospitals that have reported that they have installed CPOE systems, out of a total of 5,759 accredited hospitals.33

Preliminary Work

The current study protocol for this three year-project, as proposed to the granting agency and approved by appropriate human subjects committees, included an extensive qualitative study to be followed by a national survey. In the initial qualitative study, we used semi-structured interviews and observations at five sites with experience in using CPOE to find out what types of unintended consequences exist.32 Study sites for qualitative data gathering included: Wishard Memorial Hospital, a county hospital in Indianapolis which uses a system developed by the Regenstrief Institute; Brigham and Women’s Hospital in Boston, which developed its own system; Massachusetts General Hospital in Boston, which uses a modified version of that same system; The Faulkner Hospital in Jamaica Plain, MA, which uses a commercially available system; and Alamance Regional Medical Center in Burlington, NC, which also uses a commercial product. Institutional review boards at Oregon Health & Science University, Kaiser Permanente Northwest, and each of the individual study sites approved the qualitative study.

An experienced interviewer (JSA) conducted all of the taped interviews following an interview guide of questions. Thirty-two individuals, representing clinical, administrative, and technology roles, were formally interviewed. A multidisciplinary team of Ph.D.-level informatics researchers, physicians, a nurse, and a pharmacist conducted 390 hours of observation with a purposive sample of 142 individuals representing different clinical disciplines. The team performed 120 person-hours of observation at Wishard, 144 at the three Boston sites, and 126 at Alamance. For each visit, five to six researchers spent four days at each site in the period between August 2004 and April 2005. Observations took place in medicine, surgery, emergency, and intensive care areas in addition to the pharmacy, outpatient offices, and clinics.

Qualitative data analysis followed accepted principles for assuring trustworthy results.34 35 36 37 38 All field notes and transcripts were entered into QSR N6, a qualitative analysis software program, and were individually coded by team members. The multidisciplinary team met 36 times to review transcripts and agree that what was seen or discussed was an unintended consequence and of what type.

Results of this preliminary study included identification of nine types of unintended consequences. Each of these categories can include desirable as well as undesirable consequences, depending on one’s point of view. However, our focus has been on those that are undesirable (adverse consequences), because institutions must manage them. The nine types are:

  • “More/New Work Issues: Physicians find that CPOE adds to their workload by forcing them to enter required information, respond to alerts, deal with multiple passwords, and expend extra time.

  • Workflow Issues: Many unintended consequences result from mismatches between the clinical information system (CIS) and workflow and include workflow process issues, workflow and policy/procedure issues, workflow and human computer interaction issues, workflow and clinical personnel issues, and workflow and situation awareness issues.

  • Never Ending Demands: Because CPOE requires hardware technically advanced enough to support the clinical software, there is a continuous need for new hardware, more space in which to put this hardware, and more space on the screen to display information. In addition, maintenance of the knowledge base for decision support and training demands are ongoing.

  • Paper Persistence: It has long been hoped that CIS will reduce the amount of paper used to communicate and store information, but we found that this is not necessarily the case since it is useful as a temporary display interface.

  • Communication Issues: The CIS changes communication patterns among care providers and departments, creating an “illusion of communication,” meaning that people think that just because the information went into the computer the right person will see it and act on it appropriately.

  • Emotions: These systems cause intense emotions in users. Unfortunately, many of these emotions are negative and often result in reduced efficacy of system use, at least in the beginning.

  • New Kinds of Errors: CPOE tends to generate new kinds of errors such as juxtaposition errors, in which clinicians click on the adjacent patient name or medication from a list and inadvertently enter the wrong order.

  • Changes in the Power Structure: The presence of a system that enforces specific clinical practices through mandatory data entry fields changes the power structure of organizations. Often the power or autonomy of physicians is reduced, while the power of the nursing staff, information technology specialists, and administration is increased.

  • Overdependence on Technology: As hospitals become more dependent on these systems, system failures can wreak havoc when paper backup systems are not readily available.”39

Methods

Survey Development

We developed an interview script, which included a short description of the purpose of the survey, five questions about the kind of CPOE system in place, and eight questions about possible unintended consequences. The five questions about the system were designed to collect descriptive information about the degree to which the system is infused into the institution. Infusion of technology is defined as the extent to which one uses an innovation in a complete and sophisticated manner,40 41 and it occurs as information technology applications become more deeply embedded within the organization’s work systems.42

Though we had previously discovered nine types of unintended adverse consequences, we decided not to ask interviewees about one of them, paper persistence, because as the role of paper has changed from a long-term storage medium to a temporary memory aid and disposable display device, its persistence should not now be considered adverse. The unintended consequences typology developed from the qualitative study was therefore transformed into eight survey questions, which were pilot tested (see Table 1). The questions were designed to be as neutral as possible to avoid bias. We prefaced the questions by stating that the study concerned “what organizations have learned about CPOE.” We asked about surprises, “things that happened that you didn’t expect,” so the focus was on both unintended and unanticipated consequences; responses might then involve either desirable (beneficial) or undesirable (adverse) effects. The questions were asked first as yes/no queries to determine if the respondent thought the unintended consequence existed at that site (0 = No). If the answer was yes, the importance was determined on a 1 to 5-point Likert scale ranging from 1 for “not at all important” to 5 for “very important.” We received institutional review board approval from Oregon Health & Science University and Kaiser Permanente Northwest to conduct the survey.

Table 1

Unintended Consequences Survey Questions*

Survey Administration

We attempted to contact the entire population of U.S. hospitals using CPOE, instead of a sub-population from which we would make statistical inferences, because we wanted to gain insight into the nature of unintended consequences related to CPOE from organizations beyond those we visited. We selected acute care hospitals from the 2004 HIMSS Analytics™ Database that reported having CPOE in place (N= 448). Since that database did not include U.S. Veterans Affairs hospitals, which the authors believed to constitute important models of CPOE use, we also surveyed them (N= 113). We contacted hospital staff listed as appropriate contacts via electronic mail and then made phone appointments for the 10 to 20 minute survey. Follow-up included multiple phone calls until a knowledgeable informant was found. Each question asked the interviewee to respond on a 5-point Likert scale about the extent of the problem; researchers took notes on anything that was volunteered beyond that. Numerical data were analyzed with descriptive statistics and the text of comments was analyzed qualitatively, again with the aid of N6. Whenever a respondent spontaneously commented while answering a question, the interviewer wrote down the comment. At the end of the interview, we asked if the interviewee had anything else they would like to share, and these comments were also recorded.

Survey Data Analysis

Quantitative data were analyzed using SPSS software (V14.0, SPSS Inc. Chicago IL). We used descriptive statistics to understand the degree of CPOE infusion in the institutions. We explored the correlation between the length of time CPOE has been in place and the extent to which unintended consequences were present. Using bed size and ownership as established proxies for “other important organizational characteristics in studies of hospital organizational issues,”43 as well as geographic location and teaching status, we compared respondents to non-respondents using logistic regression.

Qualitative data in the form of notes taken during the interviews were analyzed using QSR N6. This analysis simply entailed grouping comments with the questions to which they related, each representing a type of unintended consequence.

Results

We were able to establish contact with 265 of the 561 hospitals and discovered that 89 of those listed in the HIMSS Analytics™ database did not actually have CPOE, so they were dropped from further consideration. We therefore conducted 176 full interviews. Thirty-four additional institutions had policies against responding to surveys, so they were added to the list of non-respondents. Using the Institute for Social and Economic Research (ISER) calculation for response rate,44 which takes into consideration the ineligibility of some sites (e.g., listed as having CPOE when they actually do not), our response rate was 47%. To estimate how representative our responding sites were of all hospitals that have CPOE, we categorized the sites in four ways: bed size, management type, teaching hospital status, and geographic location45 because these measures are considered important differentiators.43 Logistic regression (forward, stepwise, likelihood ratio) comparing respondents and non-respondents showed no difference with respect to hospital bed-size and teaching hospital status. Management type differed between respondents and non-respondents, in that the VA hospitals were most likely to respond, followed by other governmental hospitals, private institutions and non-profits (compared to VA, governmental LR = 0.63, private not for-profit LR = 0.38 and private for-profit LR = 0.35; p < 0.001). There was also a difference in geographic location with a site in the Northeast region most likely to be a responder, followed by the West, South and the Midwest, in that order (compared to South, Northeast LR = 2.11, West LR= 1.65 and Midwest LR = 0.93; p = 0.007).

Infusion Levels

The length of time that CPOE had been in place ranged from 6 months to 25 years (median = five years). The percentage of orders entered electronically ranged from 1–100% (median = 91%). Greater than 96% of the sites used CPOE to enter pharmacy, laboratory, and imaging orders; 82% were able to access all aspects of the clinical information system with a single sign-on; 86% of the respondents had at least three types of clinical decision support (order sets, drug-drug interaction warnings, and pop-up alerts); and 90% had a CPOE committee in place. The overall infusion levels were therefore at the high end along all measures.40

Extent of Unintended Consequences

Table 2 summarizes the responses to the eight questions about unintended consequences. For six of the questions, the responses indicated that the unintended consequences were important in those institutions. Workflow was rated as a 3, 4, or 5 by 88%, communication 84%, dependence on technology 83%, system demands 82%, emotions 80%, and more work or new work 72%. For two of the questions, many respondents answered that the unintended consequence had not been seen or was less important than others. Changes in the power structure was rated as 0 by 36% of respondents, and either 1 or 2 by 29%. New kinds of errors were rated 0 by 9% and either 1 or 2 by 44%. We considered that the number of years that CPOE had been in place might affect the occurrence of these consequences to either a lesser or greater extent. The response rating for these questions was evaluated for a correlation between it and duration of use in years and no correlation was found for any of the eight questions (Spearman’s rho statistic).46

Table 2

Responses to the Survey Questions Regarding Unintended Consequences*

Types of Unintended Consequences

Sixty-four pages of qualitative data were analyzed from the free text responses, with representative quotes from respondents shown in Table 3. Comments are summarized below:

  • More/New Work: Since large unanticipated changes in the amount and nature of work seem to be a regular consequence of CPOE, we asked how great a problem it is hospital-wide and discovered there are many kinds of new work generated. Over 72% said it is a moderate to very important issue. Work increases because 1) sometimes the systems are slow, 2) non-standard cases call for more steps in ordering, 3) there are training issues, 4) some tasks become more difficult, 5) with more data available more is done, 6) nurses have to be diligent about taking orders off regularly, 7) the computer “forces the user to complete all steps,” and 8) information technology staff take on many new responsibilities. Respondents often noted that there were also marked shifts in work, so that the physician must do tasks formerly done by others.

  • Workflow: We asked if CPOE altered workflow and to what extent. 87% of respondents said that it is a moderate to very important issue. Many commented that CPOE increased efficiency, which is generally an expectation and therefore not an unintended consequence. Often, however, interviewees noted that while it improved workflow for some workers, it also negatively affected the workflow of others, especially physicians, who must put extra time into ordering. Some commented that it is hard to identify the effect of CPOE vs. other technological advances in communications such as cell phones, and others noted that the workflow was negatively affected in the beginning but positively affected later on.

  • System Demands: We asked about the maintenance, training, and support efforts that are ongoing and the extent of the problem. 82% of informants said it is a moderate to very important problem. Demands are greatest during the first few months, but even after that “it takes an army to build and maintain” a system, even a commercial one. There is need for continuous equipment upgrades. A positive yet unintended consequence described often is “demand from the organization exceeding our supply,” meaning that expectations escalate or, as another interviewee said, “there is such an appetite for continued growth.” One informant strongly stated “do not do this unless you can support it.”

  • Communication: The next question asked about alterations in communication patterns among clinical staff, with 84% of respondents noting that this was an important outcome of CPOE, but not necessarily for the better. Beliefs were strong one way or the other, with about half of the comments indicating that CPOE has cut down on the need for clarifying orders and the other half saying that people no longer communicate adequately. Many respondents commented that their organizations had to make a concerted effort to educate clinicians about using face-to-face communication when it is appropriate and not to rely solely on the system.

  • Emotions: We asked if users tend to express strong feelings about CPOE and 80% said it is a moderate to very important issue. Many interviewees noted that reactions go “in both directions.” One informant said “pick your favorite terms of praise or profanity. They are all used.” Physicians get upset because it takes more of their time, but most tend to accept CPOE over time. The strength of the emotions is often unanticipated, along with the change over time from strongly negative to strongly positive.

  • New Kinds of Errors: Although CPOE has been proposed as a solution to the medical errors problem, there are also new kinds of errors generated as a result of computerization. We asked if respondents had seen new patient safety issues with CPOE. 9% said it was not a problem and only 47% said it is moderately to very important. The respondents cited entering orders on the wrong patient, errors of omission, nurses not knowing an order had been generated, desensitization to alerts, loss of information during care transitions, wrong medication dosing, and overlapping medication orders. There were many comments that safeguards have been built into the systems, that there are many near misses but few actual errors and they tend to be minor, and that there is “a potential for new issues [errors] so awareness is vital.”

  • Power Shifts: We asked how the balance of power shifts when CPOE is used. Although shifts in the power structure were clearly evident during our fieldwork, our interviewees did not feel they were very important. 36% said it is not a problem, 29% said it is a less important issue, and only 36% thought it was a moderate to very important issue. They noted that power flows away from physicians to pharmacists, nurses, information technology staff, and administration. Some said it is a phenomenon that occurs early during the implementation effort, that CPOE is used as a way of forcing or monitoring physician behavior. Others said that there is “not so much a change in the balance of power, but it does shine a light on how decisions are made in a very public way.”

  • Dependence on the System: In fieldwork, we found that dependence on CPOE and surrounding systems is expected, but the extent of this dependence was an unintended outcome. Clinicians eventually become extremely dependent on the system, so downtime creates problems. We therefore asked about consequences when the CPOE becomes unavailable so that we could discover the extent of the problem. This is indeed an issue for over 82% of respondents and comments indicated that it gets worse over time. Most interviewees said that backup processes go into effect if there is downtime, but that “it is basically managed chaos with a very negative impact on productivity.”

Table 3

Comments from Respondents

Discussion

Our preliminary qualitative study identified major types of unintended adverse consequences of CPOE. This survey study goes beyond the prior study describing these types by advancing our knowledge about how widespread the problems are. The survey results verified the existence of the eight unintended adverse consequences types we asked about at most of the 176 hospitals with CPOE, and analysis of comments has offered insight into the perceived importance of the consequences. All types of consequences are indeed widespread although two of them, power shifts and new kinds of errors, were not considered as important as the others. While the current study was based on a categorization of adverse unanticipated consequences of CPOE from the prior study, we found that there are positive as well as negative unintended consequences. Often, the same consequence can be viewed different ways by different people depending on their perspectives.

Our mixed methods approach, using extensive fieldwork to gather data about types of unintended consequences at excellent sites, followed by a phone survey to verify the existence of these same types across the country and to gain further insight, yielded rich data.

The median length of time organizations have had CPOE in place is five years, which is relatively recent since systems have been available since the early 1970’s,47 although only a few pioneering hospitals had it prior to ten years ago. Interestingly, there appears to be no correlation between the importance of each type of unintended consequence and the length of time that a system has been in place. Despite the recent timing of many implementations, the systems are highly infused, indicating that they are deeply embedded into the workflow, heavily used, and sophisticated.

One limitation of this study is the difficulty of asking for a simple score in response to a complex question. This difficulty was offset by allowing respondents to provide comments after each question—a distinct advantage of telephone surveys. The ability of the researcher to give examples and explain questions was helpful, but it is possible that some respondents may have misinterpreted some questions. There was no attempt to test the reliability and validity of the instrument. We asked to speak to someone in the organization who is most knowledgeable about CPOE, and we usually spoke to someone on the information technology staff with a clinical background. While these individuals seemed to offer balanced answers, they might have given answers biased in a positive way because they believe in CPOE’s benefits. This might help explain why the two categories of power shifts and new kinds of errors did not rank highly: often the power shifted to administration or information technology staff, so informants may not recognize their own gain in power. New kinds of errors may seem minor to these interviewees who seem confident that these errors are most often caught before harm is done. In addition, as representatives of these hospitals, informants may have been reluctant to discuss the sometimes-sensitive subject of errors in heath care. As previously noted, there was a bias toward eliciting adverse unintended consequences, based on the survey taxonomy being derived from a prior study of adverse consequences.

Responding sites tended to differ from non-respondents by management type, with VA hospitals the most likely to respond and for-profit centers, the least likely. Perhaps VA centers are proud of their systems and/or staff members are simply more willing to take the time to talk about them. Unfortunately, many organizations have policies against doing surveys, an increasingly common problem.48 The reason for geographic differences in the likelihood of response is unclear. As stated, there was no difference in the level of response with respect to bed size or whether the site was a teaching institution.

Once unintended consequences are revealed, hospitals can work to either try to avoid them or to accept them as tradeoffs. For example, the negative impact of information technology on clinical workflow noted by others49 50 51 52 is especially true of CPOE because it takes more of the physician’s time to enter orders than in the manual process,53 54 but this might be accepted as a tradeoff for the added safety benefits of CPOE.

Our chief recommendation is that developers and implementers of CPOE systems consider each of the eight types of unintended adverse consequences carefully during their planning. The workflow issues are immense, and they affect nearly every hospital staff member. Communication changes,55 and implementers must take steps to prevent or offset the altered communications created by electronic systems in this respect. As people become more and more dependent on such systems,56 preparations for downtime must be more elaborate and serious. More work and new work are inevitable, as are increasing system demands, so ample resources must be made available on an ongoing basis. Finally, emotions run high for and against CPOE,57 and efforts must be made to understand and manage them. Expectation management is therefore paramount. Informants indicated that power shifts and new kinds of errors were less important kinds of unintended consequences, but we must nevertheless be diligent about identifying them.18 58 59 60 61 62 63 Unless we make a concerted effort to avoid, manage, and/or overcome unintended consequences, the implementation of clinical information systems may lead to detrimental results.64 65 66 67

The eight kinds of unintended adverse consequences of CPOE we investigated are experienced by most CPOE sites, regardless of length of use of CPOE. CPOE is highly infused, or deeply embedded, at most of these sites, which is surprising considering that most implementations of CPOE are relatively recent. The telephone survey has given us insight into the extent and importance of unintended consequences. Management of possible unintended consequences is vital for maintaining control and avoiding problems. Implementation success depends on managing the unintended, as well as the intended, consequences, of CPOE.

Footnotes

  • This research was supported by grant LM06942 from the National Library of Medicine, National Institutes of Health, titled Overcoming the Unintended Consequences of Computerized Physician Order Entry Implementation. Emily Campbell and Dr. Guappone were also supported by National Library of Medicine training grant ASMMI0031. The authors extend special thanks to Caroline Wethern, Cody Curtis, and Ashley Jones for their assistance with data collection and organization.

References

Free Sample

This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JAMIA.
View free sample issue >>

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Open Access fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.