rss
J Am Med Inform Assoc 19:360-367 doi:10.1136/amiajnl-2011-000289
  • Research and applications
  • Focus on health information technology, electronic health records and their financial impact

Medication administration quality and health information technology: a national study of US hospitals

Editor's Choice
  1. Denise L Anthony3
  1. 1Tuck School of Business, Dartmouth College, Hanover, New Hampshire, USA
  2. 2Dartmouth College, Hanover, New Hampshire, USA
  3. 3Department of Sociology, Dartmouth College, Hanover, New Hampshire, USA
  1. Correspondence to Dr Ajit Appari, Tuck School of Business, Dartmouth College, 100 Tuck Hall, Hanover, NH 03755, USA; ajit.appari{at}dartmouth.edu
  • Received 29 March 2011
  • Accepted 26 September 2011
  • Published Online First 28 October 2011

Abstract

Objective To determine whether the use of computerized physician order entry (CPOE) and electronic medication administration records (eMAR) is associated with better quality of medication administration at medium-to-large acute-care hospitals.

Data/study setting A retrospective cross-sectional analysis of data from three sources: CPOE/eMAR usage from HIMSS Analytics (2010), medication quality scores from CMS Hospital Compare (2010), and hospital characteristics from CMS Acute Inpatient Prospective Payment System (2009). The analysis focused on 11 quality indicators (January–December 2009) at 2603 medium-to-large (≥100 beds), non-federal acute-care hospitals measuring proportion of eligible patients given (or prescribed) recommended medications for conditions, including acute myocardial infarction, heart failure, and pneumonia, and surgical care improvement. Using technology adoption by 2008 as reference, hospitals were coded: (1) eMAR-only adopters (n=986); (2) CPOE-only adopters (n=115); and (3) adopters of both technologies (n=804); with non-adopters of both technologies as reference group (n=698). Hospitals were also coded for duration of use in 2-year increments since technology adoption. Hospital characteristics, historical measure-specific patient volume, and propensity scores for technology adoption were used to control for confounding factors. The analysis was performed using a generalized linear model (logit link and binomial family).

Principal findings Relative to non-adopters of both eMAR and CPOE, the odds of adherence to all measures (except one) were higher by 14–29% for eMAR-only hospitals and by 13–38% for hospitals with both technologies, translating to a marginal increase of 0.4–2.0 percentage points. Further, each additional 2 years of technology use was associated with 6–15% higher odds of compliance on all medication measures for eMAR-only hospitals and users of both technologies.

Conclusions Implementation and duration of use of health information technologies are associated with improved adherence to medication guidelines at US hospitals. The benefits are evident for adoption of eMAR systems alone and in combination with CPOE.

Introduction

Health information technologies (ITs) are at the center of the debate on improving quality and reducing cost in the US healthcare system. Federal incentives linked to the adoption of electronic health record (EHR) systems require organizations to demonstrate “meaningful use” of the technologies. Computerized physician order entry (CPOE) and electronic medication administration records (eMAR) are significant components of an EHR capable of meeting future meaningful use objectives.1 The Institute of Medicine has estimated that at least 1.5 million preventable adverse drug events occur annually in the USA, excluding errors of omission—failure to prescribe evidence-based medications that are likely to reduce morbidity and mortality.2 Advocates of health IT contend that the widespread use of systems such as CPOE and eMAR will improve the efficacy of care delivery and help with the challenges of medication management.3 However, recent studies have found inconsistent evidence supporting the association of EHR systems with quality and cost. In one such study, DesRoches and colleagues4 underlined the need to examine targeted use of EHR system components. Their results echo several studies, based on both national cross-sectional analysis and longitudinal analysis, suggesting that use of health ITs is only modestly associated with better quality.4–9

Over the past two decades, six systematic syntheses of prior work on the effect of health IT (including EHR systems, CPOE, clinical decision support systems, e-prescribing, telemedicine, and administrative functions) found some evidence of benefit.10–15 While, according to Buntin et al,10 there is growing evidence of a positive association between use of health IT and quality of care, particularly among studies using recent data, two recent meta-syntheses of review literature published during 1994–2010 (Black et al11 and Lau et al12) suggested that the gap between the postulated and empirically demonstrated benefits of health IT is still significant, and encouraged application of robust research methodologies to evaluate health IT against a comprehensive set of performance measures. Technologies such as CPOE in conjunction with clinical decision support systems have been shown to improve quality in specific organizational settings such as large academic hospitals that had developed in-house systems over long periods of time.6 7 13 14 16 17 However, critics argue that such contextual evidence may not be generalizable, because commercially available systems may vary significantly from technologies developed at pioneering institutions.8 For example, there is recent evidence of potential disadvantages in using systems such as CPOE that may lead to the introduction of unintended medication errors, potentially harming patients.18 Further, it is not simply technology alone but also experience in using the technology that matters for quality.18 19

From a policy perspective, such concerns are important given that the adoption and use of health IT in US hospitals is still quite varied. In 2009, Jha and colleagues reported that only 1.5% of US hospitals had what they term a “comprehensive” EHR system, with an additional 7.6% having a basic system.20 A subsequent survey in 2010 showed a marginal increase in EHR adoption—about 2.7% for comprehensive EHR and 7.9% for basic EHR system.21 Not surprisingly, small, public and rural hospitals are much less likely to have EHR systems. Moreover, 40% of US hospitals do not even have implementation plans for technologies such as CPOE.20 21 The uneven and non-random distribution of EHR systems in US hospitals means that simply looking at the relationship between presence, or better yet comprehensiveness, of EHR systems and quality can be problematic, even when controlling for hospital characteristics.16 22 In this retrospective observational study, our primary purpose is to examine the relationship between health IT use, measured by both presence of specific systems and the duration of their use, and adherence to medication guidelines for select conditions including acute myocardial infarction (AMI), heart failure (HF), pneumonia (PN), and surgical care infection prevention (SCIP). We focus on two core technologies supporting both ends of the medication management process: CPOE systems used for ordering medications, and eMAR systems used for administration of medications. In this study, we used a national sample of 2603 medium-to-large non-federal acute-care hospitals. We excluded small hospitals (<100 beds), which are least likely to have any IT systems. We also excluded use of pharmacy system technologies that support intermediary steps of the medication management process (eg, dispensing of medication) as they are already ubiquitous among medium-to-large hospitals (>99% had adopted). We analyzed the data using a generalized linear model with logit link to perform binomial logistic regression of medication quality measured during 2009 against health IT adoption status by 2008, controlling for several hospital characteristics. Regressing 2009 quality on IT adoption through 2008 ensured at least a 1-year lag between technology adoption and quality measurement, enabling us to tease apart the impact of technology use on hospital quality by comparing the adopters with non-adopters.

This study contributes to the literature in two important ways. First, as compared with earlier research, our results offer more consistent evidence for the association of specific health ITs (eMAR and CPOE) used in a targeted clinical area (ie, medication management process in acute-care setting) with a multitude of quality indicators related to the medication administration. Second, this study shows that the duration of technology use is associated with better performance on medication quality measures (learning by doing).

Methods

Design

We performed a retrospective cross-sectional analysis to determine whether the use of CPOE and eMAR by 2008 is associated with better quality of inpatient medication administration in 2009. The study used a national sample of 2603 non-federal acute-care hospitals to test two hypotheses with reference to 11 process quality measures. The quality measures, advanced by the Hospital Quality Alliance for inpatient settings, monitor utilization of medications for medical conditions including AMI, HF, and PN, as well as SCIP. Our primary hypothesis was that hospitals using eMAR and CPOE by 2008 would perform better on 2009 medication-related process quality measures compared with non-adopters. Recognizing that effective use of eMAR and CPOE depends on many factors, including organizational learning, simply examining the presence or absence of these technologies is not sufficient. Therefore we tested a second hypothesis to determine whether the duration of technology use by 2008 was associated with better performance in 2009.

We used a generalized linear model with logit as the linking function to perform binomial logistic regression. Analysis was conducted using Stata V.11.1. This specialized regression methodology is appropriate for the quality measures, as they are all proportions and bounded within zero and one. We conducted the analyses in two steps for 11 process quality measures as dependent variables measuring hospitals' adherence to medication-related quality for AMI, HF, PN, and SCIP. In the first step, to test the first hypothesis, we used three mutually exclusive indicator variables representing adopters of eMAR only, adopters of CPOE only, and adopters of both eMAR and CPOE. Our choice of these indicators, unlike the use of separate dummy variables for eMAR and CPOE in prior research,6 7 16 allowed us to differentiate hospitals in terms of their extent of technology in the medication management process as well capture the inherent staging of health IT adoption. In the second step, to test our second hypothesis, we used multivalued categorical variables representing duration of use (in 2-year multiples) by 2008 for adopters of eMAR only, adopters of CPOE only, and adopters of both eMAR and CPOE.

In each regression, non-adopters of both CPOE and eMAR were the reference group. To account for confounding effects, several hospital characteristics were included in the models: teaching status, profit status, rural location, qualification for disproportionate share payment, transfer-adjusted case mix index, bed size, and multihospital system membership. Additionally, regression errors were standardized at the hospital referral region level (representing the local market) to account for potential bias arising from any market level factors. Observational studies such as ours examining the relationship between hospital quality and health IT may suffer from endogeneity (selection bias),5 6 9 as high-quality hospitals may be more likely to adopt health IT.23 Past research has used several analytic strategies to overcome this methodological issue, such as panel data analysis, difference-in-difference approach, instrumental variable approach, and propensity-score-based adjustment.5 6 8 9 24 25 However, most of these studies report no significant difference in results after such adjustments, suggesting that endogeneity is not a significant concern. Yet, following this body of research, we leveraged a propensity-score-based adjustment to account for any potential estimation bias. First, we estimated IT adoption propensity scores using logistic regression analysis for health IT adoption against hospital characteristics as covariates. Subsequently, hospitals were classified using five indicators corresponding to the five quintiles of the propensity score distribution. We then included these indicators in each regression to control for endogeneity of IT use with quality.26 27 Finally, to test our second hypothesis, as several hospitals did not report adoption dates for eMAR and CPOE systems, we conducted the analysis on censored data—that is, by ignoring hospitals with missing observations on duration of use; and on imputed data, in which we derived duration of use by performing multiple imputations (n=25) using ordered logistic regression with select hospital characteristics as predictors to estimate the missing duration of use.28 Although the results based on the two approaches were qualitatively similar, we report the results from the censored data.

Setting

Data were drawn from three secondary sources. The Health Information and Management Systems Society (HIMSS) Analytics Database was licensed from HIMSS Analytics based on their survey of 5281 non-federal acute-care hospitals in the USA (December 2010 release update), which includes hospital characteristics and the operational status of clinical IT applications. In discussing advantages of HIMSS, McCullough (2008) noted, it “is the most comprehensive database of hospital IT adoption decisions” in the USA and has been available since the late 1980s.29 HIMSS follows an annual process to update the database, which involves initial data gathering conducted by phone followed by an IT inventory survey completed by hospital administrators. HIMSS provides benchmarking reports to respondents as an incentive for participation.29 The HIMSS Analytics data are used extensively by IT vendors and have been used widely in health services research.5–9 16 30–33 Some researchers have pointed to inconsistencies and low response rates to a different annual HIMSS survey of hospital chief information officers at about 200 hospital systems.20 We did not use these survey data in our analysis. In any given year, the HIMSS Analytics Database that we used represents nearly all of the 100+ bed non-federal hospitals (and >90% of all US hospitals).29

The Centers for Medicare and Medicaid Services (CMS) Hospital Compare Database provided a medication-related process quality measure for 3470+ non-federal acute-care hospitals collected during January–December 2009 (September 2010 release). To ensure a lag of at least 1 year between quality measurement and technology adoption, the adoption status for eMAR and CPOE in the 2010 HIMSS Database was censored at 2008. The CMS Acute Inpatient Prospective Payment System impact file (FY 2010 release) provided hospital characteristics data which were used as control variables.

Data collection

We constructed a national sample of acute-care hospitals by merging hospital level data from these three sources. We excluded all hospitals having fewer than 100 licensed beds, consistent with prior research.6 16 34 35 This allowed us to focus on hospitals most likely to invest in IT.20 21 Larger hospitals are also more likely to have the patient volume necessary for valid CMS quality measures.6 16 36

Measurement of IT for medication management

The HIMSS Database (December 2010 update) was used to code hospitals as users of eMAR and CPOE technologies with binary indicators. Hospitals were coded as technology users if they had reported these technologies to be in operational use by 2008 or before. Any hospital reporting these technologies to be in operational use after 2008 was recoded as a non-user. For analysis purposes, health IT use was operationalized through three mutually exclusive indicators: (1) eMAR-only adopters; (2) CPOE-only adopters; and (3) adopters of both technologies. The non-adopters of both CPOE and eMAR technologies were the reference group. Unlike prior research,37–41 which primarily focused on the effects of CPOE alone (or occasionally in combination with clinical decision support), this approach allowed us to better represent the extent of IT use in the ordering and administering phases of medication management.42 Although a typical medication management process at hospitals comprises ordering, dispensing, and administering of medications2 and is correspondingly supported by CPOE, a pharmacy information system, and eMAR, we ignored pharmacy information systems because they offer no meaningful variation (99.6% of the hospitals in our sample have such a system in operational use).

Additionally, to capture the effect of experience gained in using a system, hospitals were coded on an ordered scale (for each of the four technology categories) to represent duration of technology use in mutually exclusive 2-year increments: 0 for non-adopters; 1 for up to 2 years; and 2, 3, 4, and 5 for >2 to 4, >4 to 6, >6 to 8, and >8 years use, respectively. Subsequently, this variable was multiplied by the three technology adoption indicators to derive mutually exclusive, multivalued duration of technology use variables for eMAR-only adopters, CPOE-only adopters, and both eMAR and CPOE adopters, with non-adopters being the reference group.

Measurement of medication process quality

Our dependent variables were 11 evidence-based process measures for conditions including AMI, HF, and PN, as well as SCIP obtained from the CMS Hospital Compare Database (September 2010 release), which is based on data collected during January–December 2009. To ensure reliability of quality performance data, we ignored hospitals that reported quality scores for fewer than 25 eligible patients.6 43 44

Measurement of hospital characteristics

Lastly, the 2009 CMS Acute Inpatient Prospective Payment File was used for control variables on specific hospital characteristics. Consistent with prior research,4–6 44 we controlled for hospital size (natural log of number of licensed beds), transfer-adjusted case mix index (ratio), teaching status (1 if teaching, 0 otherwise), profit status (1 if for-profit, 0 otherwise), rural location (1 if rural, 0 otherwise), membership in a multihospital system (1 if yes, 0 otherwise), and whether a hospital qualified for Medicare disproportionate share adjustment (1 if qualified, 0 otherwise). In addition, we controlled for any hospital level learning effect with the natural log transform of condition-specific cumulative patient volume in the past 5 years—that is, 2004–2008.45–47

Results

Descriptive statistics

In our sample of 2603 hospitals, 30.9% (n=804) had both the eMAR and CPOE systems in operational use, 37.9% (n=986) had the eMAR system alone, 4.4% (n=115) had the CPOE system alone, and 26.8% (n=698) had neither technology by 2008 (table 1). The overall adoption rate for eMAR and CPOE systems was 68.8% (n=1790) and 35.3% (n=919), respectively. Among the eMAR and CPOE users, about 44% (n=788) and 39% (n=357) hospitals, respectively, did not report the year of technology adoption. Using historical data releases from HIMSS for the 1990s up to 2007, we were able to determine the year of adoption for some missing observations (128 hospitals for eMAR and 68 hospitals for CPOE) leading to about 33.4% (n=598) and 27.3% (n=251) hospitals, respectively, with missing adoption years for eMAR and CPOE. Among hospitals reporting the adoption year, the majority adopted eMAR and CPOE systems during 2003–2004 (eMAR, 27%; CPOE, 26%) and 2005–2006 (eMAR, 16.9%; CPOE, 17.0%). To verify differences in quality performance between hospitals reporting technology adoption year and those not reporting, we conducted a two-tailed t test and found no systematic differences (see table A1 in the online appendix).

Table 1

Adoption and duration of use of eMAR and CPOE technologies in medium-to-large acute-care hospitals in 2008 (N=2603)

Table 2 summarizes the characteristics of all acute-care hospitals in the sample. Among the 2603 hospitals, 37.5% were teaching, 19.1% were for-profit, 72.2% were affiliated with a multihospital health system, 16.2% were located in rural areas, and 81.1% qualified for disproportionate share payment. Finally, the majority of hospitals had between 100 and 299 beds (100–199 beds, 39%; 200–299 beds, 24%), and the mean transfer-adjusted case mix index was 1.49 (SD=0.24). Hospitals with eMAR systems (n=1790), on the one hand, were similar to the overall sample of hospitals on most characteristics, except that eMAR adopters had a higher transfer-adjusted case mix, varied by bed size, and were less likely to qualify for disproportionate share payments. On the other hand, CPOE adopters were more likely to be teaching, non-profit, urban hospitals, and had higher transfer-adjusted case mix.

Table 2

Characteristics of medium-to-large acute-care hospitals (n=2603)*

Not all of the 2603 hospitals reported on all 11 quality measures (table 3). The number of hospitals reporting quality measures varied from a low of 1071 (for AMI3: eligible AMI patients given angiotensin converting enzyme (ACE) inhibitor or angiotensin receptor blockers (ARB) for left ventricular systolic dysfunction) to a high of 2296 (for PN5: eligible PN patients given initial antibiotics within 6 h of arrival). The overall sample averages for reporting hospitals on each quality measure during January–December 2009 ranged from 92.17% to 98.71%. For example, on average 98.71% (SD=1.61%) of eligible AMI patients (with no known contraindications) were given aspirin upon arrival, whereas 92.19% (SD=6.39%) of eligible surgery patients received treatment to prevent blood clots within 24 h before or after selected surgeries. The average quality scores for reporting hospitals with eMAR systems ranged over 92.57–98.82%, and without eMAR systems ranged over 90.95–98.40%. Likewise, the average quality scores among reporting hospitals with CPOE systems ranged over 92.83–98.85%, and without CPOE systems ranged 91.58–98.64%. While the quality scores for our sample averaged in the 90s, hospital level scores were distributed over large ranges: nine of 11 measures had minimum scores from low 10s to 50s, and the remaining two measures had minimum scores of 67% and 76% (for brevity, the range of quality scores is not shown in table 3). For example, in the case of PN, the proportion of eligible patients given the most appropriate initial antibiotics varied from 10% to 100% among 2279 reporting hospitals, and in the case of AMI, the proportion of eligible patients given ACE inhibitor or ARB for left ventricular systolic dysfunction varied from 67% to 100% among 1071 reporting hospitals. These data show that the adherence to medication-related guidelines not only varies substantially across hospitals, but also the number of hospitals reporting quality of care data varies across measures.

Table 3

Summary statistics of observed medication process quality at medium-to-large acute-care hospitals in 2009 (n=2603)

Association of eMAR and CPOE use with medication process quality

Our results are summarized in figure 1, which shows the adjusted ORs (AORs) and 95% CIs for all the measures in each group (eMAR-only, CPOE-only, and with both technologies) relative to non-adopters of both technologies (reference group). We found that hospitals that implemented eMAR only performed better on 10 of 11 process quality measures, relative to non-adopters of both eMAR and CPOE. For these hospitals, the odds of adhering to recommended medication guidelines were 14–29% higher than for non-adopters. For example, the odds of giving aspirin to AMI patients at admission were 29% (AOR 1.29; 95% CI 1.11 to 1.48) higher, whereas the odds of administering preventive antibiotic(s) within 1 h before incision for surgical patients were 14% (AOR 1.14; 95% CI 0.98 to 1.34) higher among adopters of eMAR-only systems compared with non-adopters. Only the measure tracking the prescription of β blockers at discharge to AMI patients showed no statistically significant difference between the two groups of hospitals.

Figure 1

Adjusted ORs (95% CIs) for electronic medication administration records (eMAR) and computerized physician order entry (CPOE) adoption on medication process quality at medium-to-large acute-care hospitals in the USA (n=2603). AMI1: given aspirin at admission; AMI2: prescribed aspirin at discharge; AMI3: given angiotensin converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) for left ventricular systolic dysfunction; AMI5: prescribed β blocker at discharge; HF3: given ACE inhibitor or ARB for left ventricular systolic dysfunction; PN5: given initial antibiotic(s) within 6 h of arrival; PN6: given most appropriate initial antibiotic(s); SCIPINF1: received preventive antibiotic within 1 h before incision; SCIPINF2: received most appropriate antibiotic(s) for surgery; SCIPINF3: stopped preventive antibiotic(s) within 24 h after surgery; SCIPVTE2: treatment to prevent blood clots within 24 h before or after surgery. The estimates reported are based on logistic regression operationalized through the generalized linear model with logit link. All regression models were adjusted for hospital characteristics including teaching status, profit status, membership in multihospital system, rural location, transfer-adjusted case mix index, qualified for disproportionate share payment, and natural log of licensed bed size and cumulative condition-specific patient volume during 2044–2008. All medication process quality measures were observed for the period January–December 2009, and eMAR and CPOE systems were reported to be in operational use by 2008. to control for endogeneity effects, in each regression, propensity score (of eMAR or CPOE adoption) based indicators corresponding to five quintiles of propensity score distribution were included. *Estimates significant at p<0.10.

In contrast, hospitals that implemented CPOE only showed little difference in quality from non-adopters. In only two of the 11 measures did CPOE-only adopters have higher quality than non-adopters: the odds of giving the most appropriate initial antibiotic(s) to PN patients (AOR, 1.29; 95% CI 1.08 to 1.54), and giving treatment to prevent blood clots within 24 h before or after select surgery to eligible patients (AOR, 1.27; 95% CI 0.99 to 1.63; significant at p<0.10).

Hospitals that adopted both eMAR and CPOE performed better on 10 of the 11 measures. The odds of adherence to medication guidelines were 13–38% higher among adopters of both technologies compared with non-adopters. For example, the odds of prescribing aspirin at discharge were 38% (AOR, 1.38; 95% CI 1.38 to 1.68) higher, whereas the odds of administering preventive antibiotic(s) within 1 h before incision to surgical patients were 13% (AOR, 1.13; 95% CI 1.00 to 1.29) higher among hospitals using both eMAR and CPOE compared with non-adopters.

In all, these results show substantive support for hypothesis 1, with the exception of those hospitals that adopted CPOE only. We will explore this issue further in the comments section.

Association of duration of eMAR and CPOE use with medication process quality

In our sample, a substantial number of hospitals did not report adoption dates for eMAR and CPOE technologies (33% and 27%, respectively, among adopters). To examine the relationship between duration of use (measured in 2-year increments) of health IT and hospital quality (hypothesis 2), we conducted the analysis in two ways. First, we performed the analysis with a censored dataset—that is, by ignoring hospitals with missing observations on duration of use. Next we repeated the analysis using imputed data, in which we derived duration of use by performing multiple imputations (n=25) using ordered logistic regression, with select hospital characteristics as predictors to estimate the missing duration of use. The results from the two approaches were qualitatively similar. Here we report the results using censored data.

Figure 2 provides a summary of the AORs and 95% CIs for duration of eMAR and CPOE use on the medication-related quality measures (see table A2 in the online appendix for the results based on multiple imputations). For each additional 2 years of use among hospitals with eMAR-only systems, the odds of adhering to the recommended medication guidelines were significantly higher on all 11 measures, with gains of 6–15% compared with hospitals with neither eMAR nor CPOE systems. For experience with CPOE-only systems, however, we found no difference in quality between such hospitals and non-adopters of both eMAR and CPOE systems on most measures; only the odds of prescribing β blocker to AMI patients (AOR, 1.14; 95% CI 1.02 to 1.28), giving most appropriate initial antibiotic(s) to PN patients (AOR, 1.08; 95% CI 1.01 to 1.16), and giving appropriate medication to prevent blood clots within 24 h before or after surgery to surgical patients (AOR, 1.10; 95% CI 1.01 to 1.19) were higher. In contrast, for each additional 2 years of use of both eMAR and CPOE systems, the odds of adhering to the recommended medication guidelines were 9–15% higher across nine of the 11 quality measures. These results suggest that duration of technology use is associated with better quality of medication administration (supporting hypothesis 2).

Figure 2

Adjusted ORs (95% CIs) for duration of electronic medication administration records (eMAR) and computerized physician order entry (CPOE) use, measured in the increment of 2 years, on medication process quality at medium-to-large acute-care hospitals in the USA (n=2603). AMI1: given aspirin at admission; AMI2: prescribed aspirin at discharge; AMI3: given ACE inhibitor or ARB for left ventricular systolic dysfunction; AMI5: prescribed β blocker at discharge; HF3: given angiotensin converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) for left ventricular systolic dysfunction; PN5: given initial antibiotic(s) within 6 h of arrival; PN6: given most appropriate initial antibiotic(s); SCIPINF1: received preventive antibiotic within 1 h before incision; SCIPINF2: received most appropriate antibiotic(s) for surgery; SCIPINF3: stopped preventive antibiotic(s) within 24 h after surgery; SCIPVTE2: treatment to prevent blood clots within 24 h before or after surgery. The estimates reported are based on logistic regression operationalized through the generalized linear model with logit link. All regression models were adjusted for hospital characteristics including teaching status, profit status, membership to multihospital system, rural location, transfer adjusted case mix index, qualified for disproportionate share payment, and natural log of licensed bed size and cumulative condition-specific patient volume during 2044–2008. All medication process quality measures were observed for the period January–December 2009, and eMAR and CPOE systems were reported to be in operational use by 2008. To control for endogeneity effects, in each regression, indicators based on the propensity score (of eMAR or CPOE adoption) corresponding to five quintiles of propensity score distribution were included.

Association of hospital characteristics with medication process quality

In our sample, teaching status, profit status, and rural hospitals had generally no significant association with medication process quality measures, except that rural hospitals were likely to perform better on one PN and one surgical infection prevention measure. Further, membership to multihospital systems was positively associated with most medication quality measures; however, hospital bed size, case mix index, qualification for disproportionate share payment, and cumulative patient volume were negatively associated. Finally, technology adoption propensity was mostly non-significant (see tables A3 and A4 in the online appendix for details).

Comments

This study sought to quantify the association between the use of eMAR and CPOE with the quality of inpatient medication administration. In particular, it compared hospitals using eMAR only, CPOE only, and both eMAR and CPOE against hospitals not using either technology, as well as duration of use of those technologies, with respect to their adherence to 11 medication-related clinical guidelines. The results show higher adherence to 10 of 11 measures for hospitals using both eMAR and CPOE (AORs ranging from 13% to 38%) as well as hospitals using eMAR only (AORs ranging from 14% to 29%). Similar results hold for duration of technology use, indicating that hospital experience with the technologies is important (AORs ranged from 6% to 15% for eMAR-only hospitals and users of both eMAR and CPOE). These AORs translate to about 0.4 to 1.4 percentage point improvements in the adherence to medication guidelines among hospitals using eMAR only, and about 0.4 to 2.0 percentage point improvements among hospitals using both eMAR and CPOE systems compared with non-adopters of both technologies (see table A5 in the online appendix for details of marginal effects of eMAR and CPOE). These marginal improvements associated with use of health IT are substantive and consistent with prior research findings reported by DesRoches et al,4 Jones et al,5 and McCullough et al,7 despite the “ceiling effects”—that is, the average quality scores on all quality measures are in the 90s.5

Prior studies of health IT and quality, examining national samples of hospitals, have reported mixed findings.6 16 48 49 For example, studies that focused on full-scale EHR systems or combinations of several technologies have found limited evidence for improved medication quality.8 9 42 48 49 Two studies found that hospitals with CPOE outperform hospitals without CPOE on AMI and PN medication measures reported for 2004.6 16 In both studies, however, CPOE adoption was negatively associated with administration of initial antibiotics within 4 h of arrival, a measure also included in our study (note that the time threshold was extended by Hospital Quality Alliance to 6 h in 2007). Our findings suggest that hospitals with CPOE perform better on this measure, but only when the hospitals also have eMAR (AOR=1.19, 95% CI 1.04 to 1.35). In CPOE-only hospitals, there was no significant difference on this measure compared with non-adopting hospitals. The positive result on this measure in our analysis may be due to several factors including the concurrent use of eMAR systems in the medication management process, the increase in the time threshold from 4 to 6 h in the guidelines, our use of more recent data, or our focus on 100+ bed hospitals.

In summary, this study showed positive evidence that use of, and experience with, eMAR technology, alone and in combination with CPOE, is associated with a better quality of medication administration compared with hospitals without such technologies. The effects of using eMAR in combination with CPOE, however, do not appear to be significantly different from the effects of eMAR alone. This lack of marginal improvement with the addition of CPOE is surprising, but may be due to several confounding factors not included in this study, such as variations in functional capabilities of CPOE systems, usability factors, and variations in the extent of technology use among clinical staff. Such clarity over capabilities and extent of use is not reflected in our data.50 As David Bates51 noted “A CPOE system can have a major effect on care improvement, but realizing its potential benefits will demand that it be used well.” This observation is supported by many implementation challenges in the clinical decision support module, a key component in CPOE.52

This study has a number of limitations. Cross-sectional analyses such as those reported in this study do not represent causal effects of IT use on process quality. We examined IT use as of 2008 on quality in 2009, and employed duration of technology use to try to determine the relationship between quality and effects of technology use over time and find a significant positive relationship. However, future studies are needed that use panel datasets to tease out the causal relationship. A second limitation is that the data for a substantial number of hospitals were missing technology adoption year, so could not be included in the analysis of duration of technology use. We found no significant differences in the quality scores between reporting and non-reporting hospitals, suggesting that non-reporting hospitals do not differ systematically from reporting hospitals. Still, we imputed duration of use from hospital characteristics in order to include missing hospitals in our analysis. Third, our results focus on acute-care hospitals with 100 or more beds, so they do not necessarily apply to small hospitals. Fourth, we cannot rule out the possibility that hospitals with health IT systems, such as those studied here, were merely better at recording their medication administration than non-adopting hospitals. Finally, our measures of eMAR and CPOE did not capture aspects of functionality or other important dimensions of technology use, such as extent of use among clinicians, that may influence the quality of medication administration. Despite these limitations, the findings of this study show a positive relationship between specific types of health IT use and quality of medication administration in medium and large US hospitals.

Conclusions

Acute-care hospitals that adopted eMAR and CPOE systems before 2008 performed better on 2009 medication quality measures related to conditions including AMI, HF, and PN, as well as preventive medications for surgery patients. Adoption of eMAR alone, and in combination with CPOE, is associated with greater adherence to medication guidelines. These findings, compared with past research, offer more consistent and stronger evidence for the benefits of health IT. Additionally, duration of use for these technologies (eMAR alone, and the combination of eMAR and CPOE) is also associated with better adherence, thus plausibly demonstrating the effects of “learning by doing”. However, hospitals using CPOE alone (about 5% of the sample) did not perform better than non-adopters of eMAR and CPOE. Further work is needed to understand the role of the specific functionality of eMAR and CPOE systems in medication quality, such as the variation in capabilities of these systems, the extent of technology use among clinical staff, as well as how health IT fits with specific organizational and practice characteristics that facilitate the delivery of high-quality care.

Acknowledgments

We thank HIMSS Analytics for licensing data on hospitals' adoption of eMAR and CPOE. We thank Dr Ohno-Machado (editor-in-chief), associate editor, and anonymous reviewers of JAMIA for their valuable suggestions. Additionally, we thank Academy Health conference attendees, and seminar participants at the School of Informatics, Indiana University Purdue University at Indianapolis for their helpful comments.

Footnotes

  • Funding National Science Foundation (NSF), Grant Number NSF-CNS-0910842; Rockefeller Center for Public Policy at Dartmouth College. Neither National Science Foundation nor Rockefeller Center for Public Policy at Dartmouth College had any role in the design and conduct of the study, collection, management, analysis, and interpretation of the data, or the approval of manuscript.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

References

Related Article

Free Sample

This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JAMIA.
View free sample issue >>

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Open Access fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.