The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials
- K Ann McKibbon1,
- Cynthia Lokker1,
- Steven M Handler2,
- Lisa R Dolovich3,
- Anne M Holbrook4,
- Daria O'Reilly1,
- Robyn Tamblyn5,
- Brian J Hemens1,
- Runki Basu1,
- Sue Troyan1,
- Pavel S Roshanov1
- 1Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada
- 2Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, USA
- 3Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada
- 4Department of Medicine, McMaster University, Hamilton, Ontario, Canada
- 5Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, Quebec, Canada
- Correspondence to Ann McKibbon, Health Information Research Unit, Dept of Clinical Epidemiology and Biostatistics, McMaster University, CRL 137, 1280 Main Street West, Hamilton, ON L8S 4K1, USA;
- Received 7 April 2011
- Accepted 11 July 2011
- Published Online First 18 August 2011
Objective The US Agency for Healthcare Research and Quality funded an evidence report to address seven questions on multiple aspects of the effectiveness of medication management information technology (MMIT) and its components (prescribing, order communication, dispensing, administering, and monitoring).
Materials and Methods Medline and 11 other databases without language or date limitations to mid-2010. Randomized controlled trials (RCTs) assessing integrated MMIT were selected by two independent reviewers. Reviewers assessed study quality and extracted data. Senior staff checked accuracy.
Results Most of the 87 RCTs focused on clinical decision support and computerized provider order entry systems, were performed in hospitals and clinics, included primarily physicians and sometimes nurses but not other health professionals, and studied process changes related to prescribing and monitoring medication. Processes of care improved for prescribing and monitoring mostly in hospital settings, but the few studies measuring clinical outcomes showed small or no improvements. Studies were performed most frequently in the USA (n=63), Europe (n=16), and Canada (n=6).
Discussion Many studies had limited description of systems, installations, institutions, and targets of the intervention. Problems with methods and analyses were also found. Few studies addressed order communication, dispensing, or administering, non-physician prescribers or pharmacists and their MMIT tools, or patients and caregivers. Other study methods are also needed to completely understand the effects of MMIT.
Conclusions Almost half of MMIT interventions improved the process of care, but few studies measured clinical outcomes. This large body of literature, although instructive, is not uniformly distributed across settings, people, medication phases, or outcomes.
- Clinical decision support
- clinical decision support systems
- decision support
- drug therapy
- electronic prescribing
- evidence-based medicine
- health information technology
- informatics education
- information retrieval
- library science
- machine learning
- medical informatics
- medication systems
- predictive modeling
- privacy technology
- randomized trials
- research methodology
- son's name
- statistical learning
- systematic reviews
Medication management is a major component of the healthcare system. Currently, approximately 10% of the US healthcare budget is spent on prescription medication.1 Suboptimal prescribing, order communication, dispensing, administering, and monitoring of medications can lead to medication errors, therapeutic failures, adverse drug withdrawal events, and adverse drug events (ADEs).2 ADEs are associated with approximately 1.2 million or 3.1% of all hospitalizations in the USA. A meta-analysis of ADEs suggests that these events are between the fourth and sixth leading cause of death in the USA.3 Each ADE is estimated to increase the length of hospital stay by 2.2 days and to increase hospital costs by US$3244.4
Many US organizations endorse the use of medication management information technology (MMIT) to improve the medication management process and patient safety. These include Leapfrog, the Institute of Medicine, the Agency for Healthcare Research and Quality (AHRQ), and the Office of the National Coordinator for Health Information Technology. In addition to these groups, the recent US Health Information Technology for Economic and Clinical Health Act will authorize incentive payments through Medicaid and Medicare for implementing and meeting certain goals using electronic health records for patient care. Achievements in healthcare processes and outcomes will be tied to incentive payments starting in 2011. The goal of the US Health Information Technology for Economic and Clinical Health Act is not just based on the adoption, but also on the ‘meaningful use’ of electronic health records and their integration with other systems. Meaningful use is defined by a set of 25 core health information technology (HIT) objectives that constitute an essential starting point, as well as an additional menu of activities that providers and hospitals may choose to implement.5 Examples of these core measures related to medication management are computerized provider order entry (CPOE) for e-prescribing, drug–drug and drug–allergy checking, and medication reconciliation capabilities.
The AHRQ recognizes that HIT is an important and promising means of improving healthcare outcomes. They contracted with McMaster University to produce a comprehensive report summarizing the existing evidence of the potential for MMIT to improve healthcare processes and support patients needing prescription medication. Previous systematic reviews generally report the effect of MMIT on one of the medication management phases (prescribing, order communication, dispensing, administering, and monitoring). For example, Mollon and colleagues6 reviewed randomized controlled trials (RCTs) of clinical decision support systems (CDSSs) for prescribing and found 37 reports that showed changes in the behavior of healthcare providers, but only five of these noted improvements in patient outcomes. Similarly, Eslami and colleagues7 reviewed mostly observational studies of CPOE applications in outpatient prescribing. They showed that although CPOE and other information systems are often costly, some evidence supports medication safety benefits. However, they also found some studies showing unintended negative consequences including increased error rates (existing and new errors caused directly by the systems or indirectly by changes in workflow) and ADEs with CPOE.
The AHRQ contract specified summarization of the evidence on many aspects of MMIT. The report addressed seven broad questions related to medication management and HIT (effectiveness; gaps in knowledge and evidence; the value proposition of MMIT; system characteristics and their association with decisions to purchase, implement, or use MMIT; sustainability of MMIT; two-way electronic data interchange for order communication; and RCTs for CDSSs).8 The full report, available online, includes 378 studies across multiple qualitative and quantitative methods and many outcomes and encompasses more than 1000 pages. For this review, which is a subset of the full report, we describe the process of collecting and synthesizing only the findings from RCTs on the effectiveness and effects of MMIT on all phases of medication management. We report results for process changes, clinical outcomes, and other outcomes related to use, usability, knowledge, skills, and attitude. Other outcomes are provided in the full report.
Defining the process of medication management
Bell and colleagues9 model the medication management continuum in five phases, which we use in this review: prescribing or ordering the appropriate medication; order communication or transmitting and perfecting the prescription through the interaction between the prescriber and the pharmacist; dispensing the medication in its requested form and strength; administering the medication; and monitoring (ongoing oversight to review the benefits and potential adverse effects of the medication such as therapeutic failures, adverse drug withdrawal events, and ADEs). We also evaluated the evidence concerning the reconciliation of medication at transitions between settings of care and education. Both education or training for the health professionals associated with installation of MMIT applications and patient education related to medication management were part of our mandate.
Data sources and searches
We searched electronic databases: MEDLINE, EMBASE, CINAHL (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts, Compendex, INSPEC (which includes IEEE), Library and Information Science Abstracts, E-Prints in Library and Information Science, PsycINFO, Sociological Abstracts, and Business Source Complete. The search terms are included in the full online evidence report.8
Supplemental searches targeted gray literature: the New York Academy of Medicine databases, SIGLE, US Health and Human Services Health Information Technology, Health Technology Assessment reports and economic studies from the UK Centre for Reviews and Dissemination, ProQuest Dissertations, National Library for Health United Kingdom (includes Bandolier), ProceedingsFirst, PapersFirst, US National Technical Information Service, and Google. As part of our gray literature search, we reviewed the AHRQ Knowledge Library on e-prescribing, bar coding, and CPOE.
The literature search details are in the full report.8 When possible, we excluded letters, editorials, commentaries, and animal studies. No limits were placed on language, date, subject domain, or geography.
The references of review articles were screened for eligibility. Members of the AHRQ technical expert panel8 and writing team provided articles from their personal files. Our search was peer reviewed by a librarian following the PRESS checklist for systematic review searches.10
Independent reviewers from a team of reviewers screened titles and abstracts in duplicate. This first screen was to detect articles that described the use of MMIT. We defined MMIT as electronic systems that collect, process, or exchange health information about the need for and use of medication for patients and their formal caregivers. We included articles only if the medication management system was integrated with at least one other HIT system such as an electronic medical record system. In addition, the MMIT systems had to integrate patient-specific information and provide processed data (eg, trending data for time in therapeutic range for patients taking warfarin), advice (eg, allergy alert for penicillin) or suggestions to healthcare providers or patients and families on medication issues related to health care and wellness (eg, lower lithium dose based on blood concentrations). We excluded stand-alone medical devices such as infusion pumps not linked into electronic medical records (ie, no integration) with the exception of personal digital assistants or handheld devices into which clinicians or patients entered patient-specific information to assist in medication management (eg, dosing calculations for pediatric patients).
For this systematic review, we selected only RCTs that assessed the effectiveness and effects of MMIT (two of the seven contract questions). The full AHRQ evidence report addresses the other five questions as well as qualitative articles and observational studies.8
Data extraction and quality assessment
Data from each article were abstracted by one reviewer and checked for accuracy by a second. The reviewers were not blinded to authors, institutions, or journal. Senior staff performed final accuracy checks. Extracted data included general study characteristics: study design, intervention, study population, setting, disease and drugs of interest, and a description of the medication management phase and MMIT application. We abstracted primary and secondary outcomes from each article. If no primary outcome measures were indicated, we focused on outcomes related to medication management and clinical outcomes.
We recorded whether the outcomes were positively changed by the intervention, negatively, or not changed (not significant). If more than one primary or secondary outcome was reported, the direction of the majority of outcomes was used to determine the overall effect. We used the framework suggested by Chaudry and colleagues11 to present and analyze our findings: process changes (eg, proportion of orders that were adherent to CDSS recommendations, prescription error reduction, improved rates of appropriate antibiotics), knowledge, skills and attitudes; satisfaction; clinical outcomes; and costs.
Experienced reviewers (CL, KAM, AMH, ST, DO'R) assessed the included studies for quality of reporting. Quantitative studies were assessed using the same criteria employed by Jimison et al12 in their AHRQ evidence report. RCT scoring was based on Delphi consensus work done by Verhagen and colleagues.13 Up to nine points were awarded for random allocation, allocation concealment, comparable groups at baseline, the presence of eligibility criteria, blinding (patients, caregivers and outcome assessors), measurement of the variability of outcomes, and intention to treat analyses. In addition, we scored studies on their use of clustered design, analytical adjustments for clustering, and at least 80% follow-up.
Data synthesis and analysis
We did not perform meta-analyses because of study differences in intervention, populations, technologies, and outcomes measured. Data were summarized by counts of articles that showed positive, negative or no difference in outcome measures.
Our searches retrieved 32 785 articles that were screened at the abstract level (figure 1). Of those, 4356 were further assessed in full text, 87 of which were RCTs studying the effect of MMIT applications. The full AHRQ evidence report includes analysis of data from 378 articles (191 were not RCTs).
General study characteristics
The RCTs focused mainly on assessing the prescribing or monitoring medication management phases (table 1). Twenty-seven articles studied the effect of MMIT on both of these medication management phases. Order communication, dispensing, administering, education, or reconciliation were seldom studied. Supplementary appendix tables 1–6 (available online only) contain study information by setting (hospital and ambulatory) and by endpoints (process changes, clinical and other intermediate measures).
Most of the 87 trials were performed in the USA (n=63, 72.4%), Europe (n=16, 18.4%), and Canada (n=6; 6.9%), and two each in Australia and Israel. Several pioneering US institutions were strongly represented: Harvard, Brigham Women's, and Partners in Massachusetts (n=14), Regenstreif and Wishard in Indiana (n=12), Kaiser Permanente (n=8), Veterans Affairs (n=5), and University of Washington (n=4). Most studies were published after 2000 (86%) (figure 2). Although the number of RCTs increased over time, the proportion of RCTs to all of the literature declined over the same period (data not shown).
Seventy-eight trials (90%) had a CDSS component: eight of these assessed CPOE plus CDSSs,14 26 27 37 53 58 76 82 two assessed e-prescribing systems plus CDSSs.68 83 To differentiate between the two, we tagged systems as being CPOE if they included medication ordering and often other orders in hospitals and communicated with inhospital pharmacies while e-prescribing systems were more often ambulatory based and allowed communication with community pharmacies. Two studies assessed pharmacy information systems with CDSS.21 87 Three studies looked at a CPOE system with minimal or no decision support.63 74 77 Of these, two were older with no or limited CDSS and one was based on order sets.100 Three evaluated a personal health record system16 90 94 and one was a medication reconciliation system.99
Quality and reporting
The articles rated an average score of 4.5 out of 9 (95% CI 4.1 to 4.8) on the Verhagan quality score. Fifty of the trials had a clustered design (57%); 44 of which indicated analysis and adjustment for the clustering, if needed. Sample sizes were five to 156 772 participants in 73 of the trials (mean 11 435, 95% CI 5320 to 17 551) and 10 to 778 healthcare providers in 31 studies (mean 148, 95% CI 84 to 212). Thirty trials reported greater than 80% follow-up, seven reported lower than 80%, and 50 failed to report follow-up.
Most of the trials evaluated prescribing. Eighty trials measured one or more changes in process: 69 changes were primary (table 2) and 30 were secondary endpoints (table 3). Nineteen of the 24 trials set in hospitals measured process as the primary endpoint. Eight of these measured changes in prescribing behavior, all of which showed significant improvements (table 2). Ten trials measured adherence to prescribing and monitoring advice, eight of which showed improvement. Errors and time to perform tasks were seldom studied and no hospital trials measured workflow.
In the ambulatory setting, 46 of 58 trials assessed process as their primary measures (table 2). Most focused on prescribing or monitoring behavior changes and adherence to CDSS advice. Many of the interventions simultaneously influenced prescribing and monitoring. Seven of 10 trials found improvements in prescribing behaviors, 12 of 20 showed significant improvements in adherence to prescribing advice, and 10 of 16 showed improved adherence to monitoring advice. Both trials that measured time for activities showed reductions.63 76 Two trials found significant improvements in composite endpoints,61 89 and two found improvements in dispensing.21 22 The only study measuring the effect of MMIT on patient compliance detected no effect.64 Three trials measured contacts made by healthcare providers47 83 87 and two found no effect of MMIT.47 83 These three trials studied telephone consultation between patients and physicians around prescribing for antibiotics,47 pharmacist and physician telephone calls to perfect the prescription before dispensing,83 and pharmacist discussion with patients for prophylactic aspirin use in diabetes.87
Two trials took place in pharmacies with significant reductions in the rate at which pharmacists called physicians for clarification (callback rate)83 87 and increased patient encounters to prompt aspirin use for people with diabetes.83 87 One study in long-term care assessed prescribing changes as their primary endpoint (appropriateness of antidepressant orders) and found significant improvements.14
Nine hospital-based trials measured process changes as secondary outcomes (table 3). Five of these showed improvements,29 52 65 95 three showed no difference with the use of MMIT,23 66 99 and one showed an increase in time to write orders.77 Nineteen trials in ambulatory settings looked at secondary process changes and nine showed significant improvements (table 3).
Two trials in nursing homes assessed secondary process outcomes; one found significant improvements in prescribing behavior,14 the other found no difference in the number of potential ADEs with MMIT.15
Twenty-three trials evaluated clinical endpoints as their main outcome measures (table 4) and 26 had secondary outcomes of clinical endpoints (table 5). Counting articles with multiple outcomes individually, 38 trials measured a clinical endpoint. Only three hospital-based trials assessed a clinical primary endpoint, of which two showed significant improvements 40 59 and one showed no difference.82
Twenty ambulatory trials measured clinical endpoints; 12 included such physiological measures as blood pressure, cholesterol, or blood glucose levels. Overall, eight of these 20 trials showed clinical improvements (table 4).
Seventeen trials measured 27 secondary clinical endpoints (table 5). One trial showed significant improvements in both hospitalizations and mortality.84 Another found increased emergency department visits for intervention patients.94 All other trials showed no differences for these secondary outcomes (table 5).
In other settings, Gurwitz and colleagues15 found no difference in the rate of ADEs in nursing homes incorporating CDSSs and CPOE compared with homes with CPOE alone. In a pilot study of five patients using a hand-held insulin regimen optimizer, Holman and colleagues 88 reported improved hemoglobin A1c levels.
Other outcomes: primary endpoints
Three trials had costs as a main endpoint; all three showed improvement including reduced costs per practice with thiazide prescribing for hypertension (US$540 per practice),29 US$37.64 less per patient on antimicrobial agents with the addition of a CDSS,77 and a 13% reduction in costs with CPOE.25 Two trials measured knowledge, skills, or attitudes, one of which was positive,37 and the other not significant.94 One trial examined satisfaction, with no significant difference seen.68
Other outcomes: secondary endpoints
Costs (one not significant95 and one positive73) and perceptions (one not significant75 and one positive83) were measured as secondary outcomes in two articles and satisfaction in three trials (one positive,32 two not significant44 94). All primary cost trials reported cost reductions measured in multiple ways with the use of HIT. Satisfaction with systems and perceptions showed mixed results.
A large number of RCTs of MMIT exist. They are not uniform in number or intensity of study (eg, length of study, complexity or extent of MMIT and its integration, number of healthcare providers involved) across medication management phases, settings, types of MMIT, personnel, or outcomes. Prescribing (n=74) and monitoring (n=38) phases were much more frequently studied in the 87 RTCs. We identified very few MMIT interventions that targeted order communication, dispensing, administering, continuing professional or patient education, or medication reconciliation. Most trials were conducted in US centers with a strong history of MMIT: Boston, Indianapolis, Washington state, Kaiser Permanente, and Veterans Affairs hospitals. Most trials were published after 1999. The trials concentrated on CDSSs and CPOE systems and few evaluated MMIT systems used by non-physicians. MMIT in long-term care settings such as nursing homes, pharmacies, homes, and community was also not well studied.
Overall, the quality of the articles was poor, with scores indicating that only half reported the use of methods generally accepted as minimizing bias. In addition, we found varying definitions and methods of measurement of outcomes and lack of consensus on reporting. An RCT may not be the only method necessary for a full assessment of the effects of HIT. Programatic analyses or assessment methods for complex interventions can also provide valuable insights into the effects of MMIT, but we did not identify these in our literature review. The full AHRQ report contains studies of various other designs that address important aspects of MMIT systems that are not usually identified using RCT methods.8 For example, qualitative studies identified substantial unintended consequences of MMIT, and case studies of implementations can provide valuable qualitative insights.
Consistent with other reviews of MMIT, most studies measured changes in process and the majority of these showed benefit. Few trials studied clinical outcomes. Despite using a broad definition of clinical outcomes that included physiological measurements (eg, blood pressure and blood glucose levels), very few studies showed improvement in primary outcomes (11 of 26), and even fewer studies that used clinical endpoints as secondary outcomes found benefit (two of 43 showed benefit and one showed harm).
The literature of MMIT is dominated by implementation and demonstration projects (see full report). These were not designed to evaluate and establish clinical benefit for patients in settings that used MMIT systems compared with those who received usual care without MMIT. Few studies measured clinical improvements for patients or the quality of health care provided. Potential harm to patients resulting from the use of MMIT was rarely evaluated. Future research should address these gaps. In particular, it is also important that the effect of MMIT on the consumer be considered.
The consistent application of a set of agreed-upon standards for the assessment, evaluation, and description of MMIT (analogous to CONSORT guidelines for RCT reporting and PRISMA guidelines for reviews) could improve the quality and generalizability of future research. Consistency and greater depth of reporting is also needed in published trials, especially with respect to MMIT itself and settings.
The findings of this review are consistent with those of the full AHRQ report, which included studies using multiple research designs.8 The AHRQ report also includes summaries of qualitative studies, sustainability of MMIT systems, the value proposition of MMIT, and feature sets associated with the likelihood of purchase, implementation and use.
This review has several important limitations. First, the studies reported only limited data on systems, installations, institutions, and targets of the intervention making complex synthesis difficult. We also found problems with methods and analyses, a wide variation in the number of studies in certain areas, and a broad range of MMIT systems. We attempted to address this limitation by basing our work on well-defined analytical frameworks and by identifying not only the systems used but also their functional capabilities. Third, this review summarizes only the RCTs from the 378 studies included in the full report. Although we used only the studies with strong research methods for this overview of MMIT, we feel that they provided a very similar assessment of the effects and effectiveness provided across all articles. Our requirement that MMIT systems were integrated with other HIT may also have led to the exclusion of important evidence. In addition, missing in this review are the qualitative studies that portray a rich understanding of effects of MMIT on clinicians and patients.
Medication management is a complex and important component of health care. MMIT systems have been implemented in many organizations to improve the safe and appropriate use of medication. Many studies demonstrated benefits on process measures for MMIT focused on improving medication management. Little evidence of benefit is available for the effectiveness of the use of MMIT on clinical outcomes. The body of evidence from studies of MMIT is not uniform across domains, settings, phases, and geography, or held to the same standards as the pharmaceutical industry. Future research directions can be based on the findings and gaps of this report: phases of order communication, dispensing, administering, and reconciliation; settings (long-term care, communities, and homes); MMIT beyond CDSSs and CPOE and especially the MMIT systems used by non-physicians; and systems designed for patients and caregivers. Also, multiple study methods need to be used to produce a comprehensive analysis of the benefits, harms and costs of MMIT systems. Reporting standards for all HIT are also recommended for all studies.
The research team would like to thank those who helped with screening, abstracting and article retrievals: Danika Walden, Nathan McKibbon, Catherine Salmon, Jan Burke-Gafney, Connie Freeborn, Jamie O'Donnell, Rizwan Bhaloo, Bryan Cheeseman, Hafsa Jan Qureshi, and Pouyan Ahangar. The authors would like to offer special thanks to Mary Gauld, Maureen Rice, Julianna Beckett, and Roxanne Cheeseman for helping with project management and editorial perfection. They acknowledge the hard work of Nicholas Hobson, their computer programmer for creating the evolving systematic review management software. The authors accomplished this task with the expert guidance of Rebecca Roper of the Agency for Healthcare Research and Quality as their task order officer.
Funding This project was funded under contract HHSA 290-2007-10060-I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services. McMaster EPC was tasked with the original report and it can be found at http://www.ahrq.gov/clinic/tp/medmgttp.htm. The authors of this review are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.