Impact of a clinical decision support system on antibiotic prescribing for acute respiratory infections in primary care: quasi-experimental trial
- 1Department of Family Medicine, Medical University of South Carolina, Charleston, South Carolina, USA
- 2Division of Biostatistics and Epidemiology, Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA
- Correspondence to Dr Arch Mainous III, Department of Family Medicine, Medical University of South Carolina, 295 Calhoun Street, Charleston, SC 29425 USA;
Contributors AM, CAL and PJN contributed to the conception and design of the study and participated in the analysis of data. All authors had access to the data and participated in drafting the article or revising it critically for important intellectual content. All authors gave final approval of the version to be published.
- Received 17 November 2011
- Accepted 8 June 2012
- Published Online First 3 July 2012
Objective To assess the effect of a clinical decision support system (CDSS) integrated into an electronic health record (EHR) on antibiotic prescribing for acute respiratory infections (ARIs) in primary care.
Materials and methods Quasi-experimental design with nine intervention practices and 61 control practices in the Practice Partner Research Network, a network of practices which all use the same EHR (Practice Partner). The nine intervention practices were located in nine US states. The design included a 3-month baseline data collection period (October through December 2009) before the introduction of the intervention and 15 months of follow-up (January 2010 through March 2011). The main outcome measures were the prescribing of antibiotics in ARI episodes for which antibiotics are inappropriate and prescribing of broad-spectrum antibiotics in all ARI episodes.
Results In adult patients, prescribing of antibiotics in ARI episodes where antibiotics are inappropriate declined more (−0.6%) among intervention practices than in control practices (+4.2%) (p=0.03). However, among adults, the CDSS intervention improved prescribing of broad-spectrum antibiotics, with a decline of 16.6% among intervention practices versus an increase of 1.1% in control practices (p<0.0001). A similar effect on broad-spectrum antibiotic prescribing was found in pediatric patients with a decline of 19.7% among intervention practices versus an increase of 0.9% in control practices (p<0.0001).
Conclusions A CDSS embedded in an EHR had a modest effect in changing prescribing for adults where antibiotics were inappropriate but had a substantial impact on changing the overall prescribing of broad-spectrum antibiotics among pediatric and adult patients.
Antibiotics probably provide little benefit for most acute respiratory tract infections (ARIs) that are seen in primary care.1 However, respiratory tract infections account for 60% of all antibiotic prescribing in primary care.2 Moreover, there has been an increased use of broad-spectrum antibiotics for these conditions.3 Because of the implications for increased antibiotic resistance, a variety of guidelines including ones from the National Institute for Health and Clinical Excellence (NICE) and the Centers for Disease Control and Prevention (CDC) have been created to try to improve antibiotic prescribing for ARIs in primary care.4–6 Attempts to increase adherence to clinical practice guidelines for judicious antibiotic prescribing have used multiple different strategies.7 ,8
Electronic clinical decision support systems (CDSS) may have particular value in changing practice and increasing adherence to clinical practice guidelines.9 ,10 In particular, CDSS can provide clinicians with patient-specific assessments and treatment recommendations in real-time decision-making at the point of care. CDSS have demonstrated effectiveness in increasing adherence to guidelines.11
Studies using various types of CDSS have been used in an effort to improve clinician adherence to guidelines for antibiotic prescribing for ARIs. Although these studies indicate that when the CDSS are employed they can improve prescribing, because the CDSS tend to have low rates of use by clinicians their benefits are limited.12 ,13 Samore et al14 demonstrated in a randomized trial that a CDSS plus a community-based intervention decreased antibiotic prescribing for ARIs more than in communities with only a community-based intervention. In that study, clinicians who used the hand-held computer or paper and pencil CDSS showed a greater decrease in antibiotic prescriptions.
One promising approach to improving care in clinical practice is through the delivery of CDSS at the point of care within electronic health records (EHRs). What is unclear is whether integration of a CDSS into EHRs will be adopted and effective in decreasing antibiotic prescribing in episodes for which antibiotics are not indicated, as well as decreasing the prescribing of broad-spectrum antibiotics across all ARIs. The purpose of this study was to assess the impact of a CDSS integrated into an EHR designed to improve antibiotic prescribing for ARI in primary care practices using a multi-method intervention to facilitate CDSS adoption.
Materials and methods
The study used a quasi-experimental design to conduct a trial with nine intervention practices and 61 control practices in the Practice Partner Research Network (PPRNet). The research design included a 3-month baseline data collection period (October through December 2009) before the introduction of the intervention and 15 months of follow-up once the intervention was implemented (January 2010 through March 2011). PPRNet is a primary care research network across the USA whose members use a common EHR (Practice Partner (PP) by McKesson, Inc, San Francisco, California, USA) and pool data quarterly for quality improvement and research projects. Practice Partner is a certified EHR and medical billing software solution that instantaneously populates patient data across the entire chart. Searchable patient data is then automatically generated.
Nine PPRNet practices in nine US states, representing 27 physicians, six nurse practitioners and six physician's assistants, volunteered to participate in this study in response to an email sent to PPRNet members. All providers agreed to use the CDSS for the duration of the study when evaluating patients presenting with ARI symptoms. Participating practices were not given monetary incentives to participate but were provided with practice reports of their prescribing behavior throughout the study.
Additional PPRNet practices that met the requirements of having six quarters of complete data, at least one MD on the staff, total numbers of providers within the range of the intervention practices (ie, 2–8), and total number of ARIs in the baseline timeframe within the range of the intervention practices (ie, 96–1701) were employed as a control group. The study was approved by the institutional review board at the Medical University of South Carolina.
A multi-method intervention was used to facilitate provider adoption of the CDSS, including quarterly EHR based audit and feedback, ‘best-practice’ dissemination during meetings of practice representatives and practice site visits for academic detailing, performance review, and CDSS training. Quarterly reports presented practice use of the CDSS for ARI encounters, percentage of use of antibiotics for ARI diagnoses, and the proportion of broad-spectrum antibiotics. Two liaison personnel from each practice (one provider and one clinical staff member) attended two project meetings during the first year of the study. The practice liaison personnel were members of the participating practice and were expected to work with their colleagues in explaining the CDSS and gathering information on the challenges of using the CDSS in the practice to share with the study team.
At the initial meeting, antibiotic guidelines were reviewed, and the CDSS was presented. The CDSS and how to integrate it into the EHR was demonstrated to all the participants at the first project meeting and each participant had an opportunity to examine a demonstration version of the CDSS integrated into laptop computers. The CDSS was revised by the study team based on feedback from practice liaison personnel before implementation in each practice's EHR. All practices were required to install the CDSS into their EHR within a week of the initial meeting and the research team checked to make sure that this went smoothly.
A follow-up project meeting with all of the participating practices' liaison personnel and the study investigators was held in most cases in month 10 of the intervention and ‘best practice’ strategies of practice liaison personnel in overcoming barriers related to implementation of the CDSS and adherence to guidelines for antibiotic prescribing were discussed. The participating practices each hosted two half-day site visits during the first year of the intervention, in which members of the study team came to the practice. Initial site visits to the participating practices were conducted during the first 2 months of the intervention, and follow-up site visits were held during months 9 through 11, immediately before the second ARI season. At these site visits quarterly reports on use of the CDSS and antibiotic prescribing were reviewed, antibiotic prescribing guidelines were presented, and CDSS training was delivered to providers and staff of the participating practices. During site visits, the research team could also make minor modifications to the CDSS to accommodate the practices' workflow and meet the needs of the practice providers. Monthly phone calls were made to each of the intervention sites to determine problems with the implementation of the CDSS, and its acceptance by the staff.
The control group did not receive any information on the intervention, the CDSS, or educational materials. The control practices were unaware of the intervention and were simply observed during the study period using data captured in the EHR.
Clinical decision support system
The CDSS was designed by the research team as a PP EHR progress note template to be available at the point of care. PP includes the ability to customize progress note templates; creating this CDSS tool used existing features of the EHR, and no additional programming by the vendor was required.
The CDSS tool reflects guidelines from the CDC ‘Get Smart’ programme.15 Recommendations are based on the patient's predominant presenting symptoms and the patient's age. Decision support is provided in several ways. First, the CDSS includes diagnostic criteria for various ARIs to assist providers with making the appropriate diagnosis. For example, the template guides providers to distinguish the common cold from acute sinusitis based on duration of symptoms and physical examination findings. Scoring strategies (eg, Centor criteria for streptococcal pharyngitis) are also embedded within the template to facilitate diagnosis.16 Second, the CDSS template is used voluntarily by the provider when writing the note and is pulled up when providers determine that a patient has an ARI. A provider can write a note and make a diagnosis or treatment decision without using the template if they choose to bypass the template. As mentioned above, the template helps the provider in deciding on the appropriate diagnosis. Once a diagnosis has been made, the CDSS includes prompts about appropriate antibiotic use, and, when appropriate, recommended first-line antibiotics. Finally, embedded hyperlinks provided access to printable patient and provider education from the CDC. A screen shot of the CDSS and the sequence of screens is shown in figure 1.
Measurement of appropriate/inappropriate antimicrobial use
Diagnoses of an acute respiratory infection were extracted for all active patients (defined as having had a visit within 1 year and not designated as deceased, transferred, or inactive) within these PPRNet practices. PP primarily uses free-text diagnoses, although some templates used by PPRNet practices do have ICD-9 codes linked to diagnoses. Free-text diagnoses and ICD-9 codes abstracted from patients' electronic medical records are routinely ‘mapped’ by a physician into a standardized diagnosis data dictionary by the study team, which we used as the basis for the ARI classifications (see online Appendix). As part of PPRNet's normal data management and analysis processes, quarterly extracts are analyzed using complex algorithms to correctly and appropriately identify diagnoses, including searching for specific text strings and ICD-9 codes. This logical mapping and linking of text strings with ICD-9 codes is performed by clinicians within the PPRNet leadership and agreed within the PPRNet team before implementation by the data analysts.
An episode of care was defined as the 3-day period beginning with diagnosis of an ARI. Diagnoses for which antibiotics are usually considered inappropriate were designated as appropriate if an antibiotic-appropriate diagnosis was made on the same day or up to 3 days later.
Following CDC guidelines, diagnoses for which antibiotics are generally inappropriate comprise nonspecific upper respiratory infections, acute bronchitis, acute pharyngitis (but not streptococcal pharyngitis (group A β-hemolytic streptococcal pharyngitis)), and otitis media with effusion. Diagnoses for which antibiotics are indicated comprise acute sinusitis, streptococcal pharyngitis, pneumonia, acute otitis media, and chronic obstructive pulmonary exacerbations (in adults only).
Prescribed antibiotics were classified as either narrow or broad spectrum. Broad-spectrum antibiotics included amoxicillin/clavulanate, azithromycin, clarithromycin, second- and third-generation cephalosporins, and quinolones.17
All other antibiotics were classified as narrow spectrum; these included amoxicillin, penicillin, first-generation cephalosporins, tetracyclines, erythromycin, and trimethoprim/sulfamethoxazole. Intravenous formulations, polymyxins, aminoglycosides, and antimycobacterial agents were excluded from the analysis.
This primary outcome was limited to ARI episodes involving diagnoses for which antibiotics are inappropriate, and it was calculated by dividing the number of ARI episodes with diagnoses in the ‘inappropriate’ category that included an antibiotic prescription by the total number of ARI episodes with diagnoses for which antibiotics are ‘inappropriate’.
Broad-spectrum antibiotic use
An additional secondary outcome was calculated by dividing the number of all ARI episodes (episodes considered either inappropriate or appropriate for antibiotics) with a broad-spectrum antibiotic prescription by the total number of ARI episodes with an antibiotic prescription. Broad-spectrum antibiotics are of particular interest because frequent use of these agents promotes bacterial resistance.
Two measures were used to assess whether prescribers changed their patterns of diagnosing during the study. One was calculated by dividing the number of ARI episodes with diagnoses for which antibiotics are appropriate by the total number of ARI episodes with an antibiotic prescription. The other measure was calculated by dividing the number of ARI episodes for which antibiotics are inappropriate by the total number of ARI episodes.
In addition to the outcome measures, because the CDSS templates were not required to be used in an office visit by the EHR, we also measured within the intervention practices how often the CDSS templates were used and examined whether the study outcome measures defined above differed by CDSS template use/non-use.
Study outcomes were measured quarterly for each practice. Practice-level observations were weighted by the number of ARI episodes during the quarter. In this manner, more weight was given to practices with greater numbers of relevant episodes and to time points (within practices) that involve greater numbers of relevant episodes.
Weighted means and 95% CIs were determined for intervention and control practices at each time point, separately for adult (aged ≥18 years) and pediatric (<18 years of age) patients. Baseline means in study outcome measures were compared between intervention and control groups using weighted independent-sample t tests. Linear mixed models (LMMs) for longitudinal analyses were then used to compare changes among intervention and control practices across the 18-month study period (baseline + 15 months of follow-up).18 All models included time and an indicator variable representing intervention/control status, and all models adjusted for practice characteristics, including practice specialty (family medicine vs internal medicine), number of providers in the practice, region and baseline number of ARIs. To examine whether intervention practice outcomes changed more during the study than control practices, the interaction term involving time and intervention/control status was the primary parameter of interest in each model. Random practice effects were included in the LMMs to account for the clustering of repeated measures on practices over time. All statistical analyses were performed using SAS V.9.2 (SAS Institute Inc), and p values <0.05 were considered statistically significant.
Table 1 shows the practice characteristics at baseline. The median number of providers is the same for the intervention and control groups, and both groups had a median number of adult ARI episodes greater than 300 in the baseline period. The median numbers of ARI episodes for each practice are listed by quarter in table 2, along with the median number with diagnoses for which antibiotics were inappropriate, reflecting the seasonal trends in ARIs. At baseline, the weighted percentage of ARI episodes with diagnoses for which antibiotics were inappropriate that included an antibiotic prescription was similar (p=0.27) between adult patients in intervention (mean: 46.2%; 95% CI 33.4% to 58.9%) and control practices (mean: 53.0%; 95% CI 48.5% to 57.5%), and different (p=0.04) between pediatric patients in intervention (mean: 21.6%; 95% CI 10.8% to 32.3%) and control practices (mean: 34.3%; 95% CI 29.9% to 38.8%).
Figure 2A,B illustrate the changes in the inappropriate prescribing outcome over time for adult and pediatric patients. In adult patients, results from the LMMs indicated that the percentage of ARI episodes with diagnoses for which antibiotics were inappropriate declined to a significantly greater degree among adult patients in intervention practices than among those in control practices (−0.6% vs +4.2%, p=0.03), a trend which was not seen among pediatric patients (+1.4% vs +4.2%, p=0.34).
Figure 3A,B presents the impact of the intervention on prescribing of broad-spectrum antibiotics for ARIs. At baseline, broad-spectrum antibiotic use was similar (p=0.19) among adult patients in intervention (mean: 65.5%; 95% CI 57.1% to 74.0%) and control (mean: 71.8%; 95% CI 68.3% to 75.3%) practices, and similar (p=0.62) among pediatric patients in intervention (mean: 46.1%; 95% CI 36.6% to 55.5%) and control (mean: 49.1%; 95% CI 44.6% to 53.6%) practices. Broad-spectrum antibiotic use declined to a significantly greater degree in intervention practices than in control practices, for both adult (−16.6% vs +1.2%, p<0.0001) and pediatric patients (−19.7% vs +0.9%, p<0.0001).
Although there were some significant baseline differences in the use of diagnoses appropriate for antibiotics among pediatric patients, results of the LMM also indicated that there was little evidence of diagnostic shift during the study in either the intervention or control practices (tables 3 and 4). In adult patients, the percentage of ARI episodes with an antibiotic prescription that included diagnoses for which antibiotics are appropriate increased by an average of 2.4% in intervention practices compared with 0.5% in control practices (difference between groups = 1.9%; 95% CI −2.1% to 5.9%; p=0.35), and the percentage of ARI episodes (out of all ARI episodes) with diagnoses for which antibiotics are typically inappropriate increased by an average of 0.4% in intervention practices compared with a decrease of 0.2% in control practices (difference between groups = 2.5%; 95% CI −0.8% to 5.9%; p=0.14). In pediatric patients, the percentage of ARI episodes with an antibiotic prescription that included diagnoses for which antibiotics are appropriate decreased by 1.6% in intervention practices compared with a 2.5% increase among control practices (difference between groups = −4.1%; 95% CI −9.7% to 1.5%; p=0.15), and the percentage of ARI episodes (out of all ARI episodes) with diagnoses for which antibiotics are typically inappropriate increased an average of 8.6% in intervention practices compared with a decrease of 5.0% in control practices (difference between groups = 13.6%; 95% CI 8.9% to 18.3%; p<0.0001).
Within the intervention practices, template use for ARI episodes increased from 0% to 57.8% by March 2010, with a slower increase during the rest of the study period, to 68.5% by March 2011. Template use did not appear to influence two of the outcomes (inappropriate prescribing and diagnostic shift); however, by the end of the 15-month study period, adult and pediatric subjects with ARIs were less likely to receive a broad-spectrum antibiotic if the template was used (adult patients: 45.9% vs 56.8%, p<0.04; pediatric patients: 24.6% vs 35.0%, p<0.0001).
The results of this project indicate that a CDSS embedded in an EHR to improve antibiotic prescribing for ARIs in primary care had a modest effect in changing prescribing for adults with diagnoses for which antibiotics are generally inappropriate but had a substantial impact on changing prescribing of broad-spectrum antibiotics for all ARIs among both pediatric and adult patients.
It could be hypothesized that physicians might bend the rules to be able to justify antibiotics. Thus, once they are told that they should not give antibiotics for certain conditions like common colds but that antibiotics may be appropriate for other conditions like sinusitis then the physicians will begin classifying patients with a group of symptom that they used to diagnose as colds and now call that sinusitis. The CDSS did not lead to playing the system by having providers ‘shift’ their diagnoses to justify prescribing antibiotics. In this project, the decision support may have actually worked to improve diagnoses because among pediatric patients in the intervention group, the proportion of diagnoses for which antibiotics are inappropriate increased. Over time, once the physician better understands the defining parts of the diagnosis then they may more accurately classify the group of symptoms as a cold rather than as sinusitis.
This modest impact of this intervention on antibiotic prescribing overall for ARIs for which antibiotics are generally inappropriate is not too dissimilar from the results of several previous studies.8 ,14 ,19 ,20 Although the effect on overall antibiotic prescribing for adult patients was statistically significant, because the effect was relatively modest, interpretations of the clinical significance of the finding as a successful strategy for changing practice should be made with caution. However, the intervention did have a dramatic effect on the use of broad-spectrum antibiotics for ARIs. This finding has considerable implications for improving quality of care. As the proportion of antibiotic prescriptions accounted for by broad-spectrum antibiotics has increased, the CDC has focused on decreasing the number of broad-spectrum antibiotic prescriptions in ARIs to control antibiotic resistance. Thus, an effective strategy to decrease broad-spectrum antibiotic prescriptions that could be easily implemented in EHRs would be particularly useful in improving quality of care. Broad-spectrum antibiotic use has been attributed as a factor in the rise in Clostridium difficile.21 ,22
This study has several limitations. First, there was the potential for diagnostic ascertainment bias, since the template facilitated recording of the diagnoses. A potential further confounding factor in regard to diagnoses might have been that the study period overlapped with the 2009 H1N1 influenza pandemic. That the total number of diagnoses did not differ much between the intervention and control groups mitigates this possible limitation. Second, when physicians are required to use a CDSS, it may improve prescribing.23 Although in this study in which the use of the CDSS was voluntary the adoption of the CDSS was 68%, other studies of CDSS have shown relatively low rates of its use.12 ,13 ,24 In a trial of the AHRQ quality dashboard, only 28% of the clinicians used the dashboard at least once, a rate much lower than in this study.20 Third, we used naturally occurring groups (volunteers in the intervention group and other similar practices in PPRNet) in this quasi-experimental trial. It is possible that the volunteers might be more oriented toward controlling antibiotic use. However, the two groups were similar at baseline so any particular orientation toward controlling antibiotic prescribing in the intervention group was not evident before the intervention. Further, in this quasi-experimental design we did adjust for a variety of potentially confounding variables, thereby increasing the validity of the research design.25 Fourth, the feedback on prescribing was given as practice level feedback rather than as individual prescribing data. It is possible that individual feedback might be more powerful in changing behavior. For this study the intervention was focused on the entire practice as a location of change because of the importance of the roles of multiple personnel in the practice in decisions to prescribe antibiotics for ARIs. Fifth, it is unclear whether the implementation of just a CDSS system into an EHR would be an effective strategy without the additional activities of performance reports and practice participation in site visits and project meetings. Findings from other CDSS implementation research suggest that dissemination of these tools without more robust implementation strategies may limit the impact of the CDSS.12 Additionally, other outcomes that might have been the result of changing antibiotic prescribing patterns, with potentially negative relapse rates and clinical sequelae, were not measured in the study and might have been unintended effects of the intervention.
In conclusion, this CDSS embedded in an EHR as a strategy to promote judicious use of antibiotics for ARIs in primary care was particularly successful in decreasing the use of broad-spectrum antibiotics. Future research focusing on the sustainability of this intervention and the impact of these practice changes on the prevalence of resistant organisms is needed.
We thank Steve Ornstein, MD, Cara Litvin, MD, Andrea Wessell, PharmD, and Lynn Nemeth, RN, PhD for their contributions with the clinical decision support system. We also thank Ruth Jenkins, PhD for her help with data management and Michele Knoll, MA for her help with insightful comments.
Funding This study was funded by the Agency for Healthcare Research and Quality, Contract No HHSA290200710015I.
Competing interests None.
Ethics approval Ethics approval was obtained from the Medical University of South Carolina Institutional Review Board for this research study.
Provenance and peer review Not commissioned; externally peer reviewed.