Use of a Personal Digital Assistant for Managing Antibiotic Prescribing for Outpatient Respiratory Tract Infections in Rural Communities

Introduction

Computerized clinical decision-support systems (CDSSs), which integrate clinical, patient, and reference information to provide assistance with decision-making in patient care, are receiving increasing attention as instruments for improving the care and safety of patients1 2 3. These systems have the ability to synthesize the vast quantity of information available to support evidence-based decision-making, and also have the potential to reduce medical errors and improve practitioner performance and patient outcomes1 2 3 4. Systems have already been successfully used in a variety of settings, providing support such as diagnostic assistance,5 6 reminders for immunizations and other preventive health measures,7 8 alerts for critical laboratory results,9 and assistance with drug dosing and prescription order entry.10 11 12 The effects of these systems on performance and outcomes, however, have thus far been inconsistent and understudied.1 13 14 15

The potential benefits offered by some computerized CDSSs, such as order entry systems, have led some groups to endorse their widespread adoption by healthcare organizations.16 Implementation of such technologies can be a particularly difficult task for healthcare organizations operating in resource-poor areas, such as small towns in rural areas of the United States.17 Nevertheless, the potential impact of CDSSs even in small rural healthcare organizations should not be underestimated, given that roughly 1 out of every 5 ambulatory visits in the U.S. occurs in a rural setting.18 Solutions for this healthcare setting would need to provide patient-specific clinical decision support in a system that is inexpensive, flexible, and adaptable to clinician work-flow. Portable handheld computers (personal digital assistants; PDAs) fit this description, and are currently being used in medicine and the health sciences for a variety of purposes, including patient tracking and medical and pharmaceutical reference.19 To date, however, few investigations have examined the convenience, flexibility, and economy of these devices for the purpose of patient-specific clinical decision support at the point-of-care.

Our work has focused on interventions to improve inappropriate antimicrobial prescribing for acute respiratory tract infections (RTIs) in the rural outpatient setting, as the overwhelming majority of these infections are viral in origin and do not benefit from antimicrobial therapy. Previously,20 21 we demonstrated that the repetitive use of algorithms for the diagnosis and treatment of RTIs, as part of a multifaceted intervention, resulted in improvements in outpatient antimicrobial prescribing for acute RTIs by rural healthcare providers. Although these algorithms were designed originally on paper, in the latter study21 a subset of providers were allowed to use the same algorithms programmed onto PDAs. Here we provide an additional analysis of the data collected only from those PDA tools, in an effort to gain a better understanding of how these providers used and accepted the PDA-based CDSS, and to more closely examine the richness of the PDA-derived data.

Methods

Design

This was an observational study to analyze the usage patterns and acceptability of a standalone PDA-based CDSS tool. The source data for this study come from a large, community-randomized trial of an intervention to improve antimicrobial prescribing in the rural ambulatory care setting.21 The intervention consisted of standalone clinical decision-support tools for the diagnosis and treatment of acute outpatient respiratory infections at the point of care. Three different CDSS tools were offered; two were paper-based and one was programmed on a handheld PDA device. Primary care providers at each study site were offered their choice of which CDSS tool they preferred. The tools were used between January, 2002 and March, 2004. The combined effect of all three decision-support tools on antimicrobial prescribing is reported elsewhere.21

The current study examines in more depth only the data collected using the PDA-based CDSS, as well as the response of PDA users to an exit questionnaire assessing their experience with these devices. The study was approved by the University of Utah Institutional Review Board and the Western Institutional Review Board (Olympia, WA).

Study Sites and Participants

Communities and participants were selected as part of the randomized intervention trial.21 Eligible communities were those in rural Utah and Idaho that contained at least one primary care clinic and inpatient facility. Each was located in a non-metropolitan area based on the definition from the U.S. Office of Management and Budget; that is, an area with a total population less than 100,000, consisting of cities with populations no greater than 50,000 each. Two smaller (population < 25,000) communities and one larger (population ≥ 25,000) community from each state (6 communities total) were randomly selected to receive the intervention, with six similar communities serving as controls. The remaining features of the randomized trial are described elsewhere.21

The CDSS tools were offered to all primary care providers in these six communities (n = 191); for our purposes, primary care providers included emergency room providers, family practice physicians, internists, pediatricians, nurse practitioners, physician’s assistants, and osteopaths. In all, 99 (52%) primary care providers opted to use the PDA-based CDSS. Some of the characteristics of these users are shown in Table 1.

PDA-based CDSS Tool

The handheld PDA-based CDSS tool was developed to address a variety of acute RTIs including pharyngitis, otitis media, bronchitis, sinusitis, nonspecific upper respiratory tract infection (URI), pneumonia, croup, and influenza. The tool generated diagnostic and therapeutic recommendations on the basis of provider-entered, patient-specific information about the suspected diagnosis, such as the presence or absence of specific symptoms and signs. Therapeutic recommendations included over-the-counter medications for symptom control as well as prescription antimicrobials. For pediatric patients, the advice was customized to the patient’s weight and age. For cases of pneumonia, the system also calculated the patient’s pneumonia severity index score.22 The devices used in our study were Palm™ m515 handheld PDA computers running Palm OS™ 4.1 (palmOne, Inc., Sunnyvale, CA).

The tools were introduced to primary care providers through educational lectures, small group meetings, and one-on-one interactions between the providers and physician members of the study team. Providers were asked to use the algorithms on at least 200 consecutive patients with acute respiratory infections. Data from cases entered into the PDA were collected via electronic transmission when users docked their PDA on synchronization cradles, and were stored in a Microsoft™ Access 2000™ database (Microsoft Corp., Redmond, WA). Providers were allowed to keep the PDA at the end of the study if they completed at least 100 cases.

PDA Data Analysis

The computerized CDSS allowed us to collect and analyze specific diagnostic, testing, and treatment data for each of the various respiratory diagnoses, more efficiently than could be done using chart review. For most diagnoses, the CDSS collected a combination of patient history, clinical presentation, and laboratory testing data, providing us with a ready dataset with which we could review patient-level factors and management decisions for each diagnosis category.

We were able to analyze these data for each of the top diagnoses encountered in the rural outpatient setting (pharyngitis, otitis media, bronchitis, sinusitis, and nonspecific URI). For the current manuscript, we chose to present our analysis from one example diagnosis, acute pharyngitis. We present these data here not primarily to review how well providers performed using the CDSS with patients presenting with symptoms of pharyngitis, but rather to demonstrate the breadth and depth of data that are readily captured by the CDSS at all stages of the patient encounter (symptoms, exam findings, test results, and treatment decisions).

Exit Questionnaire

At the end of the study period, all providers participating in the study were asked to complete an exit questionnaire assessing their opinions of the project, the management algorithms, and the intervention tools. The questionnaire consisted of an assortment of questions ranging from fixed choice, rating scales, and Likert scales, to open comments. A subset of six questions were targeted specifically toward users of the Palm-based CDSS, which are focused on in this report (see Appendix, available as an on-line document at http://www.jamia.org)

Statistical Methods

For each patient encounter during which the CDSS was used, the provider made a prescribing decision that either adhered or did not adhere to the CDSS recommendation, providing a binary outcome for analysis. Adherence was defined here as either (a) no antibiotic prescription when antibiotics were not recommended by the CDSS; or (b) a prescription for the same antibiotic at the same dose and for the same duration as that recommended by the CDSS when a prescription was indicated. To account for the correlation of these binary adherence outcomes (where outcomes for the same provider are more alike than outcomes between different providers), a random-effects logistic regression model was fitted to the data. The model contained a three-level structure with observations nested within provider, and provider nested within clinic.

To account for the correlation in the analysis, a second summary measure approach was used that took into account the expected increase in provider adherence with greater use of the tool. In this summary measure approach, the proportion of adherence for each provider’s first one-third of cases and subsequent two-thirds of cases was computed. This method collapsed the data for each provider into two continuous measurements, which were then compared using a paired t-test. The mean proportion of adherence for the first one-third and subsequent two-thirds of cases are reported, providing an alternative measure of effect. For both approaches, providers with less than 30 cases (n = 10) were excluded from the analysis.

All reported P-values are for a two-sided comparison. All statistical analyses were performed using the Stata™ 9.0 statistical software (StataCorp LP™, College Station, TX) and the GLLAMM program module23.

Results

Characteristics of CDSS Users

Ninety-nine primary care providers in rural areas of Utah and Idaho received the PDA-based CDSS for use at the point-of-care. The characteristics of these users are shown in Table 1. Based on the number of acute RTI cases completed and submitted using the tool, individuals were categorized into heavy (≥ 200 cases), moderate (100–199 cases), or light (< 100 cases) users of the CDSS, with approximately one-third of users fitting into each category. The majority of users were physicians (n = 71; 72%), most of whom were family practitioners (n = 35; 35%), internists (n = 14; 14%), pediatricians (n = 11; 11%), or emergency physicians (n = 10; 10%). Physician assistants, nurse practitioners, and osteopaths were also represented in the user base (Table 1).

The total number of cases completed using the PDA-based CDSS from January 2002 to March 2004 was 14,393. Although providers were asked to use the tool with at least 200 patients suspected of having acute RTIs, the mean number of cases per provider was 145.4 ± 95.0 (range: 1–594). On average, heavy CDSS users submitted 243.3 ± 74.4 cases, moderate users submitted 150.3 ± 35.4 cases, and light users submitted 45.3 ± 32.0 cases. More than half of the cases submitted (7,624; 53%) were from family medicine practitioners.

Antibiotic prescribing and adherence

Table 2 provides an overview of antibiotic prescribing, including adherence with CDSS recommendations, for the five most common diagnoses encountered (pharyngitis, otitis media, bronchitis, sinusitis, and nonspecific URI). We measured provider adherence with CDSS recommendations as the proportion of cases for which the antibiotic prescribed matched that recommended by the CDSS (including no prescription for diagnoses which do not require antibiotic therapy, such as viral pharyngitis). Adherence with CDSS recommendations for these five diagnoses was high, averaging 82% overall (range: 72% for sinusitis to 94% for URI). Situations of non-adherence most often occurred when an antibiotic was prescribed, but differed from that recommended by the CDSS in terms of agent, dose, or duration of therapy (n = 1342; 10% of all cases; data not shown). Notably, there were few cases where an antibiotic was prescribed even though one was not indicated (n = 895; 7% of all cases; data not shown).

For these five diagnoses, the overall proportion of cases for which an antibiotic was prescribed was 53% (Table 2). Antibiotic prescribing was low for diagnoses which typically do not require antibiotics (nonspecific URI and bronchitis; 6% and 26% respectively), but considerably higher for diagnoses which sometimes require antibiotics (pharyngitis, otitis media, and sinusitis; 63%, 91%, and 75%, respectively). When antibiotics were recommended, the agent suggested by the CDSS was frequently prescribed (69% for sinusitis and 79% for pharyngitis and otitis media; Table 2).

Diagnosis, testing, and treatment of acute pharyngitis

The PDA device allowed us to easily collect and analyze specific diagnostic, testing, and treatment data for the various respiratory diagnoses. Data from a representative example diagnosis, pharyngitis, are shown in Table 3. To assist in the diagnosis of streptococcal pharyngitis, we used a combination of patient history, clinical presentation (i.e., Centor criteria24), and testing (e.g., rapid streptococcal antigen tests and throat cultures) in our CDSS. The data in Table 3 are stratified by the number of Centor criteria indicated by the provider, with a higher number of Centor criteria associated with a higher likelihood of streptococcal pharyngitis.24 The majority of patients had either 2 or 3 Centor criteria (n = 1984; 72%), and these patients received the most testing. The rate of positivity of the rapid streptococcal test (and all tests combined) increased as the number of Centor criteria increased (Table 3). Eligibility for antibiotic therapy, defined as a positive test result or the presence of appropriate historical data, also increased with the number of documented Centor criteria, as did the percentage of patients prescribed an antibiotic. The exception to both was patients who presented with all 4 Centor criteria, among whom eligibility was less than expected (47%) and antibiotic prescribing was high (95%). This was likely due to the low rate of testing performed on this group (only 36% of patients with 4 criteria received testing) and the fact that it is not universally accepted that testing is required for such patients25. Also of note, almost one quarter (23%) of patients with no Centor criteria and no positive test result were still prescribed an antibiotic (Table 3).

When providers chose to deviate from the treatment recommendation offered by the CDSS, they were asked to record the alternate antibiotic they prescribed. The information in Table 4 summarizes these data for the example diagnosis (pharyngitis). For cases of streptococcal pharyngitis, for which either amoxicillin or penicillin was recommended as first-line therapy, providers indicated they prescribed either erythromycin (n = 48; 15%) or cephalexin (n = 42; 13%) instead most often, both of which were recommended as second-line agents by the CDSS. For cases of viral pharyngitis, for which no antibiotics should be given, providers instead prescribed amoxicillin or penicillin most often (n = 101; 47%). Providers failed to record their prescribed antibiotic in a substantial number of cases (Table 4).

Adherence with CDSS recommendations over time

To assess whether adherence with CDSS recommendations improved as users became more familiar with the system, we sought to detect a change in overall provider adherence from the first one-third of their submitted cases to the second two-thirds of their cases (top portion, Table 5). Because of the design of the analysis, we chose to exclude providers who submitted fewer than 30 cases (n = 10 providers). Overall adherence with CDSS recommendations was 81.1%, increasing from 79.3% in the first one-third of provider’s cases to 82.0% in the second two-thirds (an increase of 2.7%; P = 0.016). Total adherence was higher with diagnoses for which an antibiotic was not indicated (84.8% vs. 75.7% for diagnoses warranting antibiotics), and providers showed a significant improvement in adherence over time for cases not requiring antibiotics (an increase of 2.7%; P = 0.039).

As an alternative method of measuring the association between CDSS use and adherence with CDSS recommendations, we fitted a random-effects logistic regression model to the data using adherence as the binary dependent variable (bottom portion, Table 5). Each set of 10 completed cases was associated with a 2% increase in the odds of being adherent with CDSS recommendations (P = 0.001). When each individual set of 10 completed cases was entered into the model, we found that almost every set of 10 completed cases from the 4th set beyond was associated with a significant increase in the odds of being adherent with CDSS recommendations (Table 5).

Exit Questionnaire Results

Exit questionnaires were distributed to all CDSS users at the end of the study period; responses were received from 65 individuals (66%). Although there was no significant difference between survey respondents and non-respondents in terms of demographics (specifically, graduate degree or specialty; data not shown), those who responded submitted a higher mean number of cases than those who did not (167.5 vs. 112.7; P = 0.004). More than half of respondents (n = 36; 55%) reported prior experience with a PDA device.

We asked a series of questions related to the ease of use and acceptability of the PDA-based CDSS tool (see Appendix, available as an on-line document at http://www.jamia.org). When asked to rate the ease of use of the tool on a scale from 1 (very difficult) to 5 (very easy), users responded with an average score of 4.6 regardless of their prior experience with a PDA device. Users were also asked to rate the degree to which they believed the CDSS algorithms reflected acceptable guidelines for the diagnosis and treatment of acute respiratory infections; on a scale from 1 (many concerns) to 5 (completely acceptable), users responded with an average score of 4.0. Similarly, users were asked to assess how closely the CDSS algorithms reflected their prior practice patterns. On a scale from 1 (not even close) to 5 (complete match), users gave an average response score of 3.5. We observed no differences between groups when these measures were stratified by user type (light, moderate, or heavy users; data not shown).

There was some concern among providers at the start of the study that using the CDSS tool at the point-of-care would increase the time needed to complete each patient encounter for respiratory illnesses. We therefore asked providers in the exit questionnaire to estimate the average time per patient encounter (for respiratory illnesses) while using the CDSS tool compared with their average time per encounter prior to using the tool (see Appendix, available as an on-line document at http://www.jamia.org). Just over two-thirds of users (n = 44; 68%) reported that patient encounters were either the same duration (n = 33), or slightly faster (n = 11), when using the CDSS tool compared with their usual practice. In contrast, only 32% of users (n = 21) believed that patient encounters took slightly longer, and no respondents answered with “much longer”. When respondents were stratified by their level of PDA use (light, moderate, or heavy), a smaller proportion of light users gave a favorable response (“slightly quicker” or “about the same”) when compared with moderate or heavy users (50% vs. 74% and 69%, respectively; Figure 1). Of note, respondents with no prior PDA experience were more likely to report that their encounter times were about the same or slightly quicker (23/29 respondents; 79%) than users with prior PDA experience (21/36 respondents; 58%), although this difference was not significant (P = 0.07; data not shown).

Finally, we surveyed users about the types of problems they encountered while using the CDSS tool (see Appendix, available as an on-line document at http://www.jamia.org). Of the 65 surveyed CDSS users, 22 (34%) reported at least one problem, including 13 of the 36 users (36%) with some prior PDA experience and 9 of the 29 users (31%) with no prior PDA experience (data not shown). The most commonly encountered problem was difficulty with downloading data to and from the PDA (25% of individuals with prior PDA experience and 28% of individuals with no prior PDA experience). Most other problems, including hardware malfunctions or problems with technical support, were limited to a small number of users (range, 0–4 users per problem). Of note, none of the 65 CDSS users reported any problems learning to use the system, even if they had no prior experience using a PDA (data not shown).

Discussion

Computerized CDSSs have the potential to improve practitioner performance as well as patient safety and outcomes.1 4 However, they have yet to achieve widespread adoption by healthcare organizations. Foremost among the reasons for this is the prohibitive up-front and maintenance costs of many of these systems, particularly for organizations in resource-poor areas.17 26 27 In the current study, we demonstrate that a simple, standalone computerized CDSS designed for the PDA for the management of acute outpatient RTIs was a suitable tool for encouraging better antimicrobial prescribing in rural ambulatory care clinics. The system used in our study consisted of only a PDA device loaded with the custom CDSS software package, and did not require integration with an existing practice management system, electronic medical record, or network; as such, the cost of setup, installation, and maintenance was minimal. Nevertheless, the system proved to be an effective way to provide fast management recommendations at the point of care.

The CDSS generated case-specific recommendations which incorporate individual-level patient data, a feature that differentiates our system from reference programs or guidelines that are not patient-specific and therefore are not classified as a CDSS.3 13 28 The CDSS required providers to manually input symptoms, signs, and test results, as well as to confirm management choices made. Although manual data entry might have been viewed as an inconvenience, the difficulties of data entry were minimized by an efficient, streamlined interface that produced rapid recommendations. That the system succeeded in this regard is reflected, at least in part, by the rated ease of use of the system and by the fact that a majority of users believed that using the CDSS did not increase the time required per patient encounter (Figure 1), important features identified as keys to the success of any point-of-care CDSS.2 29 30 31

The system was designed to improve antibiotic prescribing for acute outpatient RTIs through the repetitive use of computerized algorithms to help guide diagnostic and therapeutic decision-making. We sought to accomplish this by asking providers to use the CDSS with at least 200 consecutive patients with symptoms of an acute RTI. This amount of formal use of the CDSS algorithms was felt to provide sufficient repetition on the management of common RTIs and to help physicians break previous clinical habits that were not consistent with judicious antibiotic prescribing. Although only about one-third of the providers in our study completed the requested 200 cases using the CDSS, the exact number of repetitions required to bring about change in antibiotic prescribing behavior is probably individually defined. However, our analysis of CDSS use both here and elsewhere21 suggests that its effect was related to the frequency of use.

We attempted to demonstrate the adoption of CDSS recommendations by measuring individual providers’ adherence with the recommendations as their use of the tool increased. Our logistic regression models indicate that provider adherence did improve slightly with each ten cases entered into the system, with most of the benefit realized by the fourth or fifth set of ten cases (Table 5). Moreover, we observed that provider adherence improved from the first one-third of their CDSS cases to the last two-thirds of their cases (Table 5). This comparison may actually underestimate the level of the effect, since provider performance during the first one-third of their cases was already influenced by the CDSS. An additional measure of behavior change—a comparison of provider performance before and after introduction of the CDSS—was also explored for the analysis of our community-randomized intervention trial,21 which showed that the magnitude of decrease in antimicrobial prescribing correlated with the number of cases completed using the CDSS. This analysis, however, included providers who used both paper-based and PDA-based CDSS tools, and was limited only to those diagnoses for which antibiotics are rarely indicated (e.g., acute bronchitis and URI).21

While the overall antibiotic prescribing rate for the five most common diagnoses (pharyngitis, otitis media, sinusitis, bronchitis, and nonspecific URI) observed in our study (53%; Table 2) is similar to that reported by others in the absence of a CDSS (approximately 54%–55%),30 31 we observed much lower antibiotic prescribing rates for bronchitis (26%) and URI (6%) than rates reported elsewhere (59%–62% for bronchitis and 30% for URI).32 33 This might reflect a greater influence of the CDSS on diseases for which antibiotics are not typically indicated. This contrasts with our observed antibiotic prescribing rate for otitis media (91%), which is higher than that observed by others (68%33 to 76%), 32 and which potentially explains our similar overall prescribing rates. It should also be noted that, when antibiotics were indicated, we observed a high degree of adherence with the regimen suggested by the CDSS (Table 2), which specifically emphasized the use of narrower-spectrum agents such as penicillin or amoxicillin. This is an important consideration in light of recent reports that suggest providers are increasingly prescribing more expensive, broad-spectrum antibiotics for these illnesses.33 34

In addition to the other advantages mentioned, the PDA-based CDSS offered the ability to easily collect and analyze case-specific data pertaining to the diagnosis and management of different acute RTIs, such as acute pharyngitis (Tables 3 and 4). In our experience, these data are considerably more laborious to collect, and typically less complete, using other methods such as manual chart review. Using these data, we observed that the distribution of patients across the number of documented Centor criteria in this study was greater for patients with 2 or 3 criteria, and less for those with 0 or 1, than similar distributions reported elsewhere.35 This suggests that providers were perhaps less likely to use the CDSS for cases that had a lower likelihood of streptococcal pharyngitis. We also observed that, even among patients with no Centor criteria and no positive test result, the antibiotic prescribing rate was still 23% (Table 3). Thus, even while using our CDSS, providers continued to prescribe antibiotics inappropriately, suggesting that some providers might be more resistant to behavior change than others.

Our study had certain limitations. Certainly, the collection and use of self-reported case data limits our ability to draw conclusions from our analyses about the patient population observed due to biases. In particular, case selection bias is likely present, as providers could have used the CDSS on non-sequential (and thus non-random) patients, such as those that were either more (or less) difficult to diagnose, or those that are more (or less) likely to require an antibiotic. In the absence of an independent chart review, we are also unable to confirm that providers actually completed the actions they indicated in the CDSS. Also, true change in prescribing behavior was not assessed in this study, as data on each provider’s antibiotic prescribing habits before and after the introduction of the CDSS were not collected. Additionally, no conclusions can be made regarding the long-term effects of the tool on antibiotic prescribing behavior until future individual provider prescribing data are analyzed.

Outpatient antibiotic prescribing still has tremendous room for improvement despite reports of a decrease nationwide over the past few years.33 34 It remains an important influence on antimicrobial resistance in the community, as well as a robust target for continued efforts at clinical practice change. Although the CDSS used in this study shows promise as a tool for encouraging better antimicrobial prescribing practices and efficiently collecting clinical data, future efforts should focus on ways of more seamlessly integrating decision-support logic into outpatient clinic work flow, such as through standalone electronic prescribing programs. Continuing to develop tools like the one used in this study may prove to be a useful solution for improving antimicrobial prescribing among healthcare organizations in resource-poor areas that lack fully computerized medical records.