Controlled Trial of Direct Physician Order Entry
Effects on Physicians' Time Utilization in Ambulatory Primary Care Internal Medicine Practices
- Correspondence and reprints: J. Marc Overhage, MD, PhD, Regenstrief Institute for Health Care, 5th Floor, Regenstrief Health Center, 1050 Wishard Boulevard, Indianapolis, IN 46202; e-mail: < >
- Received 6 February 2001
- Accepted 16 March 2001
Objective Direct physician order entry (POE) offers many potential benefits, but evidence suggests that POE requires substantially more time than traditional paper-based ordering methods. The Medical Gopher is a well-accepted system for direct POE that has been in use for more than 15 years. The authors hypothesized that physicians using the Gopher would not spend any more time writing orders than physicians using paper-based methods.
Design A randomized controlled trial of POE using the Medical Gopher system in 11 primary care internal medicine practices.
Measurements The authors collected detailed time use data using time motion studies of the physicians and surveyed their opinions about the POE system.
Results The authors found that physicians using the Gopher spent 2.2 min more per patient overall, but when duplicative and administrative tasks were taken into account, physicians were found to have spent only 0.43 min more per patient. With experience, the order entry time fell by 3.73 min per patient. The survey revealed that the physicians believed that the system improved their patient care and wanted the Gopher to continue to be available in their practices.
Conclusions Little extra time, if any, was required for physicians to use the POE system. With experience in its use, physicians may even save time while enjoying the many benefits of POE.
Thirty years ago, Morris Collen argued, “Physicians should enter medical orders directly into the computer.”1 More recently, the Institute of Medicine affirmed this view by stating, “Ideally, data in the [computer-based patient record] should be entered at its source by the record's primary user,”2 and employers have identified direct physician order entry (POE) as a key quality improvement strategy.3 Technical limitations and process barriers have prevented most organizations from achieving Dr. Collen's goal,4 and efforts to require physicians to enter orders, create problem lists, and enter progress notes have met with varying degrees of success.4 5 6 7
One of the major barriers to POE system use is the additional physician time it requires, compared with traditional paper-based methods.8 9 10 Any change that requires more physician time will meet with significant resistance from physicians—time appears to be the primary determinant of physician satisfaction with order entry.11
The Medical Gopher system, which has been used for 15 years in primary care internal medicine practices associated with Wishard Memorial Hospital, is a mature, well-accepted POE system. Throughout its development, we have tuned dozens of aspects of the Gopher to make it fast and easy to use. Even so, in 1993, we observed that, in the inpatient setting, entering daily orders using the Gopher required twice as much time as entering handwritten orders. With improvements we have made since that time, we hypothesized that the time required to use the current version of the Gopher would be comparable with that for traditional paper-based methods. To test this hypothesis, we carried out a randomized controlled trial of physician time use for POE using the Gopher compared with paper-based methods.
The Institutional Review Board of Indiana University–Purdue University at Indianapolis approved the study. We conducted the study in practices that are part of Indiana University Medical Group–Primary Care (IUMG-PC) from October 1996 to February 1998. The IUMG-PC is a physician-led primary care group that includes internists, pediatricians, family practitioners and obstetrician/gynecologists who deliver care to patients in 13 practices at 12 office locations in central Indiana. The group is an affiliate of the Indiana University School of Medicine (IUSM) and Wishard Memorial Hospital, and all physicians are faculty members of the IUSM. The IUMG-PC includes two types of practices—“community health centers,” which provide care primarily for uninsured and underinsured patients in comprehensive settings that include social work, nutrition, laboratory, pharmacy, radiology, and other services; and “commercial” practices that serve primarily a commercially insured population in a typical private practice setting.
Prior to the study, all providers handwrote their patient notes on encounter forms that were generated specifically for each patient encounter by the Regenstrief Medical Record System (RMRS).12 13 They wrote prescriptions either on standard prescription forms or on paper medication forms generated by RMRS and wrote test orders longhand in a designated area of the encounter form. Data entry personnel manually abstracted selected data, such as vital signs, and recorded them in the RMRS. Office personnel generally completed test requisitions. Physicians typically completed all documentation for an encounter before the patient left the practice. Clinicians in two of the practices (at a single location) have used various versions of the Medical Gopher system for POE since 1984.14 The RMRS provided a nearly complete electronic medical record for the physician in these practices, with access to diagnostic test results, treatment information, dictated clinician's notes, reports, and other data. A traditional paper chart was maintained and was available to the clinician at each patient encounter.
We conducted a randomized, controlled trial of the Medical Gopher10 14 system. The Gopher provides order entry, clinical documentation, diagnostic result review, access to clinical knowledge bases, and links to billing functions. After leaving the examination room, the provider enters these data on workstations in a shared workroom or in the provider's office, using a series of forms, including the one shown in Figure 1. There were no computers in the examination rooms.
For each encounter, the provider records billing diagnoses, medications prescribed, changes in allergies, test orders, referrals, nursing intervention orders and, optionally, notes. The system stores this information in a clinical data repository (the RMRS) from which providers can view the information as coded data or as a note with a more traditional appearance. The Gopher then uses the data to create billing documents (superbill), requisitions, consult requests, prescriptions, and patient education materials. When entering orders, including prescriptions, physicians can create instructions by typing them in free text, choosing fill-in-the-blank templates, or using their previous instructions for an order. They commonly use the last method when refilling medications.
In the two practice types (community health center and commercial site), we used a random number generator to choose, as intervention practices, those in which physicians had never used the Gopher. (We excluded the two IUMG-PC practices that have been using POE for many years.) The remainder served as controls. In the intervention practices, we implemented the Gopher system for POE. In each practice we phased-in the implementation, which required approximately 2 to 3 months for hardware installation, configuration, and user training. Consequently, Gopher implementation in the intervention practices spanned approximately 1 year. Because of a dramatic turnover of physicians and practice manager at one site, we did not succeed in implementing the Gopher there. Physicians received approximately 30 minutes of “just-in-time” training by a clinical information specialist at their practice when the Gopher was initially implemented.
Because physicians' work is very fragmented and the duration of time spent on a specific task is very short,15 we chose a time-motion study rather than work-sampling methods.10 16 Observation periods were one half day. We selected this interval because physicians' schedules are structured in half-day blocks, and some physicians had other responsibilities, such as administrative work or inpatient responsibilities, during some half days. In addition, physicians would typically have 8 to 10 patients scheduled for a half-day block, which ensured a somewhat representative sample of patient care.
We invited all full-time internists in the intervention and control practices to participate. Residents and fellows were not included in the study population. We chose to focus on internists because of the complexity of their task environment and to eliminate specialty as a source of variability. We attempted to observe each provider on at least three different days over a 2-year period, to allow for the day-to-day variability in physician patient loads and other non-physician factors. On some occasions, because of scheduling limitations, we observed the same physician for two consecutive half-day periods. One investigator (J.M.O.) selected observation dates on the basis of the observers' and providers' schedules. The observers attempted to begin their observations before physicians saw their first patients during the half-day block and to continue observations until the physicians had finished care for their last patients. Office personnel explained the observer's role to patients while directing them to an examination room, to limit disruptions to the physician's routine; however, most physicians introduced the observer to the patient.
We recruited observers among senior medical students at the IUSM. We selected senior medical students as observers because they are familiar with clinical practice, which enables them to identify and categorize clinical activities with less training; they are comfortable with the sights, sounds, and processes encountered in clinical settings; patients were comfortable with their presence; and they had 1-month blocks of vacation time to devote to observation. Observers were aware only that we were “attempting to understand how doctors use their time.”
We developed the activity classification categories at our institution over several years. They have been used in time motion studies of house staff physicians' inpatient activities12 and, in modified form, for physicians' outpatient activities.17 Initially, we developed a candidate list of categories during a series of “brainstorming” meetings of four internists/informatics specialists and review of published10 17 18 19 and unpublished* provider time categorizations. Two physician researchers (W.M.T. and J.M.O.) tested the candidate categories and refined the list on the basis of their experience, by using them to categorize physician activities during observation. We also took into account recommendations made by others about the degree of granularity of information it was necessary to obtain.18 We developed operational definitions for the final list of categories to assist observers in classifying activities into the correct category. We clarified and, in one case, modified these definitions during observer training.
A major criterion in developing the categories was the ease with which the observers could classify the activity the physician was beginning. This resulted in categories that correspond to visible, physical activities, such as “using the telephone.” We then developed subcategories to capture the specifics of the activity, such as “talking to a patient” and “obtaining a test result” as subcategories of “on the phone.”
These major categories facilitated observation but were much less useful for analyzing the data. We therefore developed analysis categories that regrouped category–subcategory pairs into categories that were easier to interpret. The criteria for creating the analysis categories were to define clinically meaningful groupings that had face validity to a clinical manager; to be consistent with the literature to allow comparison; and to reflect aspects of the care process which we expected POE to change. The analysis categories divide tasks into Direct Patient Care, Indirect Patient Care, Administrative, and Miscellaneous categories. Direct patient care tasks included talking with and examining the patient. Indirect patient care tasks included reviewing data and recording data in support of care of an individual patient. We further subdivided the Indirect Patient Care category into Write, Read, and Other subcategories, since we expected the Gopher to specifically affect order-writing times, which we included in the Write subcategory along with writing notes. Administrative tasks included tasks such as reading mail and discussing practice management issues with the staff. Miscellaneous tasks included eating and personal conversations. The Appendix contains a complete list of the category, subcategory, and analysis categories we used.
To perform the time motion study, we used a palm computer (Newton Message Pad 120, Newton Inc., Cupertino, California) running a program we developed specifically for recording this data. The medical student observers used a stylus on the palm computer's touch-sensitive screen, eliminating the need for a keyboard and facilitating data entry while standing. At the beginning of each half day, the observers recorded the name of the physician they observed, their own identity, and the practice in which they made the observations. During the observation period, the observers selected categories and subcategories of activities from the lists presented on the palm computer. As soon as they selected a category, the computer recorded the activity's start time, to millisecond precision. The observers could then identify the subcategory at their leisure (Figure 2). We considered the start time of each activity to be the stop time of the immediately proceeding activity. When the observers finished the observation period, they entered a specific task (Start/Stop—Done for the day) to indicate the time the observation period ended. We transferred the data from the time motion observations from the palm computer to a personal computer for storage and analysis.
One of the authors (J.M.O.) trained all the observers. In addition to giving a didactic presentation on use of the palm computer, the investigator familiarized the observers with the categories and their operational definitions. The observers then carried out 1 to 2 hours of practice observations. After the practice observations, the investigator debriefed the observers, reviewed their observation results, and answered questions.
To assess physicians' attitudes about the effects of using the Gopher, we gave them a self-administered six-question survey. The providers indicated their agreement or disagreement with statements about the effects of the Gopher, using a scale of 1 to 5, where 1 indicated strong agreement, 3 indicated neutrality, and 5 indicated strong disagreement. We sent the survey and a cover letter by fax to each provider who used the Gopher. If they did not return the surveys within 1 week, we sent a second copy and one of the investigators (J.M.O.) called them to encourage them to complete it.
The data were read into a SAS (SAS Institute, Cary, North Carolina) data set for analysis. We computed task duration as the difference between the start time of one activity and the start time of the next activity. In addition, we extracted from our appointment system data on the number of patients seen by the physician during the observation period, and obtained detailed data about the physicians' interaction with the Medical Gopher from the Gopher log files. We generated descriptive statistics by intervention and control condition.
The primary outcomes were the physician's overall time spent per patient (defined as the total observation time divided by the number of patients seen) and the time spent on Indirect Patient Care–Write tasks per patient. To examine the potential effects of practice and provider, we used SAS to fit linear mixed models. The study design was provider nested within practice and practice nested within practice type (community health center or commercial). In the analysis, we treated practice and practice type as fixed effects and provider as a random effect.
In addition, we compared the physician's overall time spent per patient and the time spent on Indirect Patient Care–Write tasks per patient between the first and last intervention observations for each provider who had at least two intervention observations, using paired t-tests to determine whether there was a learning or experience effect.
We defined duplicative tasks as activities the physician continued to perform that were unnecessary, since the Gopher was performing the same function. For example, providers would complete a paper superbill, even though the Gopher generated a superbill automatically from the data they entered. Duplicative tasks persisted, because changing processes in clinical practices requires considerable time and effort and we simply had not yet changed the processes to eliminate these tasks. To obtain more refined estimates of the time effects of POE under ideal circumstances, we identified these tasks on the basis of category–subcategory pair, labeled them as duplicative tasks, and then repeated all analyses with these duplicative tasks removed. We also removed tasks classified as administrative or miscellaneous and repeated the above analyses for overall minutes spent per patient.
Of the 35 physicians invited to participate in the study, only one declined, citing a concern that many of her patients had chosen her as a provider because they wanted a female physician and most of the observers were male. We observed 34 staff providers (14 under intervention conditions at least once and 20 under control conditions) at 11 practices (Table 1) on 110 separate observation periods, for an average of 3.5 hours per observation period, or a total of 382 hours (almost 10 weeks) of observation. The physicians were observed on 2.8±1.9 occasions. The observations included 744 patient encounters, for an average of 6.8 patient encounters per observation period.
During intervention observation periods, the providers used the Gopher to write all prescriptions and test orders for all their patients.
Figure 3 shows the time required for physicians to care for patients under intervention and control conditions. Table 2, shows the total time spent per patient divided into the six main analysis categories. The distributions of time among these categories were similar for intervention and control conditions. The largest amount of time was devoted to Direct Patient Care tasks, such as examining the patient and talking to the patient. Indirect Patient Care–Write tasks, in which we expected POE to have the greatest effect, consumed the second largest amount of time. Miscellaneous tasks (such as using the rest room and social conversation, which are not patient care or administrative tasks) consumed approximately 20 percent of the overall time.
In the linear mixed model analysis, we found that the specific practice had no effect, so we removed this factor from the models. Practice type (commercial vs. community health center) was not always significant, but it was retained in all models because it was a stratification variable and because it was expected apriori to be significant. The random effect for provider was significant for overall time spent per patient (P<0.05). This implies that variation from physician to physician in overall time spent per patient is significant. The intraclass correlation ranged from 0.25 to 0.27. The intraclass correlation for Indirect Patient Care–Write tasks per patient was 0.16 but only marginally significant (P=0.15)
The models predicted that physicians using the Gopher would spend 2.12 minutes per patient (P=0.50) longer than when using traditional paper-based methods (an average of 36.3 minutes compared with 34.2 minutes per patient). When we removed the duplicative tasks, the intervention effect was 0.92 minutes (P=0.77). When we removed administrative tasks, the intervention effect was 1.63 minutes (P=0.47). When we removed both duplicative tasks and administrative tasks, the intervention effect was 0.43 minutes (P=0.85). For Indirect Patient Care–Write tasks, the intervention effect was 0.61 minutes (P=0.40) using all observations and -0.55 minutes (P=0.43) after removing duplicative tasks. Of the 14 physicians observed under intervention conditions, 12 had at least two intervention observations, allowing us to evaluate the effect of experience. When we compared times from first and last observations, we found that, on average, the physicians spent 3.73 fewer minutes per patient overall and 1.03 minutes fewer for Indirect Patient Care–Write per patient during the last observation than the first. However, these differences were not statistically significant.
Data extracted from the Medical Gopher for intervention providers during the observation periods showed that total time spent using the computer for orders and documentation was 2.28±2.18 minutes (range, 0.1 to 11.8 minutes) per encounter. Of this time, providers spent 54 to 98 percent on ordering and the remainder on reviewing clinical data, examining reference materials, and choosing billing diagnosis, evaluation, and management codes. Providers entered 1.14±1.28 problems and 0.02±0.16 allergies per encounter. They wrote 0.6±1.4 test orders (7.7±24.0 seconds) and 2.73±3.0 prescriptions (59±65 seconds) during each session. Providers chose one half of all orders from the various menus or sets of orders provided. They entered only 11 percent of instructions for tests and drugs by typing free text, whereas they created 41 percent of instructions by choosing from preconstructed “sigs” or instructions specific to the test or drug. The remaining 48 percent were “carried over” or copied from a previous “sig” for that order. Physicians reacted to the 20 drug interaction warnings provided by the system during the time they were observed by canceling the medication ordered 19 times (95 percent).
Intervention physicians completed 13 surveys (86 percent response rate) responding on a five-point Likert scale. They agreed that their work was done faster (1.92±0.90), easier (1.58±0.67), that the quality of documentation improved (1.42±0.67), workflow was improved (1.75±0.62) and quality of care improved (1.83±0.71). When asked if we should remove the Gopher from their practice, they strongly disagreed (3.82±0.40).
Overall, physicians randomized to use the Gopher spent 2.12 minutes (6.2 percent) longer per patient writing orders than did control physicians who used paper-based methods. Unfortunately, physicians in the intervention group continued to perform certain tasks using paper-based methods even though the computer was automatically performing those tasks for them. For example, they would, in some cases, manually complete a paper superbill even though the Gopher created an acceptable superbill for them. The time wasted on these unnecessary or duplicative tasks was more than a minute per patient. If we could eliminate this wasted time through further training and process improvement, we estimate the net time cost of POE system use would be only 0.92 minutes per patient. Furthermore, the above figures include the time for users early in their experience with the computer as well as later. We found that the time costs declined with experience, so the time costs for using the POE system could be very small.
Only a few other studies report time data for order entry, and several of those report the blended results from implementing POE and a computerized patient record. Medscape, for example, cites 20 percent productivity improvements with their Logician product when implementing order entry along with a computerized patient record.19 In the Kaiser Permanente Northwest region, clinicians spent an additional 2.2 minutes per patient when the organization introduced POE and a computerized patient record into their outpatient practices.8 These physicians used the system to order tests and prescriptions only 70 to 80 percent of the time. We and others20 21 have previously reported that POE at least doubled the time to record orders for inpatient services.
Most of the time required to use POE is the time to enter individual orders. Developers have tailored the user interface for POE to minimize the time required to enter individual orders in outpatient prescription writing. Reported average times to generate prescriptions using computer-based POE systems or paper-based systems range from less than 30 seconds for the simplest to 252 seconds for complex prescriptions.22 23
Most other data come from inpatient settings. In a study in Japan, investigators found that physicians and other personnel required 102±121 seconds to enter a new prescription, 76±109 seconds to enter a laboratory order, and 54±82 seconds for a radiological study.24 Physicians using the HELP system at LDS Hospital required 132 seconds per order.25 We previously reported average test-ordering times of 30 seconds per test on an earlier version of the Gopher used by medicine residents in an inpatient setting.14 The Gopher was much faster to use than any of these previously described systems, requiring only 20 seconds on average to complete a prescription. We attribute the speed with which physicians can write orders using the Gopher to parsimonious design, ease of locating desired items, and preconstructed or previous“sigs,” which the residents used almost 90 percent of the time.
Compared with their first observed use of POE, physicians' last observed use took 3.73 minutes less per patient. Others have reported similar results, stating that while POE initially requires more time to use than paper-based methods, this difference disappears over 4 to 8 weeks.26 Schroeder and Pierpaoli27 state that “after 1 or 2 weeks of use, most physicians find that they can enter orders in the computer terminals faster than they could write them by hand.” Finally, Allen et al.28 report that order times were 82 seconds for novice users and 18 seconds for expert users in an outpatient prescription writing system. Not only are physicians learning to use the program, they are usually also learning a vocabulary, discovering what the things they want to order are called in the system.
The Medical Gopher provided a number of benefits while requiring slightly more time than paper-based methods. First, workflow was improved: Ancillary personal did not have to complete requisitions, records were legible and always available, physicians wrote consequent orders more often, and the practice saved money on forms. We have also shown, in previous studies,29 30 31 that cost efficiency can be improved by decreasing the number of tests ordered. Third, clinicians received feedback, such as drug– drug and drug–diagnosis interactions, passive and active order menus, and academic detailing as well as reminders about relevant test results that, as published evidence demonstrates, improves the quality of patient care.32 33 34 Finally, clinicians had ready access—from home, from the hospital, and from secondary practice locations—to the data they recorded using the Gopher.
The study has some limitations. First, the results are based on use of a specific POE system and may not be generalizable to other systems. However, they do indicate that direct POE using a well-designed system may require only minimal additional time and may even, with continued use, represent a time savings. The Medical Gopher has been used in our outpatient practices for more than 15 years, and during that time has been carefully adapted to the patterns and processes of our practices. Approximately a quarter of the physicians in our study had some experience using the Gopher for inpatient order entry before the study.
Second, the providers in our study, who are salaried, may not be representative of physicians in private practice, who may be even more sensitive to time pressure. However, increasing numbers of clinicians are now salaried employees.35 While the practices we studied are affiliated with a medical school and do host some resident teaching activities, they are managed to be viable independent practices, and all but one are remote from the school's campus. In addition, the volumes of patient visits we observed are consistent with Medical Group Management Association (MGMA) benchmarks of 14 to 18 patients per day for general internal medicine practices.36 Third, the subjects of this study are general internists, and our results may not apply to physicians in other specialties, such as obstetrics or orthopedics.
In conclusion, little extra time, if any, was required for physicians to use the POE system in this study. With experience in its use, physicians may even save time while enjoying its many benefits. If the time required for POE use is comparable with the time required for use of paper-based systems, we have demonstrated that one of the major barriers to adoption can be overcome.
|Major Category||Minor Category||Analysis Grouping|
|Computer||Drug Reference||Indirect Patient Care-Other|
|Computer||Looking Up Data||Indirect Patient Care-Read|
|Computer||Reviewing Dictations||Indirect Patient Care-Other|
|Computer||Writing Note||Indirect Patient Care-Write|
|Computer||Writing Orders/Scripts||Indirect Patient Care-Write|
|Examine/Read||Chart||Indirect Patient Care-Read|
|Examine/Read||Drug Reference||Indirect Patient Care-Other|
|Examine/Read||EKG||Indirect Patient Care-Read|
|Examine/Read||Patient||Direct Patient Care|
|Examine/Read||Pocket Reference||Indirect Patient Care-Other|
|Examine/Read||Radiograph||Indirect Patient Care-Read|
|Forms||Consult||Indirect Patient Care-Write|
|Forms||H & P||Indirect Patient Care-Write|
|Forms||Insurance||Indirect Patient Care-Write|
|Forms||Other||Indirect Patient Care-Write|
|Forms||Requisition||Indirect Patient Care-Write|
|Forms||Superbill||Indirect Patient Care-Write|
|Forms||Work/School Release||Indirect Patient Care-Write|
|Looking for||Chart||Indirect Patient Care-Other|
|Looking for||Colleague||Indirect Patient Care-Other|
|Looking for||Consultant||Indirect Patient Care-Other|
|Looking for||Lab Result||Indirect Patient Care-Other|
|Looking for||Other||Indirect Patient Care-Other|
|Looking for||Patient||Indirect Patient Care-Other|
|Looking for||Radiograph||Indirect Patient Care-Other|
|Looking for||Supplies||Indirect Patient Care-Other|
|Oops||Can't Find Entry||Miscellaneous|
|Phone||Getting Results||Indirect Patient Care-Read|
|Phone||Paging||Indirect Patient Care-Other|
|Phone||Patient||Direct Patient Care|
|Phone||Scheduling Tests||Indirect Patient Care-Other|
|Procedure||EKG||Direct Patient Care|
|Procedure||IV||Direct Patient Care|
|Procedure||Joint Inj/Asp||Direct Patient Care|
|Procedure||Lab Test||Direct Patient Care|
|Procedure||Other||Direct Patient Care|
|Procedure||Pelvic Exam||Direct Patient Care|
|Procedure||Phlebotomy||Direct Patient Care|
|Talking||Advance Directives||Direct Patient Care|
|Talking||Colleague/Staff||Direct Patient Care|
|Talking||Consultant||Direct Patient Care|
|Talking||Educating Patient||Direct Patient Care|
|Talking||Patient History||Direct Patient Care|
|Talking||Patient Other||Direct Patient Care|
|Talking||Patient's Family||Direct Patient Care|
|Writing||Letter||Indirect Patient Care-Write|
|Writing||Notes||Indirect Patient Care-Write|
|Writing||Orders||Indirect Patient Care-Write|
|Writing||Script||Indirect Patient Care-Write|