AsthmaCritic: Issues in Designing a Noninquisitive Critiquing System for Daily Practice
- Manon M Kuilboer, MD, MSc,
- Marc A M van Wijk, MD, PhD,
- Mees Mosseveld, MSc,
- Johan van der Lei, MD, PhD
- Affiliation of the authors: Department of Medical Informatics, Erasmus MC, University Medical Centre Rotterdam, The Netherlands
- Correspondence and reprints: Marc A. M. van Wijk, MD, PhD, Department of Medical Informatics, Erasmus MC, University Medical Centre Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands; e-mail: < >.
- Received 15 October 2002
- Accepted 15 May 2003
The noninquisitive critiquing system, AsthmaCritic, uses routinely recorded electronic patient data to select and analyze records of patients with asthma or chronic obstructive pulmonary disease (COPD). The system generates critiquing comments and adds these comments to the patient record. The system was developed by using and expanding an existing generic critiquing system. After a brief overview of the system, this report focuses on the authors' design choices in light of existing literature and the issues that underlie these design choices. Currently, AsthmaCritic is used by primary care physicians and is undergoing clinical evaluation.
Asthma and chronic obstructive pulmonary disease (COPD) are chronic diseases with a high prevalence and account for significant health care expenditures.1 Professional health care organizations disseminate paper-based guidelines, reflecting the “state of the art” in medical care.2 3 However, paper-based guidelines have had disappointingly little impact on physicians' behavior.4 5 To change daily practice, dissemination of guidelines needs to be combined with an appropriate implementation strategy.6 One possible implementation strategy is to implement guidelines in computer-based decision-support systems (CDSSs)7 8 that help the practitioner to manage patients with a particular disease by using the appropriate guidelines and protocols. Although researchers have shown that computed decision support is able to change the delivery of health care, the number of systems in daily use is limited.7 9 10 11 12 13 14 Some investigators argue that the use of electronic patient records will provide new opportunities for decision support11 15 16 by providing a natural way to integrate decision support in everyday practice.7 14 17
We designed and built a critiquing system, AsthmaCritic, for use in the domains of asthma and COPD, taking “integration into daily practice” as a precondition. In this report, we first describe the design and implementation of AsthmaCritic. We subsequently justify our design choices in light of the available literature.
In The Netherlands, most general practitioners have replaced their paper medical records with electronic patient records.18 The general practitioner records data during consultation. Based on that data, AsthmaCritic provides feedback by generating a critique of the physicians' diagnosis and treatment plan for patients with asthma or COPD. AsthmaCritic is a noninquisitive critiquing system that solely relies on the data it receives from the electronic patient record and that does not ask for additional data entry. AsthmaCritic runs autonomously; that is, the system is triggered when the physician sees an asthma or COPD patient,* starts analyzing the record when the consultation is finished, and then presents the critique. From the physician's viewpoint, AsthmaCritic is part of his or her medical-record system; when AsthmaCritic reports its critique, the interface is identical to the interface of the medical-record system (that is, screen- and data-manipulation is handled in the same fashion).
AsthmaCritic distinguishes between two different kinds of feedback: critique and transformed clinical measurement. A critique is presented to the physician as a patient-specific comment based on the current clinical situation. AsthmaCritic first presents a brief overview of all comments, ranked by clinical urgency and potential impact. Figure 1 gives an example of a patient with COPD and a decreasing forced expiratory volume in 1 second (FEV1) measurement. AsthmaCritic recognizes the decrease (in Fig. 1: “Recent FEV1 guideline for COPD patients”) and, because the patient is not yet receiving corticosteroids, suggests the possibility of treatment with inhaled corticosteroids (in Fig. 2: “Option: inh. Corticos”). In Figure 1, the physician selects “Option: inh. Corticos,” and AsthmaCritic responds with a brief explanation. When the physician requests additional information (in Fig. 1, the option “<A> Add. Info”), the system provides, based on the Dutch guidelines, the reasoning underlying this recommendation (Fig. 2).
Transformed clinical measurements constitute AsthmaCritic's second kind of comment; calculations on measurements are performed and presented to the user. Peak-flow values, for example, are presented to the physician in an overview that includes the original values, the expected value for this patient based on gender, age, and height, and the original values expressed as a percentage of the expected value.
The general practitioner controls the behavior of AsthmaCritic. The physician decides when to review the feedback: during the patient encounter, at some later time, or at the patient's next visit. When AsthmaCritic is running, it may, depending on the locally available hardware, slow down the electronic patient-record system. Therefore, the physician can interrupt the current analysis; interrupting the analysis initiates a background activity that continues processing the record. If the analysis of a record is done in background mode, a patient-specific message presenting the system's critique is attached to the medical record that is displayed the next time the physician opens the record. The general practitioner's electronic patient-record system keeps a log of the records that still have to be analyzed by AsthmaCritic and a log of critiques that have not yet been shown to the physician.
A description of AsthmaCritic's functionality and knowledge base is available to the physicians on paper. For quick reference, a small laminated yellow reference card shows on one side an overview of structured patient data relevant for asthma and COPD and, on the other side, the system's main control functions and helpdesk phone numbers.
AsthmaCritic's implementation is based on the generic critiquing model of HyperCritic.21 This generic critiquing model supports the integration of a critiquing system with an electronic patient record. HyperCritic's ability to process time-stamped data was limited. In addition, HyperCritic presented its output as text, often several pages long, with no opportunity for the physician to control the behavior of the system. Therefore, HyperCritic lacked structures that enabled integration of the system in the physician's working environment and was never tested in daily practice.
AsthmaCritic's implementation differs from HyperCritic's by (1) increasing the use of time-stamped data, (2) allowing the physician to control the output, and (3) supporting functions that allow a better integration with the electronic patient record (for example, running the system in background mode, attaching messages to a patient's records, adding the results of the analysis to the medical record, or monitoring what feedback still has to be shown to the physician).
The generic critiquing model distinguishes between critiquing knowledge (knowledge that initiates and guides the critiquing process) and medical knowledge (the medical base for critiquing). The analysis of patient data and medical knowledge is controlled by critiquing knowledge. Critiquing knowledge is described as a hierarchical set of critiquing tasks. Each individual critiquing task describes a specific procedure that defines for which clinical situations a critiquing statement should be generated.
Critiquing tasks are triggered by event descriptions that identify the criteria that have to be met by the patient data for the task to be executed. Executing the critiquing task may result in a critiquing statement. Figure 3, for example, shows a critiquing task that is executed whenever a drug is prescribed. This task searches for interactions between drugs. If such an interaction is found, the critiquing statement “possible interaction” is generated.21
AsthmaCritic's critiquing knowledge is divided into four categories of critiquing tasks: diagnostic tasks, therapy-related tasks, referral-related tasks, and follow-up–related tasks. Each category is further subdivided into more specific tasks. For example, a therapy-related task contains nine different subtasks (such as checking for contraindications, checking for drug interactions, verifying dosing schedules). In total, AsthmaCritic contains 131 specific tasks.
Critiquing tasks, however, are solely procedural specifications. Based on the existing medical knowledge, the total number of different clinical situations that can be distinguished is much higher. For example, screening for contraindications is a single task, but the number of clinical situations that will be detected depends on the number of drugs and contraindications known in present-day medicine. As such, critiquing tasks only specify the process of a critique, not the content. For the critiquing statement to be generated, medical knowledge needs to be available (for example, dosing schedules, side effects, interactions).
AsthmaCritic's medical knowledge base has been built from national guidelines,22 23 24 pharmaceutical reference books,25 guides on interactions and side effects,26 and the existing consensus among a group of specialists in asthma and COPD. Building the knowledge base was a three-year iterative process under the guidance of a medical content board consisting of four local specialists (two general practitioners and two pulmonologists†) and seven national specialists.‡ Members of the medical-content board reviewed each new version of the knowledge base.
In building a decision-support system, designers have to make numerous decisions. The available literature includes little data to guide those decisions. System builders have to select, without prior verification of the consequences or feasibility, between the available options. In the case of AsthmaCritic, only one design decision was tested prior to building the system. That decision involved whether the system could be made noninquisitive or, phrased differently, whether the electronic patient record of the general practitioner contained sufficient medical data to enable the generation of critique without asking additional information. In a simulation study, we showed that the medical record did contain enough information,35 based on that simulation study, we decided to build a noninquisitive system. When building the system, numerous other design decisions were made based on personal judgment and intuition. Throughout the design process, we were guided by three principles: (1) strive for a consistent information-technology environment, (2) respect the user as a professional, and (3) respect the limitations of a busy practice.
Physicians use a number of systems in their work—the decision-support system is only one of them. Researchers have argued that a computerized decision-support system should be integrated with an electronic patient record system to avoid reentering medical data34 35 and reduce the number of workflow interruptions.11 27 28 29 30 Others have emphasized that data entry itself should be minimal, that forced data entry should be avoided,31 32 33 and that the decision-support system should not be solely dependent on the physician's initiative to activate the system.9 33 From the physician's viewpoint, the different systems should, ideally, be integrated and present the user with a consistent environment.
In designing AsthmaCritic, we made sure that both the electronic patient records and the decision-support system could be used from the same physical location (in Table 1, “Location Singularity”) without the user's having to switch applications (“Application Consistency”). In addition, we decided that the interfaces to the electronic record system and the decision-support system should be, as far as possible, identical (in Table 1, “Interface Consistency”). AsthmaCritic is a noninquisitive system that relies only on routinely recorded data and does not ask the user for additional data (in Table 1, “Data-Entry Effort”). Finally, AsthmaCritic continuously monitors the electronic patient record for data that may warrant detailed analysis of the record (in Table 1, “Active Case Finding”).
The User as Professional
As early as 1989, Shortliffe36 described “loss of control” as one of the barriers to the introduction of decision-support systems. Perhaps this sense of loss of control was partly due to the fear physicians had about being replaced by “expert systems,” which, in turn, leads to a rejection of the decision-support concept altogether.36 37 Taylor38 described another form of loss of control that involves the psychology of being denied the reward of using one's own skills. Treating the user as a professional also means that the decision-support system is able to explain its recommendations. Previous experience has shown that users' trust in generated advice increases if the system is able to justify its recommendations.43 44
We designed AsthmaCritic as a critiquing system that provides a physician with feedback based on the physician's treatment plan as recorded in the patient's electronic patient record. He or she makes the decision. The system thus preserves the physician's professional autonomy by leaving him or her in control of the decision-making process (in Table 1, “Professional Autonomy”). In addition, AsthmaCritic's feedback explains why it was generated and gives the underlying reasoning, including references to the source material (e.g., references to the appropriate sections of the guidelines) or the members of the medical content board responsible for reviewing that section of the knowledge base (in Table 1, “Information Justification”).
Limitations of a Busy Practice
In daily practice, physicians often do not have the time to read all available information and to select the specific information that they need.41 Medical practice is characterized by (relative) information overload, and the available time and information requirements vary for each person.39 40 If physicians are expected to read and process information when time is limited, the amount of information has to be limited as well.42 Lobach and Hammond,9 for example, reports that physicians prefer a telegraphic style when guideline information is presented. Timing and freedom are important issues; physicians should be able to read or interrupt reading the information provided at any moment.45 Forcing physicians into a specific structure may lead to rejection.31
In designing AsthmaCritic, we gave the user control over timing. That is, the user can at any time interrupt AsthmaCritic. However, AsthmaCritic records that interruption and allows the user to continue at a later moment (Table 1, “Timing”). The feedback is presented initially in a short telegraphic style (in Table 1, “Information Conciseness”), and the user controls what additional information she or he wants to read at any given moment (in Table 1, “Personal Autonomy”).
AsthmaCritic reuses a previously published generic critiquing model that supports integration with an electronic patient record.21 However, the prototype of this generic model, HyperCritic, was never tested in daily practice. AsthmaCritic develops the critiquing model further by supporting additional functions to accommodate requirements for use in daily practice.
Of all the design decisions made in the development of AsthmaCritic, only one single decision was verified experimentally prior to building the system: the decision to build a noninquisitive system. We wanted to be able to integrate AsthmaCritic with the general practitioner's information system; AsthmaCritic should not interrupt the user to request specific or additional data. However, the availability of medical data depends on the ability and willingness of the physician to record data in a structured fashion.46 These recording habits vary a great deal. The question was whether, given this variability, the choice for noninquisitiveness was feasible. Therefore, in a previous study, we investigated the feasibility of using data routinely recorded in electronic patient records for the generation of patient-specific feedback.35 We concluded that enough structured data were available to generate relevant feedback for general practitioners. In the study, we also investigated the need to ask physicians for missing data.47 Our study showed that information that was missed by reviewers often was available elsewhere but did not make much difference in the generation of comments by AsthmaCritic.47
The critiquing process can be divided into two stages: first, the generation of critique followed by using that critique. With design issues playing a role, we minimized user dependency during the generation of critique, and we maximized user dependency during the phase of using the critique. That is, during the phase of generating output, control by the physician is undesirable, but during the phase of using the output, control by the physician is mandatory.48 Table 1 summarizes the issues and our choices for each phase.
The decisions all have their pros and cons. For example, the decision to tightly integrate the decision-support system with a medical-record system requires tailoring to that system. Using AsthmaCritic together with another electronic record system requires an effort that includes developing a new interface consistent with that record system. Ideally, a system developer knows how each choice influences system operation. If he or she knows these relationships, design choices can be optimized, considering the constraints and options of the system's intended working environment. That knowledge, however, often is not available. We believe that further research into the relationship between CDSSs and their operating environments is needed.
Currently, AsthmaCritic is the subject of a randomized clinical trial in which we measure the impact of the system in actual use. Results of that study will show whether the system has an impact on physician performance. It will not measure the impact, if any, of specific design decisions. We believe that trials comparing alternative methods of building comparable system functionality are required to begin to understand the consequences of each individual design decision.
Supported by The Netherlands Asthma Foundation (#92.62). The authors thank their local medical content board: B. Ponsioen, MD, Prof. E. van der Does, MD, PhD, Prof. J. C. de Jongste, MD, PhD, and S. E. Overbeek, MD, PhD, and their national medical content board: B. Bottema, MD, PhD, P. N. R. Dekhuyzen, MD, PhD, E. J. Duiverman, MD, PhD, E. E. M. van Essen, MD, PhD, Th. B. Voorn, MD, PhD, M. H. J. Vaessen, MD, and A. van der Kuy, MD, PhD, for their part in the development of AsthmaCritic knowledge base. The authors also thank M. Kroneman, MSc, PhD, for her valuable contributions to this report.
↵* AsthmaCritic is triggered by the ICPC codes (International Classification of Primary Care19) for asthma (R96), chronic bronchitis (R91), emphysema (R95), other chronic pulmonary diseases (R83.4), and the ATC codes (Anatomical, Therapeutic and Chemical coding system of the World Health Organization20) for prescriptions used in the treatment of asthma or COPD (R03).
↵† Dr. B. Ponsioen, Professor Dr. E. van der Does, Dr. J. C. de Jongste, Dr. S. E. Overbeek.
↵‡ Dr. B. Bottema, Dr. P. N. R. Dekhuyzen, Dr. E. J. Duiverman, Dr. E. E. M. van Essen, Dr. Th. B. Voorn, Dr. M. H. J. Vaessen, Dr. A. van der Kuy.