To understand the complex effects of interruption in healthcare.
As interruptions have been well studied in other domains, the authors undertook a systematic review of experimental studies in psychology and human–computer interaction to identify the task types and variables influencing interruption effects.
63 studies were identified from 812 articles retrieved by systematic searches. On the basis of interruption profiles for generic tasks, it was found that clinical tasks can be distinguished into three broad types: procedural, problem-solving, and decision-making. Twelve experimental variables that influence interruption effects were identified. Of these, six are the most important, based on the number of studies and because of their centrality to interruption effects, including working memory load, interruption position, similarity, modality, handling strategies, and practice effect. The variables are explained by three main theoretical frameworks: the activation-based goal memory model, prospective memory, and multiple resource theory.
This review provides a useful starting point for a more comprehensive examination of interruptions potentially leading to an improved understanding about the impact of this phenomenon on patient safety and task efficiency. The authors provide some recommendations to counter interruption effects.
The effects of interruption are the outcome of a complex set of variables and should not be considered as uniformly predictable or bad. The task types, variables, and theories should help us better to identify which clinical tasks and contexts are most susceptible and assist in the design of information systems and processes that are resilient to interruption.
The Consolidated Standards for Reporting Trials (CONSORT) were published to standardize reporting and improve the quality of clinical trials. The objective of this study is to assess CONSORT adherence in randomized clinical trials (RCT) of disease specific clinical decision support (CDS).
A systematic search was conducted of the Medline, EMBASE, and Cochrane databases. RCTs on CDS were assessed against CONSORT guidelines and the Jadad score.
32 of 3784 papers identified in the primary search were included in the final review. 181 702 patients and 7315 physicians participated in the selected trials. Most trials were performed in primary care (22), including 897 general practitioner offices. RCTs assessing CDS for asthma (4), diabetes (4), and hyperlipidemia (3) were the most common. Thirteen CDS systems (40%) were implemented in electronic medical records, and 14 (43%) provided automatic alerts. CONSORT and Jadad scores were generally low; the mean CONSORT score was 30.75 (95% CI 27.0 to 34.5), median score 32, range 21–38. Fourteen trials (43%) did not clearly define the study objective, and 11 studies (34%) did not include a sample size calculation. Outcome measures were adequately identified and defined in 23 (71%) trials; adverse events or side effects were not reported in 20 trials (62%). Thirteen trials (40%) were of superior quality according to the Jadad score (≥3 points). Six trials (18%) reported on long-term implementation of CDS.
The overall quality of reporting RCTs was low. There is a need to develop standards for reporting RCTs in medical informatics.
The US Agency for Healthcare Research and Quality funded an evidence report to address seven questions on multiple aspects of the effectiveness of medication management information technology (MMIT) and its components (prescribing, order communication, dispensing, administering, and monitoring).
Medline and 11 other databases without language or date limitations to mid-2010. Randomized controlled trials (RCTs) assessing integrated MMIT were selected by two independent reviewers. Reviewers assessed study quality and extracted data. Senior staff checked accuracy.
Most of the 87 RCTs focused on clinical decision support and computerized provider order entry systems, were performed in hospitals and clinics, included primarily physicians and sometimes nurses but not other health professionals, and studied process changes related to prescribing and monitoring medication. Processes of care improved for prescribing and monitoring mostly in hospital settings, but the few studies measuring clinical outcomes showed small or no improvements. Studies were performed most frequently in the USA (n=63), Europe (n=16), and Canada (n=6).
Many studies had limited description of systems, installations, institutions, and targets of the intervention. Problems with methods and analyses were also found. Few studies addressed order communication, dispensing, or administering, non-physician prescribers or pharmacists and their MMIT tools, or patients and caregivers. Other study methods are also needed to completely understand the effects of MMIT.
Almost half of MMIT interventions improved the process of care, but few studies measured clinical outcomes. This large body of literature, although instructive, is not uniformly distributed across settings, people, medication phases, or outcomes.
Adverse drug events (ADEs), defined as adverse patient outcomes caused by medications, are common and difficult to detect. Electronic detection of ADEs is a promising method to identify ADEs. We performed this systematic review to characterize established electronic detection systems and their accuracy.
We identified studies evaluating electronic ADE detection from the MEDLINE and EMBASE databases. We included studies if they contained original data and involved detection of electronic triggers using information systems. We abstracted data regarding rule characteristics including type, accuracy, and rationale.
Forty-eight studies met our inclusion criteria. Twenty-four (50%) studies reported rule accuracy but only 9 (18.8%) utilized a proper gold standard (chart review in all patients). Rule accuracy was variable and often poor (range of sensitivity: 40%–94%; specificity: 1.4%–89.8%; positive predictive value: 0.9%–64%). 5 (10.4%) studies derived or used detection rules that were defined by clinical need or the underlying ADE prevalence. Detection rules in 8 (16.7%) studies detected specific types of ADEs.
Several factors led to inaccurate ADE detection algorithms, including immature underlying information systems, non-standard event definitions, and variable methods for detection rule validation. Few ADE detection algorithms considered clinical priorities. To enhance the utility of electronic detection systems, there is a need to systematically address these factors.
Studies on the impact and value of health information technology (HIT) have often focused on outcome measures that are counts of such things as hospital admissions or the number of laboratory tests per patient. These measures with their highly skewed distributions (high frequency of 0s and 1s) are more appropriately analyzed with count data models than the much more frequently used variations of ordinary least squares (OLS). Use of a statistical procedure that does not properly fit the distribution of the data can result in significant findings being overlooked. The objective of this paper is to encourage greater use of count data models by demonstrating their utility with an example based on the authors' current work.
Researchers conducting impact and outcome studies related to HIT.
We review and discuss count data models and illustrate their value in comparison to OLS using an example from a study of the impact of an electronic health record (EHR) on laboratory test orders. The best count data model reveals significant relationships that OLS does not detect. We conclude that comprehensive model checking is highly recommended to identify the most appropriate analytic model when the dependent variable being examined contains count data. This strategy can lead to more valid and precise findings in HIT evaluation studies.
To expand an emerging classification for problems with health information technology (HIT) using reports submitted to the US Food and Drug Administration Manufacturer and User Facility Device Experience (MAUDE) database.
HIT events submitted to MAUDE were retrieved using a standardized search strategy. Using an emerging classification with 32 categories of HIT problems, a subset of relevant events were iteratively analyzed to identify new categories. Two coders then independently classified the remaining events into one or more categories. Free-text descriptions were analyzed to identify the consequences of events.
Descriptive statistics by number of reported problems per category and by consequence; inter-rater reliability analysis using the statistic for the major categories and consequences.
A search of 899 768 reports from January 2008 to July 2010 yielded 1100 reports about HIT. After removing duplicate and unrelated reports, 678 reports describing 436 events remained. The authors identified four new categories to describe problems with software functionality, system configuration, interface with devices, and network configuration; the authors' classification with 32 categories of HIT problems was expanded by the addition of these four categories. Examination of the 436 events revealed 712 problems, 96% were machine-related, and 4% were problems at the human–computer interface. Almost half (46%) of the events related to hazardous circumstances. Of the 46 events (11%) associated with patient harm, four deaths were linked to HIT problems (0.9% of 436 events).
Only 0.1% of the MAUDE reports searched were related to HIT. Nevertheless, Food and Drug Administration reports did prove to be a useful new source of information about the nature of software problems and their safety implications with potential to inform strategies for safe design and implementation.
Systematic analysis of observational medical databases for active safety surveillance is hindered by the variation in data models and coding systems. Data analysts often find robust clinical data models difficult to understand and ill suited to support their analytic approaches. Further, some models do not facilitate the computations required for systematic analysis across many interventions and outcomes for large datasets. Translating the data from these idiosyncratic data models to a common data model (CDM) could facilitate both the analysts' understanding and the suitability for large-scale systematic analysis. In addition to facilitating analysis, a suitable CDM has to faithfully represent the source observational database. Before beginning to use the Observational Medical Outcomes Partnership (OMOP) CDM and a related dictionary of standardized terminologies for a study of large-scale systematic active safety surveillance, the authors validated the model's suitability for this use by example.
To validate the OMOP CDM, the model was instantiated into a relational database, data from 10 different observational healthcare databases were loaded into separate instances, a comprehensive array of analytic methods that operate on the data model was created, and these methods were executed against the databases to measure performance.
There was acceptable representation of the data from 10 observational databases in the OMOP CDM using the standardized terminologies selected, and a range of analytic methods was developed and executed with sufficient performance to be useful for active safety surveillance.
To compare the clinical relevance of medication alerts in a basic and in an advanced clinical decision support system (CDSS).
A prospective observational study.
We collected 4023 medication orders in a hospital for independent evaluation in two pharmacotherapy-related decision support systems. Only the more advanced system considered patient characteristics and laboratory test results in its algorithms. Two pharmacists assessed the clinical relevance of the medication alerts produced. The alert was considered relevant if the pharmacist would undertake action (eg, contact the physician or the nurse). The primary analysis concerned the positive predictive value (PPV) for clinically relevant medication alerts in both systems.
The PPV was significantly higher in the advanced system (5.8% vs 17.0%; p<0.05). Significant differences were found in the alert categories: drug–(drug) interaction (9.9% vs 14.8%; p<0.05), drug–age interaction (2.9% vs 73.3%; p<0.05), and dosing guidance (5.6% vs 16.9%; p<0.05). Including laboratory values and other patient characteristics resulted in a significantly higher PPV for the advanced CDSS compared to the basic medication alerts (12.2% vs 23.3%; p<0.05).
The advanced CDSS produced a higher proportion of clinically relevant medication alerts, but the number of irrelevant alerts remained high. To improve the PPV of the advanced CDSS, the algorithms should be optimized by identifying additional risk modifiers and more data should be made electronically available to improve the performance of the algorithms. Our study illustrates and corroborates the need for cyclic testing of technical improvements in information technology in circumstances representative of daily clinical practice.
To identify the frequency of medication administration errors and their potential risk factors in units using a computerized prescription order entry program and profiled automated dispensing cabinets.
Prospective observational study conducted within two clinical units of the Gastroenterology Department in a 1537-bed tertiary teaching hospital in Madrid (Spain).
Medication errors were measured using the disguised observation technique. Types of medication errors and their potential severity were described. The correlation between potential risk factors and medication errors was studied to identify potential causes.
In total, 2314 medication administrations to 73 patients were observed: 509 errors were recorded (22.0%)—68 (13.4%) in preparation and 441 (86.6%) in administration. The most frequent errors were use of wrong administration techniques (especially concerning food intake (13.9%)), wrong reconstitution/dilution (1.7%), omission (1.4%), and wrong infusion speed (1.2%). Errors were classified as no damage (95.7%), no damage but monitoring required (2.3%), and temporary damage (0.4%). Potential clinical severity could not be assessed in 1.6% of cases. The potential risk factors morning shift, evening shift, Anatomical Therapeutic Chemical medication class antacids, prokinetics, antibiotics and immunosuppressants, oral administration, and intravenous administration were associated with a higher risk of administration errors. No association was found with variables related to understaffing or nurse's experience.
Medication administration errors persist in units with automated prescription and dispensing. We identified a need to improve nurses' working procedures and to implement a Clinical Decision Support tool that generates recommendations about scheduling according to dietary restrictions, preparation of medication before parenteral administration, and adequate infusion rates.
Adverse drug events (ADEs) are common and account for 770 000 injuries and deaths each year and drug interactions account for as much as 30% of these ADEs. Spontaneous reporting systems routinely collect ADEs from patients on complex combinations of medications and provide an opportunity to discover unexpected drug interactions. Unfortunately, current algorithms for such "signal detection" are limited by underreporting of interactions that are not expected. We present a novel method to identify latent drug interaction signals in the case of underreporting.
We identified eight clinically significant adverse events. We used the FDA's Adverse Event Reporting System to build profiles for these adverse events based on the side effects of drugs known to produce them. We then looked for pairs of drugs that match these single-drug profiles in order to predict potential interactions. We evaluated these interactions in two independent data sets and also through a retrospective analysis of the Stanford Hospital electronic medical records.
We identified 171 novel drug interactions (for eight adverse event categories) that are significantly enriched for known drug interactions (p=0.0009) and used the electronic medical record for independently testing drug interaction hypotheses using multivariate statistical models with covariates.
Our method provides an option for detecting hidden interactions in spontaneous reporting systems by using side effect profiles to infer the presence of unreported adverse events.
To evaluate the impact of a real-time computerized decision support tool in the emergency department that guides medication dosing for the elderly on physician ordering behavior and on adverse drug events (ADEs).
A prospective controlled trial was conducted over 26 weeks. The status of the decision support tool alternated OFF (7/17/06–8/29/06), ON (8/29/06–10/10/06), OFF (10/10/06–11/28/06), and ON (11/28/06–1/16/07) in consecutive blocks during the study period. In patients ≥65 who were ordered certain benzodiazepines, opiates, non-steroidals, or sedative-hypnotics, the computer application either adjusted the dosing or suggested a different medication. Physicians could accept or reject recommendations.
The primary outcome compared medication ordering consistent with recommendations during ON versus OFF periods. Secondary outcomes included the admission rate, emergency department length of stay for discharged patients, 10-fold dosing orders, use of a second drug to reverse the original medication, and rate of ADEs using previously validated explicit chart review.
2398 orders were placed for 1407 patients over 1548 visits. The majority (49/53; 92.5%) of recommendations for alternate medications were declined. More orders were consistent with dosing recommendations during ON (403/1283; 31.4%) than OFF (256/1115; 23%) periods (p≤0.0001). 673 (43%) visits were reviewed for ADEs. The rate of ADEs was lower during ON (8/237; 3.4%) compared with OFF (31/436; 7.1%) periods (p=0.02). The remaining secondary outcomes showed no difference.
Single institution study, retrospective chart review for ADEs.
Though overall agreement with recommendations was low, real-time computerized decision support resulted in greater acceptance of medication recommendations. Fewer ADEs were observed when computerized decision support was active.
To demonstrate the feasibility of capturing the knowledge required to create guideline recommendations in a systematic, structured, manner using a software assistant. Practice guidelines constitute an important modality that can reduce the delivery of inappropriate care and support the introduction of new knowledge into clinical practice. However, many guideline recommendations are vague and underspecified, lack any linkage to supporting evidence or documentation of how they were developed, and prove to be difficult to transform into systems that influence the behavior of care providers.
The BRIDGE-Wiz application (Building Recommendations In a Developer's Guideline Editor) uses a wizard approach to address the questions: (1) under what circumstances? (2) who? (3) ought (with what level of obligation?) (4) to do what? (5) to whom? (6) how and why? Controlled natural language was applied to create and populate a template for recommendation statements.
The application was used by five national panels to develop guidelines. In general, panelists agreed that the software helped to formalize a process for authoring guideline recommendations and deemed the application usable and useful.
Use of BRIDGE-Wiz promotes clarity of recommendations by limiting verb choices, building active voice recommendations, incorporating decidability and executability checks, and limiting Boolean connectors. It enhances transparency by incorporating systematic appraisal of evidence quality, benefits, and harms. BRIDGE-Wiz promotes implementability by providing a pseudocode rule, suggesting deontic modals, and limiting the use of ‘consider’.
Users found that BRIDGE-Wiz facilitates the development of clear, transparent, and implementable guideline recommendations.
To develop a population-based proband-oriented pedigree information system that can be easily applied to various diseases in genetic epidemiological studies, making allowance for the capture of theoretical family relationships.
A population-based proband-oriented pedigree information system with ties of consanguinity based on both population-based household registry data and Keelung Community Integrated Screening data was proposed to build a comprehensive extended family pedigree structure to accommodate a series of genetic studies on different diseases. We also developed an algorithm to efficiently assess how well theoretical family relationships affecting the occurrence of diseases across three generations with respect to the relative relationship score, a quantitative indicator of genetic influence, were captured.
We applied this population-based proband-oriented pedigree information system to estimate the rate of hypertension with various relative relationships given the selection of probands. The degree of capturing complete familial relationships was assessed for three generations. The risk for early onset of hypertension was proportional to the proband-oriented relative relationship score with 2% increased risk and 1% correction for incomplete capture.
The population-based proband-oriented pedigree information system is powerful and can support various genetic descriptive and analytic epidemiological studies.
To evaluate the use of a secure internet portal in an academic Multiple Sclerosis (MS) Center.
Retrospective case–control chart review of 240 patients during the years 2008 and 2009. Patient demographic and clinical information was extracted from our online medical records, and portal use metrics were provided by Information Systems. Descriptive statistics were utilized to explore characteristics of portal users, how the portal is used, and what associations exist between medical resource utilization and active portal use. Logistic regression identified independent patient predictors and barriers to portal use.
Portal users tended to be young professionals with minimal physical disability. The most frequently used portal feature was secure patient–physician messaging. Message content largely consisted of requests for medications or refills in addition to self-reported side effects. Independent predictors and barriers of portal use include the number of medications prescribed by our staff (OR 1.69, p<0.0001), Caucasian ethnicity (OR 5.04, p=0.007), arm and hand disability (OR 0.23, p=0.01), and impaired vision (OR 0.31, p=0.01).
MS patients use the internet in a greater proportion than the general US population, yet physical disability limits their access. Technological adaptations such as voice-activated commands and easy font-size adjustment may help patients overcome these barriers.
Future research should explore the influence of portal technology on healthcare resource utilization and cost. Additional emedicine applications could be linked to the patient portal for disease monitoring and prospective investigation.
The goal of this paper is to describe the successful application of a use case-based evaluation approach to guide the effective design, evaluation and redesign of inexpensive, commercial, off-the-shelf consumer health informatics (CHI) interventions.
Researchers developed four CHI intervention use cases representing two distinct patient populations (patients with diabetes with high blood pressure, post-bariatric surgery patients), two commercial off-the-shelf CHI applications (Microsoft HealthVault, Google Health), and related devices (blood pressure monitor, pedometer, weight scale). Three patient proxies tested each intervention for 10 days.
The patient proxies recorded their challenges while completing use case tasks, rating the severity of each challenge based on how much it hindered their use of the intervention. Two independent evaluators categorized the challenges by human factors domain (physical, cognitive, macroergonomic).
The use case-based approach resulted in the identification of 122 challenges, with 12% physical, 50% cognitive and 38% macroergonomic. Thirty-nine challenges (32%) were at least moderately severe. Nine of 22 use case tasks (41%) accounted for 72% of the challenges.
The study used two patient proxies and addressed two specific patient populations and low-cost, off-the-shelf CHI interventions, which may not perfectly generalize to a larger number of proxies, actual patient populations, or other CHI interventions.
CHI designers can employ the use case-based evaluation approach to assess the fit of a CHI intervention with patients' health work, in the context of their daily activities and environment, which would be difficult or impossible to evaluate by laboratory-based studies.
Physicians in small practices are concerned about financial barriers.
Physicians in small practices are concerned about future obsolescence.
JAMIA thanks its reviewers for ensuring the quality of information we publish. We have significantly reduced the time for review to allow more expeditious processing of manuscripts. Our reviewers have helped us to get the median processing time until the first decision below 30 days and provided insightful, constructive feedback to the authors. We recognize the effort involved in the reviewing process and hope to continue to work with these experts in the future.
Jacob Aaronson Jos Aarts Patricia Abbott Neil Abernethy Michael Ackerman Julia Adler-Milstein Ritu Agarwal David Ahern Iulian Alecu Russ Altman Ruben Amarasingham Haward Amital Elske Ammenwerth Shilo Anders James Anderson Stephen Anthony Eliah Aronoff-Spencer Dominik Aronsky Alan Aronson Joan Ash Cheryl Austein Casnoff Megan...]]>
Before starting the core of this document, I have a couple of notes. First, I would like to acknowledge the terrific work of my predecessor, outgoing AMIA Board Chair Nancy Lorenzi. During her term, Nancy made AMIA a much more effective and efficient organization. Nancy led a refinement of AMIA's strategic plan,