Retrospective electronic chart reviews of 4081 patient records across 57 practices were analyzed to determine the validity of EHR-derived quality measures and documented preventive services.

Results from this study show that workflow and documentation habits have a profound impact on EHR-derived quality measures. Compared with the manual review of electronic charts, EHR-derived measures can undercount practice performance, with a disproportionately negative impact on the number of patients captured as receiving a clinical preventive service or meeting a recommended treatment goal.

This study provides a cautionary note in using EHR-derived measurement for public reporting of provider performance or use for payment.

We searched Medline, Pre-Medline, CINAHL, Embase, and the SPIE Digital Library databases for English-language publications between 1980 and September 2010 using Medical Subject Headings terms and keywords.

Eleven studies from the USA and UK were included. All studies measured aspects of time associated with the introduction of PACS, namely the availability of images, the time a physician took to review an image, and changes in viewing patterns. Seven studies examined the impact on clinical decision-making, with the majority measuring the time to image-based clinical action. The effect of PACS on communication modes was reported in five studies.

PACS can impact on clinician work practices in three main areas. Most of the evidence suggests an improvement in the efficiency of work practices. Quick image availability can impact on work associated with clinical decision-making, although the results were inconsistent. PACS can change communication practices, particularly between the ICU and radiology; however, the evidence base is insufficient to draw firm conclusions in this area.

The potential for PACS to impact positively on clinician work practices in the ICU and improve patient care is great. However, the evidence base is limited and does not reflect aspects of contemporary PACS technology. Performance measures developed in previous studies remain relevant, with much left to investigate to understand how PACS can support new and improved ways of delivering care in the intensive care setting.

Our hypothesis was that most variation in names occurs after the first two letters, and that date of birth is highly reliable, so a single match variable consisting of a hashed string built from the first two letters of the patient's first and last names plus their date of birth would have the desired characteristics. We compared and contrasted the match algorithm characteristics (rate of false positive v. rate of false negative) for our chosen variable against both Social Security Numbers and full names.

In a data set of 19 000 records, a derived match variable consisting of a 2-character prefix from both first and last names combined with date of birth has a 97% sensitivity; by contrast, an anonymized identifier based on the patient's full names and date of birth has a sensitivity of only 87% and SSN has sensitivity 86%.

The approach we describe is most useful in situations where privacy policies preclude the full exchange of the identifiers required by more sophisticated and sensitive linkage algorithms. For data sets of sufficiently high quality this effective approach, while producing a lower rate of matching than more complex algorithms, has the merit of being easy to explain to institutional review boards, adheres to the minimum necessary rule of the HIPAA privacy rule, and is faster and less cumbersome to implement than a full probabilistic linkage.

The Ontology Development and Information Extraction corpus annotated for coreference relations consists of 7214 coreferential markables, forming 5992 pairs and 1304 chains. We trained classifiers with semantic, syntactic, and surface features pruned by feature selection. For the three system components—for the resolution of relative pronouns, personal pronouns, and noun phrases—we experimented with support vector machines with linear and radial basis function (RBF) kernels, decision trees, and perceptrons. Evaluation of algorithms and varied feature sets was performed using standard metrics.

The best performing combination is support vector machines with an RBF kernel and all features (MUC score=0.352, B³=0.690, CEAF=0.486, BLANC=0.596) outperforming a traditional decision tree baseline.

The application showed good performance similar to performance on general English text. The main error source was sentence distances exceeding a window of 10 sentences between markables. A possible solution to this problem is hinted at by the fact that coreferent markables sometimes occurred in predictable (although distant) note sections. Another system limitation is failure to fully utilize synonymy and ontological knowledge. Future work will investigate additional ways to incorporate syntactic features into the coreference problem.

We investigated computational methods for coreference resolution in the clinical narrative. The best methods are released as modules of the open source Clinical Text Analysis and Knowledge Extraction System and Ontology Development and Information Extraction platforms.

Subsets were first extracted using four graph-traversal heuristics and one logic-based technique, and were subsequently filtered with frequency information from MEDLINE. Twenty manually coded discharge summaries from cardiology patients were used as signatures and test sets. The coverage, size, and precision of extracted subsets were measured.

Graph-traversal heuristics provided high coverage (71–96% of terms in the test sets of discharge summaries) at the expense of subset size (17–51% of the size of SNOMED CT). Pre-computed subsets and logic-based techniques extracted small subsets (1%), but coverage was limited (24–55%). Filtering reduced the size of large subsets to 10% while still providing 80% coverage.

Extracting subsets to annotate discharge summaries is challenging when no previous corpus exists. Ontology modularization provides valuable techniques, but the resulting modules grow as signatures spread across subhierarchies, yielding a very low precision.

Graph-traversal strategies and frequency data from an authoritative source can prune large biomedical ontologies and produce useful subsets that still exhibit acceptable coverage. However, a clinical corpus closer to the specific use case is preferred when available.

Electronic query of EHR data identified women aged 30–65 years old who were at low-risk of cervical cancer and therefore eligible for an extended Pap testing interval of 3 years (as per professional society guidelines). Manual chart review assessed query accuracy. The use of low-value Pap tests (ie, those performed sooner than recommended) was measured, and adverse consequences of low-value Pap tests (ie, colposcopies performed as a result of low-value Pap tests) were identified.

Manual chart review confirmed query accuracy. Two-thirds (1120/1705) of low-risk women received a Pap test sooner than recommended, and 21 colposcopies were performed as a result of this low-value Pap testing.

Secondary analysis of EHR data can accurately measure the use of low-value services such as Pap testing performed sooner than recommended in women at low risk of cervical cancer. Similar application of our methodology could facilitate efforts to simultaneously improve quality and decrease costs, maximizing value in the US healthcare system.

To determine whether computerized prescribing decision support with patient-specific risk estimates would increase physician response to psychotropic drug alerts and reduce injury risk in older people.

Cluster randomized controlled trial of 81 family physicians and 5628 of their patients aged 65 and older who were prescribed psychotropic medication.

Intervention physicians received information about patient-specific risk of injury computed at the time of each visit using statistical models of non-modifiable risk factors and psychotropic drug doses. Risk thermometers presented changes in absolute and relative risk with each change in drug treatment. Control physicians received commercial drug alerts.

Injury risk at the end of follow-up based on psychotropic drug doses and non-modifiable risk factors. Electronic health records and provincial insurance administrative data were used to measure outcomes.

Mean patient age was 75.2 years. Baseline risk of injury was 3.94 per 100 patients per year. Intermediate-acting benzodiazepines (56.2%) were the most common psychotropic drug. Intervention physicians reviewed therapy in 83.3% of visits and modified therapy in 24.6%. The intervention reduced the risk of injury by 1.7 injuries per 1000 patients (95% CI 0.2/1000 to 3.2/1000; p=0.02). The effect of the intervention was greater for patients with higher baseline risks of injury (p<0.03).

Patient-specific risk estimates provide an effective method of reducing the risk of injury for high-risk older people.

clinicaltrials.gov Identifier: NCT00818285.

Statistical text classifiers based on Naïve Bayes and Support Vector Machine (SVM) algorithms were trained and tested on clinical incident reports to automatically detect extreme-risk events, defined by incidents that satisfy the criteria of Severity Assessment Code (SAC) level 1. For this purpose, incident reports submitted to the Advanced Incident Management System by public hospitals from one Australian region were used. The classifiers were evaluated on two datasets: (1) a set of reports with diverse incident types (n=120); (2) a set of reports associated with patient misidentification (n=166). Results were assessed using accuracy, precision, recall, F-measure, and area under the curve (AUC) of receiver operating characteristic curves.

The classifiers performed well on both datasets. In the multi-type dataset, SVM with a linear kernel performed best, identifying 85.8% of SAC level 1 incidents (precision=0.88, recall=0.83, F-measure=0.86, AUC=0.92). In the patient misidentification dataset, 96.4% of SAC level 1 incidents were detected when SVM with linear, polynomial or radial-basis function kernel was used (precision=0.99, recall=0.94, F-measure=0.96, AUC=0.98). Naïve Bayes showed reasonable performance, detecting 80.8% of SAC level 1 incidents in the multi-type dataset and 89.8% of SAC level 1 patient misidentification incidents. Overall, higher prediction accuracy was attained on the specialized dataset, compared with the multi-type dataset.

Text classification techniques can be applied effectively to automate the detection of extreme-risk events in clinical incident reports.

A cluster-randomized effectiveness trial with PHR and no PHR groups was conducted in two ambulatory clinics. 453 of 1686 (26.4%) patients approached were included in the analyses. A PHR tethered to the patient's electronic medical record (EMR) was the primary intervention and included security measures, patient control of access, limited transmission of EMR data, blood pressure (BP) tracking, and appointment assistance. BP was the main outcome measure. Patient empowerment was assessed using the Patient Activation Measure and Patient Empowerment Scale. Quality of care was assessed using the Clinician and Group Assessment Score (CAHPS) and the Patient Assessment of Chronic Illness Care. Frequency of use of medical services was self-reported.

No impact of the PHR was observed on BP, patient activation, patient perceived quality, or medical utilization in the intention-to-treat analysis. Sub-analysis of intervention patients self-identified as active PHR users (25.7% of those with available information) showed a 5.25-point reduction in diastolic BP. Younger age, self-reported computer skills, and more positive provider communication ratings were associated with frequency of PHR use.

Few patients provided with a PHR actually used the PHR with any frequency. Thus simply providing a PHR may have limited impact on patient BP, empowerment, satisfaction with care, or use of health services without additional education or clinical intervention designed to increase PHR use.

http://ClinicalTrials.gov Identifier: NCT01317537.

Medline, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, world wide web.

Studies were included if they described an electronic audiovisual tool used as a means to educate patients, care givers, or clinicians about diabetes management and assessed a psychological, behavioral, or clinical outcome.

Study abstraction and evaluation for clinical usefulness, sustainability, and usability were performed by two independent reviewers.

Of 12616 citations and 1541 full-text articles reviewed, 57 studies met inclusion criteria. Forty studies used experimental designs (25 randomized controlled trials, one controlled clinical trial, 14 before–after studies), and 17 used observational designs. Methodological quality and ratings for clinical usefulness and sustainability were variable, and there was a high prevalence of usability errors. Tools showed moderate but inconsistent effects on a variety of psychological and clinical outcomes including HbA1c and weight. Meta-regression of adequately reported studies (12 studies, 2731 participants) demonstrated that, although the interventions studied resulted in positive outcomes, this was not moderated by clinical usefulness nor usability.

This review is limited by the number of accessible tools, exclusion of tools for mobile devices, study quality, and the use of non-validated scales.

Few tools were identified that met our criteria for effectiveness, usefulness, sustainability, and usability. Priority areas include identifying strategies to minimize website attrition and enabling patients and clinicians to make informed decisions about website choice by encouraging reporting of website quality indicators.

To determine whether a clinical alerting system, which uses inference rules to notify providers of undocumented problems, improves problem list documentation.

Inference rules for 17 conditions were constructed and an electronic health record-based intervention was evaluated to improve problem documentation. A cluster randomized trial was conducted of 11 participating clinics affiliated with a large academic medical center, totaling 28 primary care clinical areas, with 14 receiving the intervention and 14 as controls. The intervention was a clinical alert directed to the provider that suggested adding a problem to the electronic problem list based on inference rules. The primary outcome measure was acceptance of the alert. The number of study problems added in each arm as a pre-specified secondary outcome was also assessed. Data were collected during 6-month pre-intervention (11/2009–5/2010) and intervention (5/2010–11/2010) periods.

17 043 alerts were presented, of which 41.1% were accepted. In the intervention arm, providers documented significantly more study problems (adjusted OR=3.4, p<0.001), with an absolute difference of 6277 additional problems. In the intervention group, 70.4% of all study problems were added via the problem list alerts. Significant increases in problem notation were observed for 13 of 17 conditions.

Problem inference alerts significantly increase notation of important patient problems in primary care, which in turn has the potential to facilitate quality improvement.

ClinicalTrials.gov: NCT01105923.

Using data from a 2007 statewide survey of Massachusetts physicians, we conducted multivariate regression analysis to examine relationships between practice characteristics, perceptions of EHR-related errors, perceptions of healthcare quality, and overall physician satisfaction.

30% of physicians agreed that EHRs create new opportunities for error, but only 2% believed their EHR has created more errors than it prevented. With respect to perceptions of quality, there was no significant association between perceptions of EHR-associated errors and perceptions of EHR-associated changes in healthcare quality. Finally, physicians who believed that EHRs created new opportunities for error were less likely be satisfied with their practice situation (adjusted OR 0.49, p=0.001).

Almost one third of physicians perceived that EHRs create new opportunities for error. This perception was associated with lower levels of physician satisfaction.

Survey of a total of 2010 behavioral health providers in a Midwestern state (33% response rate), with questions based on previously reported open-ended beliefs elicitation interviews.

Factor analysis resulted in four groupings: beliefs that HIE would improve care and communication, add cost and time burdens, present access and vulnerability concerns, and impact workflow and control (positively and negatively). A regression model including all four factors parsimoniously predicted attitudes toward HIE. Providers clustered into two groups based on their beliefs: a majority (67%) were positive about the impact of HIE, and the remainder (33%) were negative. There were some professional/demographic differences between the two clusters of providers.

Most behavioral health providers are supportive of HIE; however, their adoption and use of it may continue to lag behind that of medical providers due to perceived cost and time burdens and concerns about access to and vulnerability of information.

Interrupted time series analysis, 2003–2009.

A large US not-for-profit integrated health care organization.

29 hospitals in Northern and Southern California.

An integrated EHR including computerized physician order entry, nursing documentation, risk assessment tools, and documentation tools.

Percentage of patients with completed risk assessments for hospital acquired pressure ulcers (HAPUs) and falls (process measures) and rates of HAPU and falls (outcome measures).

EHR implementation was significantly associated with an increase in documentation rates for HAPU risk (coefficient 2.21, 95% CI 0.67 to 3.75); the increase for fall risk was not statistically significant (0.36; –3.58 to 4.30). EHR implementation was associated with a 13% decrease in HAPU rates (coefficient –0.76, 95% CI –1.37 to –0.16) but no decrease in fall rates (–0.091; –0.29 to 0.11). Irrespective of EHR implementation, HAPU rates decreased significantly over time (–0.16; –0.20 to –0.13), while fall rates did not (0.0052; –0.01 to 0.02). Hospital region was a significant predictor of variation for both HAPU (0.72; 0.30 to 1.14) and fall rates (0.57; 0.41 to 0.72).

The introduction of an integrated EHR was associated with a reduction in the number of HAPUs but not in patient fall rates. Other factors, such as changes over time and hospital region, were also associated with variation in outcomes. The findings suggest that EHR impact on nursing care processes and outcomes is dependent on a number of factors that should be further explored.

The project adapted software from the Informatics for Integrating Biology and the Bedside (i2b2) program to connect three Clinical Translational Research Award sites: University of Washington, Seattle, University of California, Davis, and University of California, San Francisco. The project developed an iterative spiral software development model to support the implementation and coordination of this multisite data resource.

By standardizing technical infrastructures, policies, and semantics, the project enabled federated querying of deidentified clinical datasets stored in separate institutional environments and identified barriers to engaging users for measuring utility.

The authors discuss the iterative development and evaluation phases of the project and highlight the challenges identified and the lessons learned.

The common system architecture and translational processes provide high-level (aggregate) deidentified access to a large patient population (>5 million patients), and represent a novel and extensible resource. Enhancing the network for more focused disease areas will require research-driven partnerships represented across all partner sites.

In 2008, a team of medical residents built a wiki containing institutional knowledge and reference information using Microsoft SharePoint. We tracked visit data for 3 years, and performed an audit of page views and updates in the second year. We evaluated the attitudes of medical residents toward the wiki using a survey.

Users accessed the wiki 23 218, 35 094, and 40 545 times in each of three successive academic years from 2008 to 2011. In the year two audit, 85 users made a total of 1082 updates to 176 pages and of these, 91 were new page creations by 17 users. Forty-eight percent of residents edited a page. All housestaff felt the wiki improved their ability to complete tasks, and 90%, 89%, and 57% reported that the wiki improved their experience, efficiency, and education, respectively, when surveyed in academic year 2009–2010.

A wiki is a useful and popular tool for organizing administrative and educational content for residents. Housestaff felt strongly that the wiki improved their workflow, but a smaller educational impact was observed. Nearly half of the housestaff edited the wiki, suggesting broad buy-in among the residents.

A wiki is a feasible and useful tool for improving information retrieval for house officers.

A non-randomized cross-sectional study was conducted of 48 providers in New York and 30 providers in Massachusetts, all of whom used paper prescriptions, from September 2005 to November 2006. Using standardized methodology, prescriptions and medical records were reviewed to identify errors.

9385 prescriptions were analyzed from 5955 patients. The overall prescribing error rate, excluding illegibility errors, was 36.7 per 100 prescriptions (95% CI 30.7 to 44.0) and did not vary significantly between providers from each state (p=0.39). One or more non-illegibility errors were found in 28% of prescriptions. Rates of illegibility errors were very high (175.0 per 100 prescriptions, 95% CI 169.1 to 181.3). Inappropriate abbreviation and direction errors also occurred frequently (13.4 and 4.2 errors per 100 prescriptions, respectively). Reviewers determined that the vast majority of errors could have been eliminated through the use of e-prescribing with clinical decision support.

Prescribing errors appear to occur at very high rates among community-based primary care providers, especially when compared with studies of academic-affiliated providers that have found nearly threefold lower error rates. Illegibility errors are particularly problematical.

Further characterizing prescribing errors of community-based providers may inform strategies to improve ambulatory medication safety, especially e-prescribing.

http://www.clinicaltrials.gov, NCT00225576.

The electronic medical charts of 1507 patients diagnosed with histologically confirmed primary invasive breast cancer.

The automatic drug treatment classification tool consisted of components for: (1) extraction of drug treatment-relevant information from clinical narratives using natural language processing (clinical Text Analysis and Knowledge Extraction System); (2) extraction of drug treatment data from an electronic prescribing system; (3) merging information to create a patient treatment timeline; and (4) final classification logic.

Agreement between results from the algorithm and from a nurse abstractor is measured for categories: (0) no tamoxifen or aromatase inhibitor (AI) treatment; (1) tamoxifen only; (2) AI only; (3) tamoxifen before AI; (4) AI before tamoxifen; (5) multiple AIs and tamoxifen cycles in no specific order; and (6) no specific treatment dates. Specificity (all categories): 96.14%–100%; sensitivity (categories (0)–(4)): 90.27%–99.83%; sensitivity (categories (5)–(6)): 0–23.53%; positive predictive values: 80%–97.38%; negative predictive values: 96.91%–99.93%.

Our approach illustrates a secondary use of the electronic medical record. The main challenge is event temporality.

We present an algorithm for automated treatment classification within an electronic medical record to combine information extracted through natural language processing with that extracted from structured databases. The algorithm has high specificity for all categories, high sensitivity for five categories, and low sensitivity for two categories.

Prescription-filling data for 2 million subjects derived from a payor's insurance claims were used to evaluate adherence to three chronic drugs over 1 year. We relied on patterns of prescription fills, including the length of gaps in medication possession, to measure adherence among subjects and to build models for predicting poor long-term adherence.

All prescription fills for a specific drug were sequenced chronologically into drug eras. 61.3% to 66.5% of the prescription patterns contained medication gaps >30 days during the first year of drug use. These interrupted drug eras include long-term discontinuations, where the subject never again filled a prescription for any drug in that category in the dataset, which represent 23.7% to 29.1% of all drug eras. Among the prescription-filling patterns without large medication gaps, 0.8% to 1.3% exhibited long-term poor adherence. Our models identified these subjects as early as 60 days after the first prescription fill, with an area under the curve (AUC) of 0.81. Model performance improved as the predictions were made at later time-points, with AUC values increasing to 0.93 at the 120-day time-point.

Dispensed medication histories (widely available in real time) are useful for alerting providers about poorly adherent patients and those who will be non-adherent several months later. Efforts to use these data in point of care and decision support facilitating patient are warranted.

Qualitative analysis of 114 telephone interviews conducted with representatives from 97 organizations between February and September 2010, including 24 physician practices, 48 community pharmacies, and three mail-order pharmacies actively transmitting or receiving e-prescriptions via Surescripts.

Practices and pharmacies generally were satisfied with electronic transmission of new prescriptions but reported that the electronic renewal process was used inconsistently, resulting in inefficient workarounds for both parties. Practice communications with mail-order pharmacies were less likely to be electronic than with community pharmacies because of underlying transmission network and computer system limitations. While e-prescribing reduced manual prescription entry, pharmacy staff frequently had to complete or edit certain fields, particularly drug name and patient instructions.

Electronic transmission of new prescriptions has matured. Changes in technical standards and system design and more targeted physician and pharmacy training may be needed to address barriers to e-renewals, mail-order pharmacy connectivity, and pharmacy processing of e-prescriptions.

We conducted a retrospective study of 5166 adults with heart failure in three metropolitan EDs. Patients were termed internal if prior information was in the EHR upon ED presentation, otherwise external. Associations of internality with hospitalization, mortality, length of stay (LOS), and numbers of tests, procedures, and medications ordered in the ED were examined after adjusting for age, gender, race, marital status, comorbidities and hospitalization as a proxy for acuity level where appropriate.

At two EDs internals had lower odds of mortality if hospitalized (OR 0.55; 95% CI 0.38 to 0.81 and 0.45; 0.21 to 0.96), fewer laboratory tests during the ED visit (–4.6%; –8.9% to –0.1% and –14.0%; –19.5% to –8.1%) as well as fewer medications (–33.6%; –38.4% to –28.4% and –21.3%; –33.2% to –7.3%). At one of these two EDs, internals had lower odds of hospitalization (0.37; 0.22 to 0.60). At the third ED, internal patients only experienced a prolonged ED LOS (32.3%; 6.3% to 64.8%) but no other differences. There was no association with hospital LOS or number of procedures ordered.

EHR availability was associated with salutary outcomes in two of three ED settings and prolongation of ED LOS at a third, but evidence was mixed and causality remains to be determined.

An EHR may have the potential to be a valuable adjunct in the care of heart failure patients.

We studied all ED encounters over a 13-month period in which HIE data were accessed in all major emergency departments Memphis, Tennessee. HIE access encounter records were matched with similar encounter records without HIE access. Outcomes studied were ED-originated hospital admissions, admissions for observation, laboratory testing, head CT, body CT, ankle radiographs, chest radiographs, and echocardiograms. Our estimates employed generalized estimating equations for logistic regression models adjusted for admission type, length of stay, and Charlson co-morbidity index. Marginal probabilities were used to calculate changes in outcome variables and their financial consequences.

HIE data were accessed in approximately 6.8% of ED visits across 12 EDs studied. In 11 EDs directly accessing HIE data only through a secure Web browser, access was associated with a decrease in hospital admissions (adjusted odds ratio (OR)=0.27; p<0001). In a 12th ED relying more on print summaries, HIE access was associated with a decrease in hospital admissions (OR=0.48; p<0001) and statistically significant decreases in head CT use, body CT use, and laboratory test ordering.

Applied only to the study population, HIE access was associated with an annual cost savings of $1.9 million. Net of annual operating costs, HIE access reduced overall costs by $1.07 million. Hospital admission reductions accounted for 97.6% of total cost reductions.

Access to additional clinical data through HIE in emergency department settings is associated with net societal saving.

To estimate the long-term cost-effectiveness of a CDSS linked to evidence-based treatment recommendations for type 2 diabetes.

Using the Ontario Diabetes Economic Model, changes in factors (eg, HbA1c) from a randomized controlled trial were used to estimate cost-effectiveness. The cost of implementation, development, and maintenance of the core dataset, and projected diabetes-related complications were included. The base case assumed a 1-year treatment effect, 5% discount rate, and 40-year time horizon. Univariate, one-way sensitivity analyses were carried out by altering different parameter values. The perspective was the Ontario Ministry of Health and costs were in 2010 Canadian dollars.

The cost of implementing the intervention was $483 699. The one-year intervention reduced HbA1c by 0.2 and systolic blood pressure by 3.95 mm Hg, but increased body mass index by 0.02 kg/m², resulting in a relative risk reduction of 14% in the occurrence of amputation. The model estimated that the intervention resulted in an additional 0.0117 quality-adjusted life year; the incremental cost-effectiveness ratio was $160 845 per quality-adjusted life-year.

The web-based prototype decision support system slightly improved short-term risk factors. The model predicted moderate improvements in long-term health outcomes. This disease management program will need to develop considerable efficiencies in terms of costs and processes or improved effectiveness to be considered a cost-effective intervention for treating patients with type 2 diabetes.

A retrospective cross-sectional analysis of data from three sources: CPOE/eMAR usage from HIMSS Analytics (2010), medication quality scores from CMS Hospital Compare (2010), and hospital characteristics from CMS Acute Inpatient Prospective Payment System (2009). The analysis focused on 11 quality indicators (January–December 2009) at 2603 medium-to-large (≥100 beds), non-federal acute-care hospitals measuring proportion of eligible patients given (or prescribed) recommended medications for conditions, including acute myocardial infarction, heart failure, and pneumonia, and surgical care improvement. Using technology adoption by 2008 as reference, hospitals were coded: (1) eMAR-only adopters (n=986); (2) CPOE-only adopters (n=115); and (3) adopters of both technologies (n=804); with non-adopters of both technologies as reference group (n=698). Hospitals were also coded for duration of use in 2-year increments since technology adoption. Hospital characteristics, historical measure-specific patient volume, and propensity scores for technology adoption were used to control for confounding factors. The analysis was performed using a generalized linear model (logit link and binomial family).

Relative to non-adopters of both eMAR and CPOE, the odds of adherence to all measures (except one) were higher by 14–29% for eMAR-only hospitals and by 13–38% for hospitals with both technologies, translating to a marginal increase of 0.4–2.0 percentage points. Further, each additional 2 years of technology use was associated with 6–15% higher odds of compliance on all medication measures for eMAR-only hospitals and users of both technologies.

Implementation and duration of use of health information technologies are associated with improved adherence to medication guidelines at US hospitals. The benefits are evident for adoption of eMAR systems alone and in combination with CPOE.

Cancer registry data and artificial data with increased numbers of missing values in a relevant variable are used for empirical comparisons. As a classification method, classification and regression trees were used. On the resulting binary comparison patterns, the following strategies for dealing with missingness are considered: imputation with unique values, sample-based imputation, reduced-model classification and complete-case induction. These approaches are evaluated according to the number of training data needed for induction and the F-scores achieved.

The evaluations reveal that unique value imputation leads to the best results. Imputation with zero is preferred to imputation with 0.5, although the latter shows the highest median F-scores. Imputation with zero needs considerably less training data, it shows only slightly worse results and simplifies the computation by maintaining the binary structure of the data.

The results support the ad-hoc solution for missing values ‘replace NA by the value of inequality’. This conclusion is based on a limited amount of data and on a specific deduplication method. Nevertheless, the authors are confident that their results should be confirmed by other empirical analyses and applications.

The ethnographic qualitative study was conducted over a 9-month period in six emergency departments (ED) and eight ambulatory clinics in Memphis, Tennessee, USA. Data were collected using direct observation, informal interviews during observation, and formal semi-structured interviews. The authors observed for over 180 h, during which providers used the exchange 130 times.

HIE-related workflow was modeled for each ED site and ambulatory clinic group and substantial site-to-site workflow differences were identified. Common patterns in HIE-related workflow were also identified across all sites, leading to the development of two role-based workflow models: nurse based and physician based. The workflow elements framework was applied to the two role-based patterns. An in-depth description was developed of how providers integrated HIE into existing clinical workflow, including prompts for HIE use.

Workflow differed substantially among sites, but two general role-based HIE usage models were identified. Although providers used HIE to improve continuity of patient care, patient–provider trust played a significant role. Types of information retrieved related to roles, with nurses seeking to retrieve recent hospitalization data and more open-ended usage by nurse practitioners and physicians. User and role-specific customization to accommodate differences in workflow and information needs may increase the adoption and use of HIE.

Understanding end users' perspectives towards HIE technology is crucial to the long-term success of HIE. By applying qualitative methods, an in-depth understanding of HIE usage was developed.

Mobile technologies provide a platform for electronic patient-centered medication management. MyMediHealth (MMH) is a medication management system that includes a medication scheduler, a medication administration record, and a reminder engine that sends text messages to cell phones. The object of this work was to extend MMH to allow two-way interaction using mobile phone-based SMS technology. Unprompted text-message communication with patients using natural language could engage patients in their healthcare, but presents unique natural language processing challenges. The authors developed a new functional component of MMH, the Patient-centered Automated SMS Tagging Engine (PASTE). The PASTE web service uses natural language processing methods, custom lexicons, and existing knowledge sources to extract and tag medication information from patient text messages.

A pilot evaluation of PASTE was completed using 130 medication messages anonymously submitted by 16 volunteers via a website. System output was compared with manually tagged messages.

Verified medication names, medication terms, and action terms reached high F-measures of 91.3%, 94.7%, and 90.4%, respectively. The overall medication name F-measure was 79.8%, and the medication action term F-measure was 90%.

Other studies have demonstrated systems that successfully extract medication information from clinical documents using semantic tagging, regular expression-based approaches, or a combination of both approaches. This evaluation demonstrates the feasibility of extracting medication information from patient-generated medication messages.

Peer-reviewed electronic databases and gray literature were searched to identify studies on HIT used to assist in the medication management process. Articles including an economic component were reviewed for further screening. For this review, full cost-effectiveness analyses, cost-utility analyses and cost-benefit analyses, as well as cost analyses, were eligible for inclusion and synthesis.

The 31 studies included were heterogeneous with respect to the HIT evaluated, setting, and economic methods used. Thus the data could not be synthesized, and a narrative review was conducted. Most studies evaluated computer decision support systems in hospital settings in the USA, and only five of the studied performed full economic evaluations.

Most studies merely provided cost data; however, useful economic data involves far more input. A full economic evaluation includes a full enumeration of the costs, synthesized with the outcomes of the intervention.

The quality of the economic literature in this area is poor. A few studies found that HIT may offer cost advantages despite their increased acquisition costs. However, given the uncertainty that surrounds the costs and outcomes data, and limited study designs, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money. Sophisticated concurrent prospective economic evaluations need to be conducted to address whether HIT interventions in the medication management process are cost-effective.

Markov chain models were built from 2008 web usage data to evaluate the courses' web structure and navigation patterns. The web site was then modified to resolve identified design issues and the improvement in user activity over the subsequent 12 months was quantitatively evaluated.

Web navigation paths were collected between 2008 and 2010. The probability of navigating from one web page to another was analyzed.

The continuing education courses' sequential structure design was clearly reflected in the resulting actual web usage models, and none of the skip transitions provided was heavily used. The web navigation patterns of the two different continuing education courses were similar. Two possible design flaws were identified and fixed in only one of the two courses. Over the following 12 months, the drop-out rate in the modified course significantly decreased from 41% to 35%, but remained unchanged in the unmodified course. The web improvement effects were further verified via a second-order Markov chain model.

The results imply that differences in web content have less impact than web structure design on how learners navigate through continuing education courses. Evaluation of user navigation can help identify web design flaws and guide modifications. This study showed that Markov chain models provide a valuable tool to evaluate web-based education courses. Both the results and techniques in this study would be very useful for public health education and research specialists.

Duke Medicine has deployed a hybrid solution, The Duke Integrated Subject Cohort and Enrollment Research Network (DISCERN), that combines both retrospective warehouse data and clinical events contained in prospective Health Level 7 (HL7) messages to immediately alert study personnel of potential recruits as they become eligible.

DISCERN analyzes more than 500 000 messages daily in service of 12 projects. Users may receive results via email, text pages, or on-demand reports. Preliminary results suggest DISCERN's unique ability to reason over both retrospective and real-time data increases study enrollment rates while reducing the time required to complete recruitment-related tasks. The authors have introduced a preconfigured DISCERN function as a self-service feature for users.

The DISCERN framework is adoptable primarily by organizations using both HL7 message streams and a data warehouse. More efficient recruitment may exacerbate competition for research subjects, and investigators uncomfortable with new technology may find themselves at a competitive disadvantage in recruitment.

DISCERN's hybrid framework for identifying real-time clinical events housed in HL7 messages complements the traditional approach of using retrospective warehoused data. DISCERN is helpful in instances when the required clinical data may not be loaded into the warehouse and thus must be captured contemporaneously during patient care. Use of an open-source tool supports generalizability to other institutions at minimal cost.

A diagnostic test assessment framework was used. A reference set of 2255 articles was created by hand-searching 22 biomedical journals in Medline, and each article was labeled as ‘relevant’, ‘not relevant’, or ‘possibly relevant’ for geriatric medicine. From the relevant articles, search terms were identified to compile different search strategies. The articles retrieved by the various search strategies were compared with articles from the reference set as the index test to create the search filters.

Sensitivity, specificity, precision, accuracy, and number-needed-to-read (NNR) were calculated by comparing the results retrieved by the different search strategies with the reference set.

The most sensitive search filter had a sensitivity of 94.8%, a specificity of 88.7%, a precision of 73.0%, and an accuracy of 90.2%. It had an NNR of 1.37. The most specific search filter had a specificity of 96.6%, a sensitivity of 69.1%, a precision of 86.6%, and an accuracy of 89.9%. It had an NNR of 1.15.

These geriatric search filters simplify searching for relevant literature and therefore contribute to better evidence-based practice. The filters are useful to both the clinician who wants to find a quick answer to a clinical question and the researcher who wants to find as many relevant articles as possible without retrieving too many irrelevant articles.

Prescribers of controlled substances affiliated with a regional health system were surveyed regarding current electronic prescribing (e-prescribing) activities, current prescribing of controlled substances, and expectations and barriers to the adoption of EPCS.

246 prescribers (response rate of 64%) represented a range of medical specialties, with 43.1% of these prescribers current users of e-prescribing for non-controlled substances. Reported issues with controlled substances included errors, pharmacy call-backs, and diversion; most prescribers expected EPCS to address many of these problems, specifically reduce medical errors, improve work flow and efficiency of practice, help identify prescription diversion or misuse, and improve patient treatment management. Prescribers expected, however, that it would be disruptive to practice, and over one-third of respondents reported that carrying a security authentication token at all times would be so burdensome as to discourage adoption.

Although adoption of e-prescribing has been shown to dramatically reduce medication errors, challenges to efficient processes and errors still persist from the perspective of the prescriber, that may interfere with the adoption of EPCS. Most prescribers regarded EPCS security measures as a small or moderate inconvenience (other than carrying a security token), with advantages outweighing the burden.

Prescribers are optimistic about the potential for EPCS to improve practice, but view certain security measures as a burden and potential barrier.

Records from a large healthcare system's enterprise data warehouse (EDW) were linked to a statewide population database, and a master subject index was created. The authors evaluate the linkage, along with the impact of missing information in EDW records and the coverage of the population database. The makeup of the EDW and population database provides a subset of cancer records that exist in both resources, which allows a cancer-specific evaluation of the linkage.

About 3.4 million records (60.8%) in the EDW were linked to the population database with a minimum accuracy of 96.3%. It was estimated that approximately 24.8% of target records were absent from the population database, which enabled the effect of the amount and type of information missing from a record on the linkage to be estimated. However, 99% of the records from the oncology data mart linked; they had fewer missing fields and this correlated positively with the number of patient visits.

A general-purpose research infrastructure was created which allows disease-specific cohorts to be identified. The usefulness of creating an index between institutions is that it allows each institution to maintain control and confidentiality of their own information.

The authors used 394 days of query log (August 1, 2009 to August 29, 2010) from PubMed users in the Texas Medical Center, which is the largest medical center in the world. The authors evaluated two document access models based on the work of Anderson and Schooler. The first is based on how frequently a document was accessed. The second is based on both frequency and recency.

The model based only on frequency of past access was highly correlated with the empirical data (R²=0.932), whereas the model based on frequency and recency had a much lower correlation (R²=0.668).

The frequency-only model accurately predicted whether a document will be accessed based on past use. Modeling accesses as a function of frequency requires storing only the number of accesses and the creation date for the document. This model requires low storage overheads and is computationally efficient, making it scalable to large corpora such as MEDLINE.

It is feasible to accurately model the probability of a document being accessed in the future based on past accesses.

We present an optical character recognition processing pipeline, which leverages the capabilities of existing third-party optical character recognition engines, and provides the flexibility offered by a modular custom-developed system. The system was configured and run on a selected set of form fields extracted from a corpus of handwritten ophthalmology forms.

The processing pipeline allowed multiple configurations to be run, with the optimal configuration consisting of the Nuance and LEADTOOLS engines running in parallel with a positive predictive value of 94.6% and a sensitivity of 13.5%.

While limitations exist, preliminary experience from this project yielded insights on the generalizability and applicability of integrating multiple, inexpensive general-purpose third-party optical character recognition engines in a modular pipeline.

Pre- and post-transition survey administered at six academic-affiliated ambulatory care practices from 2006 to 2008. Four practices transitioned to one commercial EHR and two practices to another. We compared respondents' expectations of, and satisfaction with, the newer EHR.

523 subjects were eligible: 217 were available before transition and 306 after transition. 162 pre-transition and 197 post-transition responses were received, yielding 75% and 64% response rates, respectively. Practitioners were more satisfied with the newer EHRs (64%) compared with the older (56%) (p=0.15) and a small majority (58%) were satisfied with the transition. Practitioners' satisfaction with the older EHRs for completing clinical tasks was high. The newer EHRs exceeded practitioner expectations regarding remote access (61% vs 74%; p=0.03). However, the newer EHRs did not meet practitioners' expectations regarding their ability to perform clinical tasks, or more globally, improve medication safety (81% vs 61%; p<0.001), efficiency (70% vs 44%; p<0.001), and quality of care (77% vs 67%; p=0.04).

Most practitioners had favorable opinions about EHRs and reported overall improved satisfaction with the newer EHRs. However, practitioners' high expectations of the newer EHRs were often unmet regarding facilitation of specific clinical tasks or for improving quality, safety, and efficiency.

To ensure practitioners' expectations, for instance regarding improvements in medication safety, are met, vendors should develop and implement refinements in their software as practices upgrade to newer, certified EHRs.

Qualitative study of six practices operating within the same multi-specialty ambulatory care organization using the same EHR system. Semistructured interviews and direct observation were conducted with all physicians, nurses, medical assistants, practice managers, and non-clinical staff from each practice.

An existing model of practice relationships was used to analyze communication patterns within the practices. Practice-level EHR use was defined and analyzed as the ways in which a practice uses an EHR as a collective or a group—including the degree of feature use, level of EHR-enabled communication, and frequency that EHR use changes in a practice. Interview and observation data were analyzed for themes. Based on these themes, within-practice communication patterns were categorized as fragmented or cohesive, and practice-level EHR use patterns were categorized as heterogeneous or homogeneous. Practices where EHR use was uniformly high across all users were further categorized as having standardized EHR use. Communication patterns and EHR use patterns were compared across the six practices.

Within-practice communication patterns were associated with practice-level EHR use patterns. In practices where communication patterns were fragmented, EHR use was heterogeneous. In practices where communication patterns were cohesive, EHR use was homogeneous. Additional analysis revealed that practices that had achieved standardized EHR use (uniformly high EHR use across all users) exhibited high levels of mindfulness and respectful interaction, whereas practices that were furthest from achieving standardized EHR use exhibited low levels of mindfulness and respectful interaction.

Within-practice communication patterns provide a unique perspective for exploring the issue of standardization in EHR use. A major fallacy of setting homogeneous EHR use as the goal for practice-level EHR use is that practices with uniformly low EHR use could be considered successful. Achieving uniformly high EHR use across all users in a practice is more consistent with the goals of current EHR adoption and use efforts. It was found that some communication patterns among practice members may enable more standardized EHR use than others. Understanding the linkage between communication patterns and EHR use can inform understanding of the human element in EHR use and may provide key lessons for the implementation of EHRs and other health information technologies.

Eighty-one participants aged 50–76 years worked through the on-line PDS tool and completed questionnaires on computer self-efficacy, computer anxiety, attitudes to and beliefs about CRC screening before and after exposure to the PDS, and perceived usability and acceptability of the tool.

Repeated measures ANOVA found that PDS exposure led to a significant increase in knowledge about CRC and screening, and more positive attitudes to CRC screening as measured by factors from the Preventive Health Model. Perceived usability and acceptability of the PDS mediated changes in attitudes toward CRC screening (but not CRC knowledge), and computer self-efficacy and computer anxiety were significant predictors of individuals' perceptions of the tool.

Interventions designed to decrease computer anxiety, such as computer courses and internet training, may improve the acceptability of new health information technologies including internet-based decision support tools, increasing their impact on behavior change.

Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI).

Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate.

The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.

Word frequency analysis for ‘values terms’ was performed on citations on diabetes, obesity, dementia, and schizophrenia (Medline; 2004–2006; 4440 citations; 1 110 291 words). Concordance® and SPSS 14.0 were used. Text words and MeSH terms of high frequency and precision were compiled into a search filter. It was validated on datasets of citations on dentistry and food hypersensitivity.

144 unique text words and 124 unique MeSH terms of moderate and high frequency (≥20) and very high precision (≥90%) were identified. Of these, 19 text words and seven MeSH terms were compiled into a ‘brief values filter’. In the derivation dataset, it had a sensitivity of 76.8% and precision of 86.8%. In the validation datasets, its sensitivity and precision were, respectively, 70.1% and 63.6% (food hypersensitivity) and 47.1% and 82.6% (dentistry).

This study provided a varied pool of search terms and a simple and highly effective tool for retrieving publications on health-related values. Further work is required to facilitate access to such research and enhance its chances of being translated into practice, policy, and service improvements.

Qualitative analysis of interviews with identified key stakeholders followed by an electronic survey.

We conducted semi-structured interviews with 29 key Canadian HIT policy and opinion leaders and used a grounded theory approach to analyze the results. The informant sample was chosen to provide views from different stakeholder groups including national representatives and regional representatives from three Canadian provinces.

Canadian informants believed that much of the current US direction is positive, especially regarding incentives and meaningful use, but that there are key opportunities for the US to emphasize direct engagement with providers, define a clear business case for them, sponsor large scale evaluations to assess HIT impact in a broad array of settings, determine standards but also enable access to resources needed for mid-course corrections of standards when issues are identified, and, finally, leverage implementation of digital imaging systems.

Not all stakeholder groups were included, such as providers or patients. In addition, as in all qualitative research, a selection bias could be present due to the relatively small sample size.

Based on Canadian experience with HIT policy, stakeholders identified as lessons for the US the need to increase direct engagement with providers and the importance of defining the business case for HIT, which can be achieved through large scale evaluations, and of recognizing and leveraging successes as they emerge.