We studied all ED encounters over a 13-month period in which HIE data were accessed in all major emergency departments Memphis, Tennessee. HIE access encounter records were matched with similar encounter records without HIE access. Outcomes studied were ED-originated hospital admissions, admissions for observation, laboratory testing, head CT, body CT, ankle radiographs, chest radiographs, and echocardiograms. Our estimates employed generalized estimating equations for logistic regression models adjusted for admission type, length of stay, and Charlson co-morbidity index. Marginal probabilities were used to calculate changes in outcome variables and their financial consequences.

HIE data were accessed in approximately 6.8% of ED visits across 12 EDs studied. In 11 EDs directly accessing HIE data only through a secure Web browser, access was associated with a decrease in hospital admissions (adjusted odds ratio (OR)=0.27; p<0001). In a 12th ED relying more on print summaries, HIE access was associated with a decrease in hospital admissions (OR=0.48; p<0001) and statistically significant decreases in head CT use, body CT use, and laboratory test ordering.

Applied only to the study population, HIE access was associated with an annual cost savings of $1.9 million. Net of annual operating costs, HIE access reduced overall costs by $1.07 million. Hospital admission reductions accounted for 97.6% of total cost reductions.

Access to additional clinical data through HIE in emergency department settings is associated with net societal saving.

We conducted a retrospective study of 5166 adults with heart failure in three metropolitan EDs. Patients were termed internal if prior information was in the EHR upon ED presentation, otherwise external. Associations of internality with hospitalization, mortality, length of stay (LOS), and numbers of tests, procedures, and medications ordered in the ED were examined after adjusting for age, gender, race, marital status, comorbidities and hospitalization as a proxy for acuity level where appropriate.

At two EDs internals had lower odds of mortality if hospitalized (OR 0.55; 95% CI 0.38 to 0.81 and 0.45; 0.21 to 0.96), fewer laboratory tests during the ED visit (–4.6%; –8.9% to –0.1% and –14.0%; –19.5% to –8.1%) as well as fewer medications (–33.6%; –38.4% to –28.4% and –21.3%; –33.2% to –7.3%). At one of these two EDs, internals had lower odds of hospitalization (0.37; 0.22 to 0.60). At the third ED, internal patients only experienced a prolonged ED LOS (32.3%; 6.3% to 64.8%) but no other differences. There was no association with hospital LOS or number of procedures ordered.

EHR availability was associated with salutary outcomes in two of three ED settings and prolongation of ED LOS at a third, but evidence was mixed and causality remains to be determined.

An EHR may have the potential to be a valuable adjunct in the care of heart failure patients.

To estimate the long-term cost-effectiveness of a CDSS linked to evidence-based treatment recommendations for type 2 diabetes.

Using the Ontario Diabetes Economic Model, changes in factors (eg, HbA1c) from a randomized controlled trial were used to estimate cost-effectiveness. The cost of implementation, development, and maintenance of the core dataset, and projected diabetes-related complications were included. The base case assumed a 1-year treatment effect, 5% discount rate, and 40-year time horizon. Univariate, one-way sensitivity analyses were carried out by altering different parameter values. The perspective was the Ontario Ministry of Health and costs were in 2010 Canadian dollars.

The cost of implementing the intervention was $483 699. The one-year intervention reduced HbA1c by 0.2 and systolic blood pressure by 3.95 mm Hg, but increased body mass index by 0.02 kg/m², resulting in a relative risk reduction of 14% in the occurrence of amputation. The model estimated that the intervention resulted in an additional 0.0117 quality-adjusted life year; the incremental cost-effectiveness ratio was $160 845 per quality-adjusted life-year.

The web-based prototype decision support system slightly improved short-term risk factors. The model predicted moderate improvements in long-term health outcomes. This disease management program will need to develop considerable efficiencies in terms of costs and processes or improved effectiveness to be considered a cost-effective intervention for treating patients with type 2 diabetes.

Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI).

Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate.

The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.

Qualitative analysis of 114 telephone interviews conducted with representatives from 97 organizations between February and September 2010, including 24 physician practices, 48 community pharmacies, and three mail-order pharmacies actively transmitting or receiving e-prescriptions via Surescripts.

Practices and pharmacies generally were satisfied with electronic transmission of new prescriptions but reported that the electronic renewal process was used inconsistently, resulting in inefficient workarounds for both parties. Practice communications with mail-order pharmacies were less likely to be electronic than with community pharmacies because of underlying transmission network and computer system limitations. While e-prescribing reduced manual prescription entry, pharmacy staff frequently had to complete or edit certain fields, particularly drug name and patient instructions.

Electronic transmission of new prescriptions has matured. Changes in technical standards and system design and more targeted physician and pharmacy training may be needed to address barriers to e-renewals, mail-order pharmacy connectivity, and pharmacy processing of e-prescriptions.

A retrospective cross-sectional analysis of data from three sources: CPOE/eMAR usage from HIMSS Analytics (2010), medication quality scores from CMS Hospital Compare (2010), and hospital characteristics from CMS Acute Inpatient Prospective Payment System (2009). The analysis focused on 11 quality indicators (January–December 2009) at 2603 medium-to-large (≥100 beds), non-federal acute-care hospitals measuring proportion of eligible patients given (or prescribed) recommended medications for conditions, including acute myocardial infarction, heart failure, and pneumonia, and surgical care improvement. Using technology adoption by 2008 as reference, hospitals were coded: (1) eMAR-only adopters (n=986); (2) CPOE-only adopters (n=115); and (3) adopters of both technologies (n=804); with non-adopters of both technologies as reference group (n=698). Hospitals were also coded for duration of use in 2-year increments since technology adoption. Hospital characteristics, historical measure-specific patient volume, and propensity scores for technology adoption were used to control for confounding factors. The analysis was performed using a generalized linear model (logit link and binomial family).

Relative to non-adopters of both eMAR and CPOE, the odds of adherence to all measures (except one) were higher by 14–29% for eMAR-only hospitals and by 13–38% for hospitals with both technologies, translating to a marginal increase of 0.4–2.0 percentage points. Further, each additional 2 years of technology use was associated with 6–15% higher odds of compliance on all medication measures for eMAR-only hospitals and users of both technologies.

Implementation and duration of use of health information technologies are associated with improved adherence to medication guidelines at US hospitals. The benefits are evident for adoption of eMAR systems alone and in combination with CPOE.

Mobile technologies provide a platform for electronic patient-centered medication management. MyMediHealth (MMH) is a medication management system that includes a medication scheduler, a medication administration record, and a reminder engine that sends text messages to cell phones. The object of this work was to extend MMH to allow two-way interaction using mobile phone-based SMS technology. Unprompted text-message communication with patients using natural language could engage patients in their healthcare, but presents unique natural language processing challenges. The authors developed a new functional component of MMH, the Patient-centered Automated SMS Tagging Engine (PASTE). The PASTE web service uses natural language processing methods, custom lexicons, and existing knowledge sources to extract and tag medication information from patient text messages.

A pilot evaluation of PASTE was completed using 130 medication messages anonymously submitted by 16 volunteers via a website. System output was compared with manually tagged messages.

Verified medication names, medication terms, and action terms reached high F-measures of 91.3%, 94.7%, and 90.4%, respectively. The overall medication name F-measure was 79.8%, and the medication action term F-measure was 90%.

Other studies have demonstrated systems that successfully extract medication information from clinical documents using semantic tagging, regular expression-based approaches, or a combination of both approaches. This evaluation demonstrates the feasibility of extracting medication information from patient-generated medication messages.

Prescribers of controlled substances affiliated with a regional health system were surveyed regarding current electronic prescribing (e-prescribing) activities, current prescribing of controlled substances, and expectations and barriers to the adoption of EPCS.

246 prescribers (response rate of 64%) represented a range of medical specialties, with 43.1% of these prescribers current users of e-prescribing for non-controlled substances. Reported issues with controlled substances included errors, pharmacy call-backs, and diversion; most prescribers expected EPCS to address many of these problems, specifically reduce medical errors, improve work flow and efficiency of practice, help identify prescription diversion or misuse, and improve patient treatment management. Prescribers expected, however, that it would be disruptive to practice, and over one-third of respondents reported that carrying a security authentication token at all times would be so burdensome as to discourage adoption.

Although adoption of e-prescribing has been shown to dramatically reduce medication errors, challenges to efficient processes and errors still persist from the perspective of the prescriber, that may interfere with the adoption of EPCS. Most prescribers regarded EPCS security measures as a small or moderate inconvenience (other than carrying a security token), with advantages outweighing the burden.

Prescribers are optimistic about the potential for EPCS to improve practice, but view certain security measures as a burden and potential barrier.

Qualitative study of six practices operating within the same multi-specialty ambulatory care organization using the same EHR system. Semistructured interviews and direct observation were conducted with all physicians, nurses, medical assistants, practice managers, and non-clinical staff from each practice.

An existing model of practice relationships was used to analyze communication patterns within the practices. Practice-level EHR use was defined and analyzed as the ways in which a practice uses an EHR as a collective or a group—including the degree of feature use, level of EHR-enabled communication, and frequency that EHR use changes in a practice. Interview and observation data were analyzed for themes. Based on these themes, within-practice communication patterns were categorized as fragmented or cohesive, and practice-level EHR use patterns were categorized as heterogeneous or homogeneous. Practices where EHR use was uniformly high across all users were further categorized as having standardized EHR use. Communication patterns and EHR use patterns were compared across the six practices.

Within-practice communication patterns were associated with practice-level EHR use patterns. In practices where communication patterns were fragmented, EHR use was heterogeneous. In practices where communication patterns were cohesive, EHR use was homogeneous. Additional analysis revealed that practices that had achieved standardized EHR use (uniformly high EHR use across all users) exhibited high levels of mindfulness and respectful interaction, whereas practices that were furthest from achieving standardized EHR use exhibited low levels of mindfulness and respectful interaction.

Within-practice communication patterns provide a unique perspective for exploring the issue of standardization in EHR use. A major fallacy of setting homogeneous EHR use as the goal for practice-level EHR use is that practices with uniformly low EHR use could be considered successful. Achieving uniformly high EHR use across all users in a practice is more consistent with the goals of current EHR adoption and use efforts. It was found that some communication patterns among practice members may enable more standardized EHR use than others. Understanding the linkage between communication patterns and EHR use can inform understanding of the human element in EHR use and may provide key lessons for the implementation of EHRs and other health information technologies.

The ethnographic qualitative study was conducted over a 9-month period in six emergency departments (ED) and eight ambulatory clinics in Memphis, Tennessee, USA. Data were collected using direct observation, informal interviews during observation, and formal semi-structured interviews. The authors observed for over 180 h, during which providers used the exchange 130 times.

HIE-related workflow was modeled for each ED site and ambulatory clinic group and substantial site-to-site workflow differences were identified. Common patterns in HIE-related workflow were also identified across all sites, leading to the development of two role-based workflow models: nurse based and physician based. The workflow elements framework was applied to the two role-based patterns. An in-depth description was developed of how providers integrated HIE into existing clinical workflow, including prompts for HIE use.

Workflow differed substantially among sites, but two general role-based HIE usage models were identified. Although providers used HIE to improve continuity of patient care, patient–provider trust played a significant role. Types of information retrieved related to roles, with nurses seeking to retrieve recent hospitalization data and more open-ended usage by nurse practitioners and physicians. User and role-specific customization to accommodate differences in workflow and information needs may increase the adoption and use of HIE.

Understanding end users' perspectives towards HIE technology is crucial to the long-term success of HIE. By applying qualitative methods, an in-depth understanding of HIE usage was developed.

Pre- and post-transition survey administered at six academic-affiliated ambulatory care practices from 2006 to 2008. Four practices transitioned to one commercial EHR and two practices to another. We compared respondents' expectations of, and satisfaction with, the newer EHR.

523 subjects were eligible: 217 were available before transition and 306 after transition. 162 pre-transition and 197 post-transition responses were received, yielding 75% and 64% response rates, respectively. Practitioners were more satisfied with the newer EHRs (64%) compared with the older (56%) (p=0.15) and a small majority (58%) were satisfied with the transition. Practitioners' satisfaction with the older EHRs for completing clinical tasks was high. The newer EHRs exceeded practitioner expectations regarding remote access (61% vs 74%; p=0.03). However, the newer EHRs did not meet practitioners' expectations regarding their ability to perform clinical tasks, or more globally, improve medication safety (81% vs 61%; p<0.001), efficiency (70% vs 44%; p<0.001), and quality of care (77% vs 67%; p=0.04).

Most practitioners had favorable opinions about EHRs and reported overall improved satisfaction with the newer EHRs. However, practitioners' high expectations of the newer EHRs were often unmet regarding facilitation of specific clinical tasks or for improving quality, safety, and efficiency.

To ensure practitioners' expectations, for instance regarding improvements in medication safety, are met, vendors should develop and implement refinements in their software as practices upgrade to newer, certified EHRs.

Eighty-one participants aged 50–76 years worked through the on-line PDS tool and completed questionnaires on computer self-efficacy, computer anxiety, attitudes to and beliefs about CRC screening before and after exposure to the PDS, and perceived usability and acceptability of the tool.

Repeated measures ANOVA found that PDS exposure led to a significant increase in knowledge about CRC and screening, and more positive attitudes to CRC screening as measured by factors from the Preventive Health Model. Perceived usability and acceptability of the PDS mediated changes in attitudes toward CRC screening (but not CRC knowledge), and computer self-efficacy and computer anxiety were significant predictors of individuals' perceptions of the tool.

Interventions designed to decrease computer anxiety, such as computer courses and internet training, may improve the acceptability of new health information technologies including internet-based decision support tools, increasing their impact on behavior change.

A diagnostic test assessment framework was used. A reference set of 2255 articles was created by hand-searching 22 biomedical journals in Medline, and each article was labeled as ‘relevant’, ‘not relevant’, or ‘possibly relevant’ for geriatric medicine. From the relevant articles, search terms were identified to compile different search strategies. The articles retrieved by the various search strategies were compared with articles from the reference set as the index test to create the search filters.

Sensitivity, specificity, precision, accuracy, and number-needed-to-read (NNR) were calculated by comparing the results retrieved by the different search strategies with the reference set.

The most sensitive search filter had a sensitivity of 94.8%, a specificity of 88.7%, a precision of 73.0%, and an accuracy of 90.2%. It had an NNR of 1.37. The most specific search filter had a specificity of 96.6%, a sensitivity of 69.1%, a precision of 86.6%, and an accuracy of 89.9%. It had an NNR of 1.15.

These geriatric search filters simplify searching for relevant literature and therefore contribute to better evidence-based practice. The filters are useful to both the clinician who wants to find a quick answer to a clinical question and the researcher who wants to find as many relevant articles as possible without retrieving too many irrelevant articles.

The authors used 394 days of query log (August 1, 2009 to August 29, 2010) from PubMed users in the Texas Medical Center, which is the largest medical center in the world. The authors evaluated two document access models based on the work of Anderson and Schooler. The first is based on how frequently a document was accessed. The second is based on both frequency and recency.

The model based only on frequency of past access was highly correlated with the empirical data (R²=0.932), whereas the model based on frequency and recency had a much lower correlation (R²=0.668).

The frequency-only model accurately predicted whether a document will be accessed based on past use. Modeling accesses as a function of frequency requires storing only the number of accesses and the creation date for the document. This model requires low storage overheads and is computationally efficient, making it scalable to large corpora such as MEDLINE.

It is feasible to accurately model the probability of a document being accessed in the future based on past accesses.

Word frequency analysis for ‘values terms’ was performed on citations on diabetes, obesity, dementia, and schizophrenia (Medline; 2004–2006; 4440 citations; 1 110 291 words). Concordance® and SPSS 14.0 were used. Text words and MeSH terms of high frequency and precision were compiled into a search filter. It was validated on datasets of citations on dentistry and food hypersensitivity.

144 unique text words and 124 unique MeSH terms of moderate and high frequency (≥20) and very high precision (≥90%) were identified. Of these, 19 text words and seven MeSH terms were compiled into a ‘brief values filter’. In the derivation dataset, it had a sensitivity of 76.8% and precision of 86.8%. In the validation datasets, its sensitivity and precision were, respectively, 70.1% and 63.6% (food hypersensitivity) and 47.1% and 82.6% (dentistry).

This study provided a varied pool of search terms and a simple and highly effective tool for retrieving publications on health-related values. Further work is required to facilitate access to such research and enhance its chances of being translated into practice, policy, and service improvements.

Markov chain models were built from 2008 web usage data to evaluate the courses' web structure and navigation patterns. The web site was then modified to resolve identified design issues and the improvement in user activity over the subsequent 12 months was quantitatively evaluated.

Web navigation paths were collected between 2008 and 2010. The probability of navigating from one web page to another was analyzed.

The continuing education courses' sequential structure design was clearly reflected in the resulting actual web usage models, and none of the skip transitions provided was heavily used. The web navigation patterns of the two different continuing education courses were similar. Two possible design flaws were identified and fixed in only one of the two courses. Over the following 12 months, the drop-out rate in the modified course significantly decreased from 41% to 35%, but remained unchanged in the unmodified course. The web improvement effects were further verified via a second-order Markov chain model.

The results imply that differences in web content have less impact than web structure design on how learners navigate through continuing education courses. Evaluation of user navigation can help identify web design flaws and guide modifications. This study showed that Markov chain models provide a valuable tool to evaluate web-based education courses. Both the results and techniques in this study would be very useful for public health education and research specialists.

An algorithm was developed to identify T2D cases and controls based on a combination of diagnoses, medications, and laboratory results. The performance of the algorithm was validated at three of the five participating institutions compared against clinician review. A GWAS was subsequently performed using cases and controls identified by the algorithm, with samples pooled across all five institutions.

The algorithm achieved 98% and 100% positive predictive values for the identification of diabetic cases and controls, respectively, as compared against clinician review. By standardizing and applying the algorithm across institutions, 3353 cases and 3352 controls were identified. Subsequent GWAS using data from five institutions replicated the TCF7L2 gene variant (rs7903146) previously associated with T2D.

By applying stringent criteria to EMR data collected through routine clinical care, cases and controls for a GWAS were identified that subsequently replicated a known genetic variant. The use of standard terminologies to define data elements enabled pooling of subjects and data across five different institutions to achieve the robust numbers required for GWAS.

An algorithm using commonly available data from five different EMR can accurately identify T2D cases and controls for genetic study across multiple institutions.

This population-based study identified all Olmsted County, Minnesota residents in 2007. We used provider-linked electronic medical record data from the two healthcare centers that provide >95% of all care to County residents (ie, Olmsted Medical Center and Mayo Clinic in Rochester, Minnesota, USA). Subjects were limited to residents with one or more encounter January 1, 2006 through December 31, 2007 at both healthcare centers. DM-relevant data on diagnoses, laboratory results, and medication from both centers were obtained during this period. The algorithm was first executed using data from both centers (ie, the gold standard) and then from Mayo Clinic alone. Positive predictive values and false-negative rates were calculated, and the McNemar test was used to compare categorization when data from the Mayo Clinic alone were used with the gold standard. Age and sex were compared between true-positive and false-negative subjects with T2DM. Statistical significance was accepted as p<0.05.

With data from both medical centers, 765 subjects with T2DM (4256 non-DM subjects) were identified. When single-center data were used, 252 T2DM subjects (1573 non-DM subjects) were missed; an additional false-positive 27 T2DM subjects (215 non-DM subjects) were identified. The positive predictive values and false-negative rates were 95.0% (513/540) and 32.9% (252/765), respectively, for T2DM subjects and 92.6% (2683/2898) and 37.0% (1573/4256), respectively, for non-DM subjects. Age and sex distribution differed between true-positive (mean age 62.1; 45% female) and false-negative (mean age 65.0; 56.0% female) T2DM subjects.

The findings show that application of an HTCP algorithm using data from a single medical center contributes to misclassification. These findings should be considered carefully by researchers when developing and executing HTCP algorithms.

We used a multi-modal strategy consisting of structured database querying, natural language processing on free-text documents, and optical character recognition on scanned clinical images to identify cataract subjects and related cataract attributes. Extensive validation on 3657 subjects compared the multi-modal results to manual chart review. The algorithm was also implemented at participating electronic MEdical Records and GEnomics (eMERGE) institutions.

An EHR-based cataract phenotyping algorithm was successfully developed and validated, resulting in positive predictive values (PPVs) >95%. The multi-modal approach increased the identification of cataract subject attributes by a factor of three compared to single-mode approaches while maintaining high PPV. Components of the cataract algorithm were successfully deployed at three other institutions with similar accuracy.

A multi-modal strategy incorporating optical character recognition and natural language processing may increase the number of cases identified while maintaining similar PPVs. Such algorithms, however, require that the needed information be embedded within clinical documents.

We have demonstrated that algorithms to identify and characterize cataracts can be developed utilizing data collected via the EHR. These algorithms provide a high level of accuracy even when implemented across multiple EHRs and institutional boundaries.

The authors proposed and implemented several methods to extract lists of key terms associated with Medical Subject Heading terms. These key terms are used to represent the document topic in a knowledge-based WSD system. They are applied both alone and in combination with local context.

A standard measure of accuracy was calculated over the set of target words in the widely used National Library of Medicine WSD dataset.

The authors report a significant improvement when combining those key terms with local context, showing that domain information improves the results of a WSD system based on the Unified Medical Language System Metathesaurus alone. The best results were obtained using key terms obtained by relevance feedback and weighted by inverse document frequency.

In this paper, we present a novel approach to predict disease genes and identify dysfunctional networks or modules, based on the analysis of differential interactions between disease and control samples, in contrast to the analysis of differential gene or protein expressions widely adopted in existing methods.

As an example, we applied our method to the study of three-stage microarray data for gastric cancer. We identified network modules or module biomarkers that include a set of genes related to gastric cancer, implying the predictive power of our method. The results on holdout validation data sets show that our identified module can serve as an effective module biomarker for accurately detecting or diagnosing gastric cancer, thereby validating the efficiency of our method.

We proposed a new approach to detect module biomarkers for diseases, and the results on gastric cancer demonstrated that the differential interactions are useful to detect dysfunctional modules in the molecular interaction network, which in turn can be used as robust module biomarkers.

A vector space model approach was developed that bridges gene disease knowledge inferred across three knowledge bases: Online Mendelian Inheritance in Man, GenBank, and Medline. The approach was then used to identify potentially related diseases for two target diseases: Alzheimer disease and Prader-Willi Syndrome.

In the case of both Alzheimer Disease and Prader-Willi Syndrome, a set of plausible diseases were identified that may warrant further exploration.

This study furthers seminal work by Swanson, et al. that demonstrated the potential for mining literature for putative correlations. Using a vector space modeling approach, information from both biomedical literature and genomic resources (like GenBank) can be combined towards identification of putative correlations of interest. To this end, the relevance of the predicted diseases of interest in this study using the vector space modeling approach were validated based on supporting literature.

The results of this study suggest that a vector space model approach may be a useful means to identify potential relationships between complex diseases, and thereby enable the coordination of gene-based findings across multiple complex diseases.

The patient's eGFR was predicted using donor–recipient characteristics available at the time of transplantation. Donors' data were obtained from Eurotransplant's database, while recipients' details were retrieved from Charité Campus Virchow-Klinikum's database. A total of 707 renal transplantations from cadaveric donors were included.

Two separate datasets were created, taking features with <10% missing values for one and <50% missing values for the other. Four established regressors were run on both datasets, with and without feature selection.

The authors obtained a Pearson correlation coefficient between predicted and real eGFR (COR) of 0.48. The best model for the dataset was a Gaussian support vector machine with recursive feature elimination on the more inclusive dataset. All results are available at http://transplant.molgen.mpg.de/.

For now, missing values in the data must be predicted and filled in. The performance is not as high as hoped, but the dataset seems to be the main cause.

Predicting the outcome is possible with the dataset at hand (COR=0.48). Valuable features include age and creatinine levels of the donor, as well as sex and weight of the recipient.

We propose an adaptive technique that utilizes individualized confidence intervals (CIs) to calibrate predictions. We evaluate this new method, adaptive calibration of predictions (ACP), in artificial and real-world medical classification problems, in terms of areas under the ROC curves, the Hosmer-Lemeshow goodness-of-fit test, mean squared error, and computational complexity.

ACP compared favorably to other calibration methods such as binning, Platt scaling, and isotonic regression. In several experiments, binning, isotonic regression, and Platt scaling failed to improve the calibration of a logistic regression model, whereas ACP consistently improved the calibration while maintaining the same discrimination or even improving it in some experiments. In addition, the ACP algorithm is not computationally expensive.

The calculation of CIs for individual predictions may be cumbersome for certain predictive models. ACP is not completely parameter-free: the length of the CI employed may affect its results.

ACP can generate estimates that may be more suitable for individualized predictions than estimates that are calibrated using existing methods. Further studies are necessary to explore the limitations of ACP.

Disease mutations aggregated from multiple databases were mapped to domains, and were classified as either cancer- or non-cancer-related. The statistical method for identifying significantly disease-associated domain positions was applied to both sets of mutations and to randomly generated mutation sets for comparison. To leverage the known function of protein domain regions, the method optionally distributes significant scores to associated functional feature positions.

Most disease mutations are localized within protein domains and display a tendency to cluster at individual domain positions. The method identified significant disease mutation hotspots in both the cancer and non-cancer datasets. The domain significance scores (DS-scores) for cancer form a bimodal distribution with hotspots in oncogenes forming a second peak at higher DS-scores than non-cancer, and hotspots in tumor suppressors have scores more similar to non-cancers. In addition, on an independent mutation benchmarking set, the DS-score method identified mutations known to alter protein function with very high precision.

By aggregating mutations with known disease association at the domain level, the method was able to discover domain positions enriched with multiple occurrences of deleterious mutations while incorporating relevant functional annotations. The method can be incorporated into translational bioinformatics tools to characterize rare and novel variants within large-scale sequencing studies.

We use a heterogeneous collection of datasets across human and mice samples from a range of tissues and different forms of immunodeficiency. We developed a method integrating Tibshirani's modified t-test (SAM) is used to interrogate differential expression within a study and Fisher's method for omnibus meta-analysis to identify differentially expressed genes across studies. The ability of this overall gene expression profile to prioritize disease associated genes is evaluated by comparing against the results of a recent genome wide association study for common variable immunodeficiency (CVID).

Our approach is able to prioritize genes associated with immunodeficiency in general (area under the ROC curve = 0.713) and CVID in particular (area under the ROC curve = 0.643).

This approach may be used to investigate a larger range of failures of the immune system. Our method may be extended to other disease processes, using RNA levels to prioritize genes likely to contain disease associated DNA variants.

To determine whether the 1000 Genomes Project includes all the variants in HapMap, we examined the overlap between single nucleotide polymorphisms (SNPs) genotyped in the two resources using merged phase II/III HapMap data and low coverage pilot data from 1000 Genomes.

Comparison of the two data sets showed that approximately 72% of HapMap SNPs were also found in 1000 Genomes Project pilot data. After filtering out HapMap variants with a MAF of <5% (separately for each population), 99% of HapMap SNPs were found in 1000 Genomes data.

Not all variants cataloged in HapMap are also cataloged in 1000 Genomes. This could affect decisions about which resource to use for SNP queries, rare variant validation, or imputation. Both the HapMap and 1000 Genomes Project databases are useful resources for human genetics, but it is important to understand the assumptions made and filtering strategies employed by these projects.

Trait-to-polymorphism (SNPs) associations confirmed in GWAS were used. A novel method to quantify trait–trait similarity anchored in Gene Ontology annotations of human proteins and information theory was developed. The results were then validated with the shortest paths of physical protein interactions between biologically similar traits.

A network was constructed consisting of 280 significant intertrait similarities among 177 disease traits, which covered 1438 well-validated disease-associated SNPs. Thirty-nine percent of intertrait connections were confirmed by curators, and the following additional studies demonstrated the validity of a proportion of the remainder. On a phenotypic trait level, higher Gene Ontology similarity between proteins correlated with smaller ‘shortest distance’ in protein interaction networks of complexly inherited diseases (Spearman p<2.2x10^–16). Further, ‘cancer traits’ were similar to one another, as were ‘metabolic syndrome traits’ (Fisher's exact test p=0.001 and 3.5x10^–7, respectively).

An imputed disease network by information-anchored functional similarity from GWAS trait-associated SNPs is reported. It is also demonstrated that small shortest paths of protein interactions correlate with complex-disease function. Taken together, these findings provide the framework for investigating drug targets with unbiased functional biomolecular networks rather than worn-out single-gene and subjective canonical pathway approaches.

An analytical method, SPADE-gen, spanning three biological scales was developed to demonstrate the mechanistic concordance between Mendelian and complex inheritance of Alzheimer's disease (AD) genes: biological functions (BP), protein interaction modeling, and protein domain implicated in the disease-associated polymorphism.

Among Gene Ontology (GO) biological processes (BP) enriched at a false detection rate <5% in 15 AD genes of Mendelian inheritance (Online Mendelian Inheritance in Man) and independently in those of complex inheritance (25 host genes of intragenic AD single-nucleotide polymorphisms confirmed in genome-wide association studies), 16 overlapped (empirical p=0.007) and 45 were similar (empirical p<0.009; information theory). SPAN network modeling extended the canonical pathway of AD (KEGG) with 26 new protein interactions (empirical p<0.0001).

The study prioritized new AD-associated biological mechanisms and focused the analysis on previously unreported interactions associated with the biological processes of polymorphisms that affect specific protein domains within characterized AD genes and their direct interactors using (1) concordant GO-BP and (2) domain interactions within STRING protein–protein interactions corresponding to the genomic location of the AD polymorphism (eg, EPHA1, APOE, and CD2AP).

These results are in line with unique-event polymorphism theory, indicating how disease-associated polymorphisms of Mendelian or complex inheritance relate genetically to those observed as ‘unique personal variants’. They also provide insight for identifying novel targets, for repositioning drugs, and for personal therapeutics.

Methodology was applied to 162 glioblastomas from The Cancer Genome Atlas to identify morphology-driven clusters and their clinical and molecular correlates. Signatures of patient-specific tumor morphology were generated from analysis of 200 million cells in 462 whole slide images. Morphology-driven clusters were interrogated for associations with patient outcome, response to therapy, molecular classifications, and genetic alterations. An additional layer of deep, genome-wide analysis identified characteristic transcriptional, epigenetic, and copy number variation events.

Analysis of glioblastoma identified three prognostically significant patient clusters (median survival 15.3, 10.7, and 13.0 months, log rank p=1.4e-3). Clustering results were validated in a separate dataset. Clusters were characterized by molecular events in nuclear compartment signaling including developmental and cell cycle checkpoint pathways. This analysis demonstrates the potential of high-throughput morphometrics for the subclassification of disease, establishing an approach that complements genomics.

HIT events submitted to MAUDE were retrieved using a standardized search strategy. Using an emerging classification with 32 categories of HIT problems, a subset of relevant events were iteratively analyzed to identify new categories. Two coders then independently classified the remaining events into one or more categories. Free-text descriptions were analyzed to identify the consequences of events.

Descriptive statistics by number of reported problems per category and by consequence; inter-rater reliability analysis using the statistic for the major categories and consequences.

A search of 899 768 reports from January 2008 to July 2010 yielded 1100 reports about HIT. After removing duplicate and unrelated reports, 678 reports describing 436 events remained. The authors identified four new categories to describe problems with software functionality, system configuration, interface with devices, and network configuration; the authors' classification with 32 categories of HIT problems was expanded by the addition of these four categories. Examination of the 436 events revealed 712 problems, 96% were machine-related, and 4% were problems at the human–computer interface. Almost half (46%) of the events related to hazardous circumstances. Of the 46 events (11%) associated with patient harm, four deaths were linked to HIT problems (0.9% of 436 events).

Only 0.1% of the MAUDE reports searched were related to HIT. Nevertheless, Food and Drug Administration reports did prove to be a useful new source of information about the nature of software problems and their safety implications with potential to inform strategies for safe design and implementation.

To validate the OMOP CDM, the model was instantiated into a relational database, data from 10 different observational healthcare databases were loaded into separate instances, a comprehensive array of analytic methods that operate on the data model was created, and these methods were executed against the databases to measure performance.

There was acceptable representation of the data from 10 observational databases in the OMOP CDM using the standardized terminologies selected, and a range of analytic methods was developed and executed with sufficient performance to be useful for active safety surveillance.

A prospective observational study.

We collected 4023 medication orders in a hospital for independent evaluation in two pharmacotherapy-related decision support systems. Only the more advanced system considered patient characteristics and laboratory test results in its algorithms. Two pharmacists assessed the clinical relevance of the medication alerts produced. The alert was considered relevant if the pharmacist would undertake action (eg, contact the physician or the nurse). The primary analysis concerned the positive predictive value (PPV) for clinically relevant medication alerts in both systems.

The PPV was significantly higher in the advanced system (5.8% vs 17.0%; p<0.05). Significant differences were found in the alert categories: drug–(drug) interaction (9.9% vs 14.8%; p<0.05), drug–age interaction (2.9% vs 73.3%; p<0.05), and dosing guidance (5.6% vs 16.9%; p<0.05). Including laboratory values and other patient characteristics resulted in a significantly higher PPV for the advanced CDSS compared to the basic medication alerts (12.2% vs 23.3%; p<0.05).

The advanced CDSS produced a higher proportion of clinically relevant medication alerts, but the number of irrelevant alerts remained high. To improve the PPV of the advanced CDSS, the algorithms should be optimized by identifying additional risk modifiers and more data should be made electronically available to improve the performance of the algorithms. Our study illustrates and corroborates the need for cyclic testing of technical improvements in information technology in circumstances representative of daily clinical practice.

Prospective observational study conducted within two clinical units of the Gastroenterology Department in a 1537-bed tertiary teaching hospital in Madrid (Spain).

Medication errors were measured using the disguised observation technique. Types of medication errors and their potential severity were described. The correlation between potential risk factors and medication errors was studied to identify potential causes.

In total, 2314 medication administrations to 73 patients were observed: 509 errors were recorded (22.0%)—68 (13.4%) in preparation and 441 (86.6%) in administration. The most frequent errors were use of wrong administration techniques (especially concerning food intake (13.9%)), wrong reconstitution/dilution (1.7%), omission (1.4%), and wrong infusion speed (1.2%). Errors were classified as no damage (95.7%), no damage but monitoring required (2.3%), and temporary damage (0.4%). Potential clinical severity could not be assessed in 1.6% of cases. The potential risk factors morning shift, evening shift, Anatomical Therapeutic Chemical medication class antacids, prokinetics, antibiotics and immunosuppressants, oral administration, and intravenous administration were associated with a higher risk of administration errors. No association was found with variables related to understaffing or nurse's experience.

Medication administration errors persist in units with automated prescription and dispensing. We identified a need to improve nurses' working procedures and to implement a Clinical Decision Support tool that generates recommendations about scheduling according to dietary restrictions, preparation of medication before parenteral administration, and adequate infusion rates.

We identified eight clinically significant adverse events. We used the FDA's Adverse Event Reporting System to build profiles for these adverse events based on the side effects of drugs known to produce them. We then looked for pairs of drugs that match these single-drug profiles in order to predict potential interactions. We evaluated these interactions in two independent data sets and also through a retrospective analysis of the Stanford Hospital electronic medical records.

We identified 171 novel drug interactions (for eight adverse event categories) that are significantly enriched for known drug interactions (p=0.0009) and used the electronic medical record for independently testing drug interaction hypotheses using multivariate statistical models with covariates.

Our method provides an option for detecting hidden interactions in spontaneous reporting systems by using side effect profiles to infer the presence of unreported adverse events.

A prospective controlled trial was conducted over 26 weeks. The status of the decision support tool alternated OFF (7/17/06–8/29/06), ON (8/29/06–10/10/06), OFF (10/10/06–11/28/06), and ON (11/28/06–1/16/07) in consecutive blocks during the study period. In patients ≥65 who were ordered certain benzodiazepines, opiates, non-steroidals, or sedative-hypnotics, the computer application either adjusted the dosing or suggested a different medication. Physicians could accept or reject recommendations.

The primary outcome compared medication ordering consistent with recommendations during ON versus OFF periods. Secondary outcomes included the admission rate, emergency department length of stay for discharged patients, 10-fold dosing orders, use of a second drug to reverse the original medication, and rate of ADEs using previously validated explicit chart review.

2398 orders were placed for 1407 patients over 1548 visits. The majority (49/53; 92.5%) of recommendations for alternate medications were declined. More orders were consistent with dosing recommendations during ON (403/1283; 31.4%) than OFF (256/1115; 23%) periods (p≤0.0001). 673 (43%) visits were reviewed for ADEs. The rate of ADEs was lower during ON (8/237; 3.4%) compared with OFF (31/436; 7.1%) periods (p=0.02). The remaining secondary outcomes showed no difference.

Single institution study, retrospective chart review for ADEs.

Though overall agreement with recommendations was low, real-time computerized decision support resulted in greater acceptance of medication recommendations. Fewer ADEs were observed when computerized decision support was active.

The BRIDGE-Wiz application (Building Recommendations In a Developer's Guideline Editor) uses a wizard approach to address the questions: (1) under what circumstances? (2) who? (3) ought (with what level of obligation?) (4) to do what? (5) to whom? (6) how and why? Controlled natural language was applied to create and populate a template for recommendation statements.

The application was used by five national panels to develop guidelines. In general, panelists agreed that the software helped to formalize a process for authoring guideline recommendations and deemed the application usable and useful.

Use of BRIDGE-Wiz promotes clarity of recommendations by limiting verb choices, building active voice recommendations, incorporating decidability and executability checks, and limiting Boolean connectors. It enhances transparency by incorporating systematic appraisal of evidence quality, benefits, and harms. BRIDGE-Wiz promotes implementability by providing a pseudocode rule, suggesting deontic modals, and limiting the use of ‘consider’.

Users found that BRIDGE-Wiz facilitates the development of clear, transparent, and implementable guideline recommendations.

A population-based proband-oriented pedigree information system with ties of consanguinity based on both population-based household registry data and Keelung Community Integrated Screening data was proposed to build a comprehensive extended family pedigree structure to accommodate a series of genetic studies on different diseases. We also developed an algorithm to efficiently assess how well theoretical family relationships affecting the occurrence of diseases across three generations with respect to the relative relationship score, a quantitative indicator of genetic influence, were captured.

We applied this population-based proband-oriented pedigree information system to estimate the rate of hypertension with various relative relationships given the selection of probands. The degree of capturing complete familial relationships was assessed for three generations. The risk for early onset of hypertension was proportional to the proband-oriented relative relationship score with 2% increased risk and 1% correction for incomplete capture.

The population-based proband-oriented pedigree information system is powerful and can support various genetic descriptive and analytic epidemiological studies.

Retrospective case–control chart review of 240 patients during the years 2008 and 2009. Patient demographic and clinical information was extracted from our online medical records, and portal use metrics were provided by Information Systems. Descriptive statistics were utilized to explore characteristics of portal users, how the portal is used, and what associations exist between medical resource utilization and active portal use. Logistic regression identified independent patient predictors and barriers to portal use.

Portal users tended to be young professionals with minimal physical disability. The most frequently used portal feature was secure patient–physician messaging. Message content largely consisted of requests for medications or refills in addition to self-reported side effects. Independent predictors and barriers of portal use include the number of medications prescribed by our staff (OR 1.69, p<0.0001), Caucasian ethnicity (OR 5.04, p=0.007), arm and hand disability (OR 0.23, p=0.01), and impaired vision (OR 0.31, p=0.01).

MS patients use the internet in a greater proportion than the general US population, yet physical disability limits their access. Technological adaptations such as voice-activated commands and easy font-size adjustment may help patients overcome these barriers.

Future research should explore the influence of portal technology on healthcare resource utilization and cost. Additional emedicine applications could be linked to the patient portal for disease monitoring and prospective investigation.

A cross-sectional survey was administered by telephone structured interview to 21 organizations to determine the types of data made available to patients through PHRs and the presence of explicit governance for PHR data release. Organizations were identified based on a review of the literature, PHR experts, and snowball sampling. Organizations that did not provide patients with electronic access to their data via a PHR were excluded.

Interviews were conducted with 17 organizations for a response rate of 81%. Half of the organizations had explicit governance in the form of a written policy that outlined the data types made available to patients. Overall, 88% of the organizations used a committee structure for the decision-making process and included senior management and information services. All organizations sought input from clinicians.

There was considerable variability in the types of clinical data and the time frame for releasing these data to patients. Variability in data release policies may have implications for PHR use and adoption.

Future policy activities, such as requirement specification for the latter stages of Meaningful Use, should be leveraged as an opportunity to encourage standardization of functionality and broad deployment of PHRs.

We conducted a telephone survey of patients in one urban general internal medicine clinic. Patients were eligible if they did not enroll within 30 days of receiving an electronic order inviting participation. Our primary outcomes were: (a) reasons for not enrolling in the patient portal; (b) reasons for not attempting enrollment; and (c) perceived benefits of the portal.

Participants' (N=159) mean age was 51 years, 48% were black, 72% female, and 70% had a college degree or greater. 63% of respondents not enrolling reported never attempting enrollment despite remembering receiving an order. Most of these 63% did not attempt enrollment because of lack of information or motivation. Smaller proportions reported not attempting enrollment because of negative attitudes toward the portal (30%) or computer related obstacles (8%). Overall, respondents favorably viewed most patient portal features, however black respondents were less likely than white respondents to consider features assisting self-management such as getting test results (69% vs 86%; p<0.05) as important. Adjusting for age, gender, education, and chronic disease did not substantially change results.

Strategies to increase enrollment in patient portals need to ensure patients understand patient portal features and receive follow-up reminders. Interventions to reduce racial disparities in enrollment must address attitudinal barriers and not focus solely on improving access.

To gain insight into community-based physicians' goals, environments, tasks, and desired support tools, we used a human–computer interaction model that was based in grounded theory. We conducted 30 recorded interviews with, and 25 observations of, primary care providers within 15 urban and rural community-based clinics across Oregon. Participants were members of three healthcare organizations with different commercial electronic health record systems. We used a grounded theory approach to analyze data and develop a user-centered definition of CDS and themes related to desired CDS functionalities.

Physicians viewed CDS as a set of software tools that provide alerts, prompts, and reference tools, but not tools to support patient management, clinical operations, or workflow, which they would like. They want CDS to enhance physician–patient relationships, redirect work among staff, and provide time-saving tools. Participants were generally dissatisfied with current CDS capabilities and overall electronic health record usability.

Physicians identified different aspects of decision-making in need of support: clinical decision-making such as medication administration and treatment, and cognitive decision-making that enhances relationships and interactions with patients and staff.

Physicians expressed a need for decision support that extended beyond their own current definitions. To meet this requirement, decision support tools must integrate functions that align time and resources in ways that assist providers in a broad range of decisions.

A 43-question anonymous survey distributed electronically to relevant online communities.

Data on user demographic characteristics, modes and purposes of use, means and frequencies of access, satisfaction with SNOMED CT content coverage and quality and with the change request mechanism were recorded.

The survey was conducted in January 2010 and elicited 215 responses. Details regarding users' profiles, modes of use and access were reported elsewhere. The coverage of SNOMED CT was perceived to be at least 85% complete by 42% of responders, and 60% were at least satisfied with its quality. Various deficiencies were encountered at least ‘somewhat often’ by 28–61% of responders. Incorrect data were more bothersome than missing data. Users indicated that significant resources should be allocated to more consistent and complete conceptual representations and to further enhance content coverage. Enhanced synonym coverage and the introduction of textual definitions were important to users (54% and 63%, respectively).

A survey format with limited control over recruitment and selection bias. Lack of information regarding the SNOMED CT version used by responders.

Despite overall satisfaction, direct users indicated a strong desire to improve consistency, quality, and completeness of conceptual representations and concept details, as well as a continued desire to expand coverage. The survey provides much needed data for informed decisions regarding the use and development goals of SNOMED CT. Focused periodical surveys are warranted.

An 18-month, retrospective study of acute care admissions at a single Veterans Affairs medical center using computerized provider order entry (CPOE).

Patients included had at least one warfarin cDDI alert. Chart reviews included baseline laboratory values and demographics, provider actions, patient outcomes, and associated factors, including other interacting medications and number of simultaneously processed alerts.

137 admissions were included (133 unique patients). Amiodarone, vitamin E in a multivitamin, sulfamethoxazole, and levothyroxine accounted for 75% of warfarin cDDI. Provider responses were clinically appropriate in 19.7% of admissions and pharmacist responses were appropriate in 9.5% of admissions. There were 50 ADE (36.6% of admissions) with warfarin; 80% were rated as having no or mild clinical effect. An increased number of non-critical alerts at the time of the reference cDDI alert was the only variable associated with an inappropriate provider response (p=0.01).

This study was limited by being a retrospective review and the possibility of confounding variables, such as other interacting medications.

The large number of CPOE alerts may lead to inappropriate responses by providers and pharmacists. The high rate of ADE suggests a need for improved medication management systems for patients on warfarin. This study highlights the possibility of alert fatigue contributing to the high prevalence of inappropriate alert over-ride text responses.

The survey is grounded in the unified theory of acceptance and use of technology and an adapted accident causation model. Development of the instrument was also informed by a review of the extant literature on prescribers' attitude toward computerized medication safety alerts and common prescriber-provided reasons for overriding. To refine and validate the survey, we conducted a two-stage empirical validation study consisting of a pretest with a panel of domain experts followed by a field test among all eligible prescribers at our institution.

The resulting survey instrument contains 28 questionnaire items assessing six theoretical dimensions: performance expectancy, effort expectancy, social influence, facilitating conditions, perceived fatigue, and perceived use behavior. Satisfactory results were obtained from the field validation; however, a few potential issues were also identified. We analyzed these issues accordingly and the results led to the final survey instrument as well as usage recommendations.

High override rates of computerized medication safety alerts have been a prevalent problem. They are usually caused by, or manifested in, issues of poor end user acceptance. However, standardized research tools for assessing and understanding end users' perception are currently lacking, which inhibits knowledge accumulation and consequently forgoes improvement opportunities. The survey instrument presented in this paper may help fill this methodological gap.

We developed and empirically validated a survey instrument that may be useful for future research on DDI alerts and other types of computerized medication safety alerts more generally.

The authors previously reviewed human-factors principles for relevance to medication-related decision support alerts. In this study, instrument items were developed for assessing the appropriate implementation of these human-factors principles in drug–drug interaction (DDI) alerts. User feedback regarding nine electronic medical records was considered during the development process. Content validity, construct validity through correlation analysis, and inter-rater reliability were assessed.

The final version of the instrument included 26 items associated with nine human-factors principles. Content validation on three systems resulted in the addition of one principle (Corrective Actions) to the instrument and the elimination of eight items. Additionally, the wording of eight items was altered. Correlation analysis suggests a direct relationship between system age and performance of DDI alerts (p=0.0016). Inter-rater reliability indicated substantial agreement between raters (=0.764).

The authors developed and gathered preliminary evidence for the validity of an instrument that measures the appropriate use of human-factors principles in the design and display of DDI alerts. Designers of DDI alerts may use the instrument to improve usability and increase user acceptance of medication alerts, and organizations selecting an electronic medical record may find the instrument helpful in meeting their clinicians' usability needs.

To develop a new methodology that combines existing data mining algorithms with chemical information by analysis of molecular fingerprints to enhance initial ADE signals generated from AERS, and to provide a decision support mechanism to facilitate the identification of novel adverse events.

The method achieved a significant improvement in precision in identifying known ADE, and a more than twofold signal enhancement when applied to the ADE rhabdomyolysis. The simplicity of the method assists in highlighting the etiology of the ADE by identifying structurally similar drugs. A set of drugs with strong evidence from both AERS and molecular fingerprint-based modeling is constructed for further analysis.

The results demonstrate that the proposed methodology could be used as a pharmacovigilance decision support tool to facilitate ADE detection.

A descriptive study using the University of Pennsylvania Health System outpatient EMR to simulate a prospective algorithm to identify exposures during pregnancy to category X medications, soon enough to intervene and potentially prevent the exposure. A subsequent post-hoc algorithm was also tested, working backwards from pregnancy endpoints, to search for possible exposures that should have been detected.

Category X medications prescribed to pregnant patients.

The alert simulation identified 2201 pregnancies with 16 969 pregnancy months (excluding abortions and ectopic pregnancies). Of these, 30 appeared to have an order for a non-hormone category X medication during pregnancy. However, none of the 30 ‘exposed pregnancies’ were confirmed as true exposures in medical records review. The post-hoc algorithm identified 5841 pregnancies with 64 exposed pregnancies in 52 569 risk months, only one of which was a confirmed case.

Category X medications may indeed be used in pregnancy, although rarely. However, most patients identified by the algorithm as exposed in pregnancy were not truly exposed. Therefore, implementing an electronic warning without evaluation would have inconvenienced prescribers, possibly hurting some patients (leading to non-use of needed drugs), with no benefit. These data demonstrate that computerized physician order entry interventions should be selected and evaluated carefully even before their use, using alert simulations such as that performed here, rather than just taken off the shelf and accepted as credible without formal evaluation.

Borrowing concepts from Entity–Attribute–Value and from enterprise data warehousing, the authors designed an alternative called the Dimensional Bus model and used it to integrate electronic medical record, sponsored study, and biorepository data. Each type of observational collection has its own table, and the structure of these tables varies to suit the source data. The observational tables are linked to the Bus, which holds provenance information and links to various classificatory dimensions that amplify the meaning of the data or facilitate its query and exposure management.

The authors implemented a Bus-based clinical research data repository with a query system that flexibly manages data access and confidentiality, facilitates catalog search, and readily formulates and compiles complex queries.

The design provides a workable way to manage and query mixed schemas in a data warehouse.

The authors' experience with informatics for integrating biology and the bedside (i2b2) use cases indicates that the privacy of the patient should be considered on three fronts: technical de-identification of the data, trust in the researcher and the research, and the security of the underlying technical platforms.

The security structure of i2b2 is implemented based on consideration of all three fronts. It has been supported with several use cases across the USA, resulting in five privacy categories of users that serve to protect the data while supporting the use cases.

The i2b2 architecture is designed to provide consistency and faithfully implement these user privacy categories. These privacy categories help reflect the policy of both the Health Insurance Portability and Accountability Act and the provisions of the National Research Act of 1974, as embodied by current institutional review boards.

By implementing a holistic approach to patient privacy solutions, i2b2 is able to help close the gap between principle and practice.

Lagged linear correlation was calculated between seven clinical laboratory values and 30 clinical concepts extracted from resident signout notes from a 22-year, 3-million-patient database of electronic health records. Time points were interpolated, and patients were normalized to reduce inter-patient effects.

The method revealed several types of associations with detailed temporal patterns. Definitional associations included low blood potassium preceding ‘hypokalemia.’ Low potassium preceding the drug spironolactone with high potassium following spironolactone exemplified intentional and physiologic associations, respectively. Counterintuitive results such as the fact that diseases appeared to follow their effects may be due to the workflow of healthcare, in which clinical findings precede the clinician's diagnosis of a disease even though the disease actually preceded the findings. Fully exploiting time by interpolating time points produced less noisy results.

Electronic health records are not direct reflections of the patient state, but rather reflections of the healthcare process and the recording process. With proper techniques and understanding, and with proper incorporation of time, interpretable associations can be derived from a large clinical database.

A large, heterogeneous clinical database can reveal clinical associations, time is an important feature, and care must be taken to interpret the results.

An analysis pipeline called eligibility criteria extraction and representation (EliXR) was developed that integrates syntactic parsing and tree pattern mining to discover common semantic patterns in 1000 eligibility criteria randomly selected from http://ClinicalTrials.gov. The semantic patterns were aggregated and enriched with unified medical language systems semantic knowledge to form a semantic representation for clinical research eligibility criteria.

The authors arrived at 175 semantic patterns, which form 12 semantic role labels connected by their frequent semantic relations in a semantic network.

Three raters independently annotated all the sentence segments (N=396) for 79 test eligibility criteria using the 12 top-level semantic role labels. Eight-six per cent (339) of the sentence segments were unanimously labelled correctly and 13.8% (55) were correctly labelled by two raters. The Fleiss' was 0.88, indicating a nearly perfect interrater agreement.

This study present a semi-automated data-driven approach to developing a semantic network that aligns well with the top-level information structure in clinical research eligibility criteria text and demonstrates the feasibility of using the resulting semantic role labels to generate semistructured eligibility criteria with nearly perfect interrater reliability.

The authors have conducted a multipart model formulation and validation process, focusing on the development of a methodological and technical approach to using CKDD to support hypothesis discovery for in silico science. The model the authors have developed is known as the Translational Ontology-anchored Knowledge Discovery Engine (TOKEn). This model utilizes a specific CKDD approach known as Constructive Induction to identify and prioritize potential hypotheses related to the meaningful semantic relationships between variables found in large-scale and heterogeneous biomedical data sets.

The authors have verified and validated TOKEn in the context of a translational research data repository maintained by the NCI-funded Chronic Lymphocytic Leukemia Research Consortium. Such studies have shown that TOKEn is: (1) computationally tractable; and (2) able to generate valid and potentially useful hypotheses concerning relationships between phenotypic and biomolecular variables in that data collection.

The TOKEn model represents a potentially useful and systematic approach to knowledge synthesis for in silico discovery science in the context of large-scale and multidimensional research data sets.

We have developed a multi-layered knowledge representation framework for structuring guideline recommendations for implementation in a variety of CDS contexts. In this framework, guideline recommendations are increasingly structured through four layers, successively transforming a narrative text recommendation into input for a CDS system. We have used this framework to implement rules for a CDS service based on three guidelines. We also conducted a preliminary evaluation, where we asked CDS experts at four institutions to rate the implementability of six recommendations from the three guidelines.

The experience in using the framework and the preliminary evaluation indicate that this approach has promise in creating structured knowledge, to implement in CDS systems, that is usable across organizations.

Pattern matching rules were manually developed through examining keywords and expression patterns of side effects to discover an individual side effect and causative drug relationship. A combination of machine learning (C4.5) using side effect keyword features and pattern matching rules was used to extract sentences that contain side effect and causative drug pairs, enabling the system to discover most side effect occurrences. Our system was implemented as a module within the clinical Text Analysis and Knowledge Extraction System.

The system was tested in the domain of psychiatry and psychology. The rule-based system extracting side effects and causative drugs produced an F score of 0.80 (0.55 excluding allergy section). The hybrid system identifying side effect sentences had an F score of 0.75 (0.56 excluding allergy section) but covered more side effect and causative drug pairs than individual side effect extraction.

The rule-based system was able to identify most side effects expressed by clear indication words. More sophisticated semantic processing is required to handle complex side effect descriptions in the narrative. We demonstrated that our system can be trained to identify sentences with complex side effect descriptions that can be submitted to a human expert for further abstraction.

Our system was able to extract most physician-asserted drug side effects. It can be used in either an automated mode for side effect extraction or semi-automated mode to identify side effect sentences that can significantly simplify abstraction by a human expert.

Using a set of quality measures published by physician specialty societies, we implemented an NLP engine that extracts 21 variables for 19 quality measures from free-text colonoscopy and pathology reports. We evaluated the performance of the NLP engine on a test set of 453 colonoscopy reports and 226 pathology reports, considering accuracy in extracting the values of the target variables from text, and the reliability of the outcomes of the quality measures as computed from the NLP-extracted information.

The average accuracy of the NLP engine over all variables was 0.89 (range: 0.62–1.0) and the average F measure over all variables was 0.74 (range: 0.49–0.89). The average agreement score, measured as Cohen's , between the manually established and NLP-derived outcomes of the quality measures was 0.62 (range: 0.09–0.86).

For nine of the 19 colonoscopy quality measures, the agreement score was 0.70 or above, which we consider a sufficient score for the NLP-derived outcomes of these measures to be practically useful for quality measurement.

The use of NLP for information extraction from free-text colonoscopy and pathology reports creates opportunities for large scale, routine quality measurement, which can support quality improvement in colonoscopy care.

14 specialty teams (46 doctors) were shadowed by the investigator while on their ward-rounds and 16 prescribers from these teams were interviewed.

Senior doctors were highly influential in prescribing decisions during ward-rounds but rarely used the computerized provider order entry system. Junior doctors entered the majority of medication orders into the system, nearly always ignored computerized alerts and never raised their occurrence with other doctors on ward-rounds. Interviews with doctors revealed that some decision support features were valued but most were not perceived to be useful.

The computerized alerts failed to target the doctors who were making the prescribing decisions on ward-rounds. Senior doctors were the decision makers, yet the junior doctors who used the system received the alerts. As a result, the alert information was generally ignored and not incorporated into the decision-making processes on ward-rounds. The greatest value of decision support in this setting may be in non-ward-round situations where senior doctors are less influential. Identifying how prescribing systems are used during different clinical activities can guide the design of decision support that effectively supports users in different situations. If confirmed, the findings reported here present a specific focus and user group for designers of medication decision support.

An international Delphi study with two quantitative rounds. 69 researchers with expertise in computerized physician order entry (CPOE) systems were asked to estimate the usefulness of 20 possible context factors, and to assess the potential impact of six innovative ways of delivering alert information on adverse drug event (ADE) rates.

Participants identified the following top five context information items (in descending order of usefulness): (1) severity of the effect of the ADE the alert refers to; (2) clinical status of the patient; (3) probability of occurrence of the ADE the alert refers to; (4) risk factors of the patient; and (5) strength of evidence on which the alert is built. The ways of delivering alert information with the highest estimated ADE reduction potential are active alerting, proactive prescription simulation and a patient medication module that gives patient-oriented alert information.

Most participants had a research-oriented focus; therefore the results may not reflect the opinions of CPOE users or CPOE implementers.

The study results may provide CPOE system developers and healthcare institutions with information on how to design more effective alert mechanisms.

This is a retrospective cohort study of 3850 computer-generated prescriptions received by a commercial outpatient pharmacy chain across three states over 4 weeks in 2008. A clinician panel reviewed the prescriptions using a previously described method to identify and classify medication errors. Primary outcomes were the incidence of medication errors; potential adverse drug events, defined as errors with potential for harm; and rate of prescribing errors by error type and by prescribing system.

Of 3850 prescriptions, 452 (11.7%) contained 466 total errors, of which 163 (35.0%) were considered potential adverse drug events. Error rates varied by computerized prescribing system, from 5.1% to 37.5%. The most common error was omitted information (60.7% of all errors).

About one in 10 computer-generated prescriptions included at least one error, of which a third had potential for harm. This is consistent with the literature on manual handwritten prescription error rates. The number, type, and severity of errors varied by computerized prescribing system, suggesting that some systems may be better at preventing errors than others.

Implementing a computerized prescribing system without comprehensive functionality and processes in place to ensure meaningful system use does not decrease medication errors. The authors offer targeted recommendations on improving computerized prescribing systems to prevent errors.

CPOE with duplicate medication order alerts was implemented in a 400-bed Northeastern US community tertiary care teaching hospital. In a pre-implementation post-implementation design, trained nurses used chart review, computer-generated reports of medication orders, provider alerts, and staff reports to identify medication errors in two intensive care units (ICUs).

Medication error data were adjudicated by a physician and a human factors engineer for error stage and type. A qualitative analysis of duplicate medication ordering errors was performed to identify contributing factors.

Data were collected for 4147 patient-days pre-implementation and 4013 patient-days post-implementation. Duplicate medication ordering errors increased after CPOE implementation (pre: 48 errors, 2.6% total; post: 167 errors, 8.1% total; p<0.0001). Most post-implementation duplicate orders were either for the identical order or the same medication. Contributing factors included: (1) provider ordering practices and computer availability, for example, two orders placed within minutes by different providers on rounds; (2) communication and hand-offs, for example, duplicate orders around shift change; (3) CDS and medication database design, for example confusing alert content, high false-positive alert rate, and CDS algorithms missing true duplicates; (4) CPOE data display, for example, difficulty reviewing existing orders; and (5) local CDS design, for example, medications in order sets defaulted as ordered.

Duplicate medication order errors increased with CPOE and CDS implementation. Many work system factors, including the CPOE, CDS, and medication database design, contributed to their occurrence.

A pop-up alert was created for providers when an individual's platelet values had decreased by 50% or to <100 000/mm³ in the setting of recent heparin exposure. The authors retrospectively compared inpatients admitted between January 24, 2008 and August 24, 2008 to a control group admitted 1 year prior to the HIT alert. The primary outcome was a change in HIT antibody testing. Secondary outcomes included an assessment of incidence of HIT antibody positivity, percentage of patients started on a direct thrombin inhibitor (DTI), length of stay and overall mortality.

There were 1006 and 1081 patients in the control and intervention groups, respectively. There was a 33% relative increase in HIT antibody test orders (p=0.01), and 33% more of these tests were ordered the first day after the criteria were met when a pop-up alert was given (p=0.03). Heparin was discontinued in 25% more patients in the alerted group (p=0.01), and more direct thrombin inhibitors were ordered for them (p=0.03). The number who tested HIT antibody-positive did not differ, however, between the two groups (p=0.99). The length of stay and mortality were similar in both groups.

The HIT alert significantly impacted provider behaviors. However, the alert did not result in more cases of HIT being detected or an improvement in overall mortality. Our findings do not support implementation of a computerized HIT alert.

A randomized study of 24 junior doctors each performing 30 simulated prescribing tasks in random order with a prototype e-prescribing system. Using a within-participant design, doctors were randomized to be shown one of three types of e-prescribing alert (modal, non-modal, no alert) during each prescribing task.

The main outcome measure was prescribing error rate. Structured interviews were performed to elicit participants' preferences for the prescribing alerts and their views on clinical decision support systems.

Participants exposed to modal alerts were 11.6 times less likely to make a prescribing error than those not shown an alert (OR 11.56, 95% CI 6.00 to 22.26). Those shown a non-modal alert were 3.2 times less likely to make a prescribing error (OR 3.18, 95% CI 1.91 to 5.30) than those not shown an alert. The error rate with non-modal alerts was 3.6 times higher than with modal alerts (95% CI 1.88 to 7.04).

Both kinds of e-prescribing alerts significantly reduced prescribing error rates, but modal alerts were over three times more effective than non-modal alerts. This study provides new evidence about the relative effects of modal and non-modal alerts on prescribing outcomes.

Adequate IT is required to sufficiently support a multicenter reporting system. The CMR system consists of a website, a database, a web-based reporting form, an application to import reports generated in other reporting systems, an application to generate an overview of reported medication incidents, and a national warning system for healthcare providers.

From the start of CMR 90 of all 93 (96.8%) hospitals and 872 of 1948 (44.8%) community pharmacies participated. Between March 2006 and March 2010 the CMR comprised 15 694 reports of incidents. In the period from March 2010 to March 2011, 1642 reports were submitted by community pharmacies in CMR and the hospitals submitted 2517 reports. CMR is similar to various systems in other countries, but it seems to use more IT applications.

The CMR is developing into a nationwide reporting system of medication incidents in The Netherlands, in which hospitals, community pharmacies, mental healthcare organizations and general practitioners participate.

The architecture of the system met the requirements of a nationwide reporting system across different healthcare providers.

Time-series analysis at a large internal medicine practice using a commercial EHR. All patients eligible for each measure were included (range approximately 100–7500).

The proportion of eligible patients in the practice who satisfied each of 15 quality measures after removing those with exceptions from the denominator. To analyze changes in performance for individual physicians, two composite measures were used: prescribing seven essential medications and completion of five preventive services.

During the year after implementing pre-encounter reminders, performance continued to improve for eight measures, remained stable for four, and declined for three. Physicians with the worst performance at the start of the pre-encounter reminders showed little absolute improvement over the next year, and most remained below the median performance for physicians in the practice.

Paper pre-encounter reminders did not appear to improve performance beyond electronic point-of-care reminders in the EHR alone. Lagging performance is likely not due to providers' EHR workflow alone, and trying to step backwards and use paper reminders in addition to point-of-care reminders in the EHR may not be an effective strategy for engaging slow adopters.

The authors analyzed data from two cross-sectional survey questionnaires distributed to nurses working in four ICUs at a northeastern US regional medical center, 3 months and 12 months after EHR implementation.

Survey items were drawn from established instruments used to measure EHR acceptance and usability, and the usefulness of three EHR functionalities, specifically computerized provider order entry (CPOE), the electronic medication administration record (eMAR), and a nursing documentation flowsheet.

On average, ICU nurses were more accepting of the EHR at 12 months as compared to 3 months. They also perceived the EHR as being more usable and both CPOE and eMAR as being more useful. Multivariate hierarchical modeling indicated that EHR usability and CPOE usefulness predicted EHR acceptance at both 3 and 12 months. At 3 months postimplementation, eMAR usefulness predicted EHR acceptance, but its effect disappeared at 12 months. Nursing flowsheet usefulness predicted EHR acceptance but only at 12 months.

As the push toward implementation of EHR technology continues, more hospitals will face issues related to acceptance of EHR technology by staff caring for critically ill patients. This research suggests that factors related to technology design have strong effects on acceptance, even 1 year following the EHR implementation.

The approach, referred to as partnership model, was developed and iteratively refined through the research team's previous work on implementing health information technology (HIT) in over 30 safety net practices. This paper uses two case studies to illustrate how the model was applied to help two nurse-managed health centers (NMHC), a particularly vulnerable primary care setting, implement EHR and get prepared to meet the meaningful use criteria.

The strong focus of the model on continuous quality improvement led to eventual implementation success at both sites, despite difficulties encountered during the initial stages of the project.

There has been a lack of research, particularly in resource-limited primary care settings, on strategies for abating provider anxiety and preparing them to manage complex changes associated with EHR uptake. The partnership model described in this paper may provide useful insights into the work shepherded by HIT regional extension centers dedicated to supporting resource-limited communities disproportionally affected by EHR adoption barriers.

NMHC, similar to other primary care settings, are often poorly resourced, understaffed, and lack the necessary expertise to deploy EHR and integrate its use into their day-to-day practice. This study demonstrates that implementation of EHR, a prerequisite to meaningful use, can be successfully achieved in this setting, and partnership efforts extending far beyond the initial software deployment stage may be the key.

A mixed-method three-group comparative design was used. The PRECEDE/PROCEED implementation model was used to qualitatively identify key factors in the implementation process. These results were further explored in a quantitative analysis.

Thirty-three primary care clinics at three institutions (Intermountain Healthcare, VA Salt Lake City Health Care System, and University of Utah) participated. The program consisted of an onsite, didactic session, QI planning and 6 months of intense implementation support.

Completion rate was 82% with an average improvement rate of 21%. Important predisposing factors for success included an established electronic record and a culture of quality. The reinforcing and enabling factors included free continuing medical education credits, feedback, IT access, and flexible support. The relationship between IT and QI emerged as a central factor. Quantitative analysis found significant differences between institutions for pre–post changes even after the number and category of implementation strategies had been controlled for.

The analysis illustrates the complex dependence between IT interventions, institutional characteristics, and implementation practices. Access to IT tools and data by individual clinicians may be a key factor for the success of QI projects. Institutions vary widely in the degree of access to IT tools and support. This article suggests that more attention be paid to the QI and IT department relationship.

eCancerCare^Bladder and the EMR were used to retrieve data and produce clinical reports. Twelve participants (four attending staff, four fellows, and four residents) used either eCancerCare^Bladder or the EMR in two clinical scenarios simulating cystoscopy surveillance visits for bladder cancer follow-up.

Time to retrieve and quality of review of the patient history; time to produce and completeness of a cystoscopy report. Finally, participants provided a global assessment of their computer literacy, familiarity with the two systems, and system preference.

eCancerCare^Bladder was faster for data retrieval (scenario 1: 146 s vs 245 s, p=0.019; scenario 2: 306 vs 415 s, NS), but non-significantly slower to generate a clinical report. The quality of the report was better in the eCancerCare^Bladder system (scenario 1: p<0.001; scenario 2: p=0.11). User satisfaction was higher with the eCancerCare^Bladder system, and 11/12 participants preferred to use this system.

The small sample size affected the power of our study to detect differences.

Use of a specific data management tool does not appear to significantly reduce user time, but the results suggest improvement in the level of care and documentation and preference by users. Also, the use of simulated scenarios in a laboratory setting appears to be a valid method for comparing the usability of clinical software.

WIISARD combined advanced networking technology with electronic triage tags that reported victims' position and recorded medical information, with wireless pulse-oximeters that monitored patient vital signs, and a wireless electronic medical record (EMR) for disaster care. The EMR system included WiFi handheld devices with barcode scanners (used by front-line responders) and computer tablets with role-tailored software (used by managers of the triage, treatment, transport and medical communications teams). An additional software system provided situational awareness for the incident commander. The WIISARD system was evaluated in a large-scale simulation exercise designed for training first responders. A randomized trial was overlaid on this exercise with 100 simulated victims, 50 in a control pathway (paper-based), and 50 in completely electronic WIISARD pathway. All patients in the electronic pathway were cared for within the WIISARD system without paper-based workarounds.

WIISARD reduced the rate of the missing and/or duplicated patient identifiers (0% vs 47%, p<0.001). The total time of the field was nearly identical (38:20 vs 38:23, IQR 26:53–1:05:32 vs 18:55–57:22).

Overall, the results of WIISARD show that wireless EMR systems for care of the victims of disasters would be complex to develop but potentially feasible to build and deploy, and likely to improve the quality of information available for the delivery of care during disasters.

We analyzed patient visits in one community in which a high proportion of providers were using an electronic health record and participating in data exchange. Using claims data from one large private payer for individuals under age 65 years, we computed the number of visits to a provider which involved transitions in care from other providers as a percentage of total visits. We calculated this ‘transition percentage’ for individual providers and medical groups.

On average, excluding radiology and pathology, approximately 51% of visits involved care transitions between individual providers in the community and 36%–41% involved transitions between medical groups. There was substantial variation in transition percentage across medical specialties, within specialties and across medical groups. Specialists tended to have higher transition percentages and smaller ranges within specialty than primary care physicians, who ranged from 32% to 95% (including transitions involving radiology and pathology). The transition percentages of pediatric practices were similar to those of adult primary care, except that many transitions occurred among pediatric physicians within a single medical group.

Care transition patterns differed substantially by type of practice and should be considered in designing incentives to foster providers' meaningful use of health data exchange services.

To develop and validate methods of automatically inferring patient problems from clinical and billing data, and to provide a knowledge base for inferring problems.

We identified 17 target conditions and designed and validated a set of rules for identifying patient problems based on medications, laboratory results, billing codes, and vital signs. A panel of physicians provided input on a preliminary set of rules. Based on this input, we tested candidate rules on a sample of 100 000 patient records to assess their performance compared to gold standard manual chart review. The physician panel selected a final rule for each condition, which was validated on an independent sample of 100 000 records to assess its accuracy.

Seventeen rules were developed for inferring patient problems. Analysis using a validation set of 100 000 randomly selected patients showed high sensitivity (range: 62.8–100.0%) and positive predictive value (range: 79.8–99.6%) for most rules. Overall, the inference rules performed better than using either the problem list or billing data alone.

We developed and validated a set of rules for inferring patient problems. These rules have a variety of applications, including clinical decision support, care improvement, augmentation of the problem list, and identification of patients for research cohorts.

We evaluated recommendations from Malawi's ART guidelines using GuideLine Implementability Appraisal criteria. Recommendations that passed selected criteria were converted into ratio-based performance measures. We queried representative EMR data to determine the feasibility of generating feedback for each performance measure, summed clinical encounters representing each performance measure's denominator, and then measured the distribution of encounter frequency for individual HCWs across nurse and clinical officer groups.

We analyzed 423 831 encounters in the EMR data and generated automated feedback for 21 recommendations (12%) from Malawi's ART guidelines. We identified 11 nurse recommendations and eight clinical officer recommendations. Individual nurses and clinical officers had an average of 45 and 59 encounters per month, per recommendation, respectively. Another 37 recommendations (21%) would support audit and feedback if additional routine EMR data are captured and temporal constraints are modeled.

It appears feasible to implement automated guideline adherence feedback that could potentially improve HCW performance and supervision. Feedback reports may support workplace learning by increasing HCWs' opportunities to reflect on their performance.

A moderate number of recommendations from Malawi's ART guidelines can be used to generate automated guideline adherence feedback using existing EMR data. Further study is needed to determine the receptivity of HCWs to peer comparison feedback and barriers to implementation of automated audit and feedback in low-resource settings.

A combination of machine learning (conditional random field and maximum entropy) and rule-based (pattern matching) techniques was used to detect negation, speculation, and hypothetical and conditional information, as well as information associated with persons other than the patient.

The best submission obtained an overall micro-averaged F-score of 0.9343.

Using semantic attributes of concepts and information about document structure as features for statistical classification of assertions is a good way to leverage rule-based and statistical techniques. In this task, the choice of features may be more important than the choice of classifier algorithm.

The authors used two machine-learning (ML) classifiers: support vector machines (SVMs) and conditional random fields (CRFs). Because SVMs and CRFs can operate on a large set of features extracted from both clinical texts and external resources, the authors address the following research question: Which features need to be selected for obtaining optimal results? To this end, the authors devise feature-selection techniques which greatly reduce the amount of manual experimentation and improve performance.

The authors evaluated their approaches on the 2010 i2b2/VA challenge data. Concept extraction achieves 79.59 micro F-measure. Assertion classification achieves 93.94 micro F-measure.

Approaching medical concept extraction and assertion classification through ML-based techniques has the advantage of easily adapting to new data sets and new medical informatics tasks. However, ML-based techniques perform best when optimal features are selected. By devising promising feature-selection techniques, the authors obtain results that outperform the current state of the art.

This paper presents two ML-based approaches for processing language in the clinical texts evaluated in the 2010 i2b2/VA challenge. By using novel feature-selection methods, the techniques presented in this paper are unique among the i2b2 participants.

A pipeline system was developed for clinical natural language processing that includes a proofreading process, with gold-standard reflexive validation and correction. The information extraction system is a combination of a machine learning approach and a rule-based approach. The outputs of this system are used for evaluation in all three tiers of the fourth i2b2/VA shared-task and workshop challenge.

Overall concept classification attained an F-score of 83.3% against a baseline of 77.0%, the optimal F-score for assertions about the concepts was 92.4% and relation classifier attained 72.6% for relationships between clinical concepts against a baseline of 71.0%. Micro-average results for the challenge test set were 81.79%, 91.90% and 70.18%, respectively.

The challenge in the multi-task test requires a distribution of time and work load for each individual task so that the overall performance evaluation on all three tasks would be more informative rather than treating each task assessment as independent. The simplicity of the model developed in this work should be contrasted with the very large feature space of other participants in the challenge who only achieved slightly better performance. There is a need to charge a penalty against the complexity of a model as defined in message minimalisation theory when comparing results.

A complete pipeline system for constructing language processing models that can be used to process multiple practical detection tasks of language structures of clinical records is presented.

We used BioTagger-GM to train machine learning taggers, which we originally developed for the detection of gene/protein names in the biology domain. Trained taggers were evaluated using the annotated clinical documents made available in the 2010 i2b2/VA Challenge workshop, consisting of documents from four data sources.

As expected, performance of a tagger trained on one data source degraded when evaluated on another source, but the degradation of the performance varied depending on data sources. A tagger trained on multiple data sources was robust, and it achieved an F score as high as 0.890 on one data source. The results also suggest that performance of machine learning taggers is likely to improve if more annotated documents are available for training.

Our study shows how the performance of machine learning taggers is degraded when they are ported across clinical documents from different sources. The portability of taggers can be enhanced by training on datasets from multiple sources. The study also shows that BioTagger-GM can be easily extended to detect clinical concept mentions with good performance.

The authors'approaches rely on both rule-based and machine-learning methods. Natural language processing is used to extract features from the input texts; these features are then used in the authors' machine-learning approaches. The authors used Conditional Random Fields for concept extraction, and Support Vector Machines for assertion and relation annotation. Depending on the task, the authors tested various combinations of rule-based and machine-learning methods.

The authors'assertion annotation system obtained an F-measure of 0.931, ranking fifth out of 21 participants at the i2b2/VA 2010 challenge. The authors' relation annotation system ranked third out of 16 participants with a 0.709 F-measure. The 0.773 F-measure the authors obtained on concept extraction did not make it to the top 10.

On the one hand, the authors confirm that the use of only machine-learning methods is highly dependent on the annotated training data, and thus obtained better results for well-represented classes. On the other hand, the use of only a rule-based method was not sufficient to deal with new types of data. Finally, the use of hybrid approaches combining machine-learning and rule-based approaches yielded higher scores.

A single support vector machine classifier was used to identify relations between concepts and to assign their semantic type. Several resources such as Wikipedia, WordNet, General Inquirer, and a relation similarity metric inform the classifier.

The techniques reported in this paper were evaluated in the 2010 i2b2 Challenge and obtained the highest F1 score for the relation extraction task. When gold standard data for concepts and assertions were available, F1 was 73.7, precision was 72.0, and recall was 75.3. F1 is defined as 2*Precision*Recall/(Precision+Recall). Alternatively, when concepts and assertions were discovered automatically, F1 was 48.4, precision was 57.6, and recall was 41.7.

Although a rich set of features was developed for the classifiers presented in this paper, little knowledge mining was performed from medical ontologies such as those found in UMLS. Future studies should incorporate features extracted from such knowledge sources, which we expect to further improve the results. Moreover, each relation discovery was treated independently. Joint classification of relations may further improve the quality of results. Also, joint learning of the discovery of concepts, assertions, and relations may also improve the results of automatic relation extraction.

Lexical and contextual features proved to be very important in relation extraction from medical texts. When they are not available to the classifier, the F1 score decreases by 3.7%. In addition, features based on similarity contribute to a decrease of 1.1% when they are not available.

The authors implemented a machine-learning-based named entity recognition system for clinical text and systematically evaluated the contributions of different types of features and ML algorithms, using a training corpus of 349 annotated notes. Based on the results from training data, the authors developed a novel hybrid clinical entity extraction system, which integrated heuristic rule-based modules with the ML-base named entity recognition module. The authors applied the hybrid system to the concept extraction and assertion classification tasks in the challenge and evaluated its performance using a test data set with 477 annotated notes.

Standard measures including precision, recall, and F-measure were calculated using the evaluation script provided by the Center of Informatics for Integrating Biology and the Bedside/VA challenge organizers. The overall performance for all three types of clinical entities and all six types of assertions across 477 annotated notes were considered as the primary metric in the challenge.

Systematic evaluation on the training set showed that Conditional Random Fields outperformed Support Vector Machines, and semantic information from existing natural-language-processing systems largely improved performance, although contributions from different types of features varied. The authors' hybrid entity extraction system achieved a maximum overall F-score of 0.8391 for concept extraction (ranked second) and 0.9313 for assertion classification (ranked fourth, but not statistically different than the first three systems) on the test data set in the challenge.

A ‘learn by example’ approach combines features derived from open-source NLP pipelines with open-source machine learning classifiers to automatically and iteratively evaluate top-performing configurations. The Fourth i2b2/VA Shared Task Challenge's concept extraction task provided the data sets and metrics used to evaluate performance.

Top F-measure scores for each of the tasks were medical problems (0.83), treatments (0.82), and tests (0.83). Recall lagged precision in all experiments. Precision was near or above 0.90 in all tasks.

With no customization for the tasks and less than 5 min of end-user time to configure and launch each experiment, the average F-measure was 0.83, one point behind the mean F-measure of the 22 entrants in the competition. Strong precision scores indicate the potential of applying the approach for more specific clinical information extraction tasks. There was not one best configuration, supporting an iterative approach to model creation.

Acceptable levels of performance can be achieved using fully automated and generalizable approaches to concept-level information extraction. The described implementation and related documentation is available for download.

The authors developed extensions to the clinical Text Analysis and Knowledge Extraction System (cTAKES) that simplify feature extraction, experimentation with various feature representations, and the development of both rule and machine-learning based document classifiers. The authors describe and evaluate their system, the Yale cTAKES Extensions (YTEX), on the classification of radiology reports that contain findings suggestive of hepatic decompensation.

The F₁-Score of the system for the retrieval of abdominal radiology reports was 96%, and was 79%, 91%, and 95% for the presence of liver masses, ascites, and varices, respectively. The authors released YTEX as open source, available at http://code.google.com/p/ytex.

The authors evaluated two well-known ASR systems on spoken clinical questions: Nuance Dragon (both generic and medical versions: Nuance Gen and Nuance Med) and the SRI Decipher (the generic version SRI Gen). The authors also explored language model adaptation using more than 4000 clinical questions to improve the SRI system's performance, and profile training to improve the performance of the Nuance Med system. The authors reported the results with the NIST standard word error rate (WER) and further analyzed error patterns at the semantic level.

Nuance Gen and Med systems resulted in a WER of 68.1% and 67.4% respectively. The SRI Gen system performed better, attaining a WER of 41.5%. After domain adaptation with a language model, the performance of the SRI system improved 36% to a final WER of 26.7%.

Without modification, two well-known ASR systems do not perform well in interpreting spoken clinical questions. With a simple domain adaptation, one of the ASR systems improved significantly on the clinical question task, indicating the importance of developing domain/genre-specific ASR systems.

We selected 6034 VAERS reports for H1N1 vaccine that were classified by medical officers as potentially positive (N_pos=237) or negative for anaphylaxis. We created a categorized corpus of text files that included the class label and the symptom text field of each report. A validation set of 1100 labeled text files was also used. Text mining techniques were applied to extract three feature sets for important keywords, low- and high-level patterns. A rule-based classifier processed the high-level feature representation, while several machine learning classifiers were trained for the remaining two feature representations.

Classifiers' performance was evaluated by macro-averaging recall, precision, and F-measure, and Friedman's test; misclassification error rate analysis was also performed.

Rule-based classifier, boosted trees, and weighted support vector machines performed well in terms of macro-recall, however at the expense of a higher mean misclassification error rate. The rule-based classifier performed very well in terms of average sensitivity and specificity (79.05% and 94.80%, respectively).

Our validated results showed the possibility of developing effective medical text classifiers for VAERS reports by combining text mining with informative feature selection; this strategy has the potential to reduce reviewer workload considerably.

We developed and tested a near-real time computerized alert system, the septic shock sniffer, based on established severe sepsis/septic shock diagnostic criteria. A sniffer scanned patients' data in the electronic medical records and notified the research coordinator on call through an institutional paging system of potentially eligible patients.

The performance of the septic shock sniffer was assessed.

The septic shock sniffer performed well with a positive predictive value of 34%. Electronic screening doubled enrollment, with 68 of 4460 ICU admissions enrolled during the 9 months after implementation versus 37 of 4149 ICU admissions before sniffer implementation (p<0.05). Efficiency was limited by study coordinator availability (not available at nights or weekends).

Automated electronic medical records screening improves the efficiency of enrollment and should be a routine tool for the recruitment of patients into time sensitive clinical trials in the ICU setting.

We parsed the Sig strings from a sample of 20 161 de-identified ambulatory e-prescriptions into variables representing the fields of the Structured and Codified Sig Format. A stratified random sample of these representations was then reviewed by a group of experts. For codified Sig fields, we attempted to map the actual words used by prescribers to the equivalent terms in the designated terminology.

Proportion of prescriptions that the Format could fully represent; proportion of terms used that could be mapped to the designated terminology.

The fields defined in the Format could fully represent 95% of Sigs (95% CI 93% to 97%), but ambiguities were identified, particularly in representing multiple-step instructions. The terms used by prescribers could be codified for only 60% of dose delivery methods, 84% of dose forms, 82% of vehicles, 95% of routes, 70% of sites, 33% of administration timings, and 93% of indications.

The findings are based on a retrospective sample of ambulatory prescriptions derived mostly from primary care physicians.

The fields defined in the Format could represent most of the patient instructions in a large prescription sample, but prior to its mandatory adoption, further work is needed to ensure that potential ambiguities are addressed and that a complete set of terms is available for the codified fields.

Forty practicing physicians, presented with standardized treatment and diagnosis questions and one question of their choosing, entered search terms which were processed in a random, blinded fashion through PubMed alone and PubMed Clinical Queries. Participants rated search retrievals for applicability to the question at hand and satisfaction.

For treatment, the primary outcome of retrieval of relevant articles was not significantly different between the groups, but a higher proportion of articles from the Clinical Queries searches met methodologic criteria (p=0.049), and more articles were published in core internal medicine journals (p=0.056). For diagnosis, the filtered results returned more relevant articles (p=0.031) and fewer irrelevant articles (overall retrieval less, p=0.023); participants needed to screen fewer articles before arriving at the first relevant citation (p<0.05). Relevance was also influenced by content terms used by participants in searching. Participants varied greatly in their search performance.

Clinical Queries filtered searches returned more high-quality studies, though the retrieval of relevant articles was only statistically different between the groups for diagnosis questions.

Retrieving clinically important research studies from Medline is a challenging task for physicians. Methodological search filters can improve search retrieval.

Unlike previous methods for automatic MeSH term assignment, we reformulate the indexing task as a ranking problem such that relevant MeSH headings are ranked higher than those irrelevant ones. Specifically, for each document we retrieve 20 neighbor documents, obtain a list of MeSH main headings from neighbors, and rank the MeSH main headings using ListNet–a learning-to-rank algorithm. We trained our algorithm on 200 documents and tested on a previously used benchmark set of 200 documents and a larger dataset of 1000 documents.

Tested on the benchmark dataset, our method achieved a precision of 0.390, recall of 0.712, and mean average precision (MAP) of 0.626. In comparison to the state of the art, we observe statistically significant improvements as large as 39% in MAP (p-value <0.001). Similar significant improvements were also obtained on the larger document set.

Experimental results show that our approach makes the most accurate MeSH predictions to date, which suggests its great potential in making a practical impact on MeSH indexing. Furthermore, as discussed the proposed learning framework is robust and can be adapted to many other similar tasks beyond MeSH indexing in the biomedical domain. All data sets are available at: http://www.ncbi.nlm.nih.gov/CBBresearch/Lu/indexing.

Since 2007, a collaborative effort to achieve clinical data sharing has been carried out at Zhejiang University in China and Kyoto University and Miyazaki University in Japan; each is running a regional clinical information center.

An international layer system named Global Dolphin was constructed with several key services, sharing patients' health information between countries using a medical markup language (MML). The system was piloted with 39 test patients.

The three regions above have records for 966 000 unique patients, which are available through Global Dolphin. Data exchanged successfully from Japan to China for the 39 study patients include 1001 MML files and 152 images. The MML files contained 197 free text-type paragraphs that needed human translation.

The pilot test in Global Dolphin demonstrates that patient information can be shared across countries through international health data exchange. To achieve cross-border sharing of clinical data, some key issues had to be addressed: establishment of a super directory service across countries; data transformation; and unique one—language translation. Privacy protection was also taken into account. The system is now ready for live use.

The project demonstrates a means of achieving worldwide accessibility of medical data, by which the integrity and continuity of patients' health information can be maintained.

We used a mixed-method analysis, consisting of cross-sectional audit log data, semi-structured interviews, and direct observation in a sample of emergency departments and ambulatory safety net clinics actively using HIE. For each site, we measured overall usage trends, user logon statistics, and data types accessed by users. We also assessed reasons for use and outcomes of use.

Overall, users accessed HIE for 6.8% of all encounters, with higher rates of access for repeat visits, for patients with comorbidities, for patients known to have data in the exchange, and at sites providing HIE access to both nurses and physicians. Discharge summaries and test reports were the most frequently accessed data in the exchange. Providers consistently noted retrieving additional history, preventing repeat tests, comparing new results to retrieved results, and avoiding hospitalizations as a consequence of HIE access.

HIE use in emergency departments and ambulatory clinics was focused on patients where missing information was believed to be present in the exchange and was related to factors including the roles of people with access, the setting, and other site-specific issues that impacted the overall breadth of routine system use. These data should form an important foundation as other sites embark upon HIE implementation.

Using neurosurgery inpatient visit data from EUH and GMH, unique patients who visited both hospitals were identified through classification by name and age at time of visit. The frequency of flow patterns, including time between visits, and the statistical significance of crossover rates for patients with particular diagnoses were determined.

The time between visits, flow patterns, and proportion of patients exhibiting crossover behavior were calculated for the total population studied as well as subpopulations.

5.25% of patients having multiple visits over the study period visited the neurosurgical departments at both hospitals. 77% of crossover patients visited the level 1 trauma center (GMH) before visiting EUH.

The true patient crossover may be under-estimated because the study population only consists of neurosurgical inpatients at EUH and GMH.

We demonstrate that detailed analysis of crossover behavior provides a deeper understanding of the potential value of HIE.

We analyzed the commonalities and divergences of empirical studies published 1990–2010 that have applied the T&M approach to examine the impact of health IT implementation on clinical work processes and workflow. The analysis led to the development of a suggested ‘checklist’ intended to help future T&M research produce compatible and comparable results. We call this checklist STAMP (Suggested Time And Motion Procedures).

STAMP outlines a minimum set of 29 data/ information elements organized into eight key areas, plus three supplemental elements contained in an ‘Ancillary Data’ area, that researchers may consider collecting and reporting in their future T&M endeavors.

T&M is generally regarded as the most reliable approach for assessing the impact of health IT implementation on clinical work. However, there exist considerable inconsistencies in how previous T&M studies were conducted and/or how their results were reported, many of which do not seem necessary yet can have a significant impact on quality of research and generalisability of results. Therefore, we deem it is time to call for standards that can help improve the consistency of T&M research in health informatics. This study represents an initial attempt.

We developed a suggested checklist to improve the methodological and results reporting consistency of T&M research, so that meaningful insights can be derived from across-study synthesis and health informatics, as a field, will be able to accumulate knowledge from these studies.

We conducted a cross-sectional survey of individuals given access to the HIE at participating organizations and examined some of the usability and usage factors related to the technology acceptance model.

We probed user perceptions using the Questionnaire for User Interaction Satisfaction, an author-generated Trust scale, and user characteristic questions (eg, age, weekly system usage time).

Overall, users viewed the system favorably (ratings for all usability items were greater than neutral (one-sample Wilcoxon test, p<0.0014, Bonferroni-corrected for 35 tests). System usage was regressed on usability, trust, and demographic and user characteristic factors. Three usability factors were positively predictive of system usage: overall reactions (p<0 0.01), learning (p<0.05), and system functionality (p<0.01). Although trust is an important component in collaborative relationships, we did not find that user trust of other participating healthcare entities was significantly predictive of usage. An analysis of respondents' comments revealed ways to improve the HIE.

We used a rapid deployment model to develop an HIE and found that perceptions of system usability were positive. We also found that system usage was predicted well by some aspects of usability. Results from this study suggest that a rapid development approach may serve as a viable model for developing usable HIEs serving communities with limited resources.