On January 1, 2005, all specialty clinic providers at the Israeli Defense Forces were divided into one of four financial classes based on their charges, class 1, the least expensive, being the most preferred, followed by classes 2–4. This list was incorporated into the EHRs used by all PCPs in primary care clinics. PCPs received comprehensive training. Target referral goals were determined for each class and measured for 4 years, together with the total cost of all specialist visits in the first year compared to the following years. Quality assessment (QA) scores were used as a measure of the program's effect on the quality of patient care.

During 2005–2008, a marginally significant decline in referrals to class 1 was observed (r=–0.254, p=0.078), however a significant increase in referral rates to class 2 was observed (r=0.957, p=0.042), concurrent with a decrease in referral rates to classes 3 and 4 (r=–0.312, p=0.024). An inverse correlation was observed between year and total costs for all visits to specialists (2008 prices; r=–0.96, p=0.04), and between the mean cost of one specialist visit over the 4 years, indicating a significant reduction in real costs (2008 prices; r=–0.995, p=0.005). QA was not affected by these changes (r=0.94, p=0.016).

From a policy perspective, our data suggest that EHR can facilitate effective utilization of healthcare providers and decrease costs.

Using cross-sectional survey data from the National Study of Small and Medium-Sized Physician Practices (July 2007–March 2009), we examined the extent to which organizational capabilities and external incentives were associated with the adoption of five key HIT functionalities by physician practices and with use of those functionalities by individual physicians.

The rate of physician practices adopting any of the five HIT functionalities was 34.1%. When practices adopted HIT functionalities, on average, about one in seven physicians did not use those functionalities. One physician in five did not use prompts and reminders following adoption by their practice. After controlling for other factors, both adoption of HIT by practices and use of HIT by individual physicians were higher in primary care practices and larger practices. Practices reporting an emphasis on patient-centered management were not more likely than others to adopt, but their physicians were more likely to use HIT.

Larger practices were most likely to have adopted HIT, but other factors, including specialty mix and self-reported patient-centered management, had a stronger influence on the use of HIT once adopted.

Adoption of HIT by practices does not mean that physicians will use the HIT.

A cross-sectional retrospective study using nationally representative data from the National Ambulatory Medical Care Survey for the period 2003–2010 was performed. Bivariate and multivariate analyzes compared EMR use among physicians of 14 specialties and assessed variation by practice setting. Differences in EMR use by geographic region, patient characteristics, and physician office settings were also assessed.

Bivariate and multivariate analysis demonstrated increased EMR use from 2003 to 2010, with 16% reporting at least partial use in 2003, rising to 52% in 2010 (p<0.001). Cardiologists, orthopedic surgeons, urologists, and family/general practitioners had higher frequencies of EMR use whereas psychiatrists, ophthalmologists, and dermatologists had the lowest EMR use. Employed physicians had higher EMR uptake than physicians who owned their practice (48% vs 31%, p<0.001). EMR uptake was lower among solo practitioners (23%) than non-solo practitioners (42%, p<0.001). Practices owned by Health Maintenance Organizations had higher frequencies of EMR use (83%) than practices owned by physicians, community health centers, or academic centers (all <45%, p<0.001). Patient demographics did not affect EMR use (p>0.05).

Uptake of EMR is increasing, although it is significantly slower in dermatology, ophthalmology, and psychiatry. Solo practitioners and owners of a practice have low frequencies of EMR use compared with non-solo practitioners and those who do not own their practice. Despite incentives for EMR adoption, physicians should carefully weigh which, if any, EMR to adopt in their practices.

To carry out a survey of the experience of prescribers in the nation's first study of EPCS implementation.

Prescribers were surveyed in a community setting before and after implementation of EPCS, to assess adoption, attitudes, and challenges.

Of the 102 prescribers enabled to use EPCS and who responded to surveys before and after implementation, 70 had sent at least one controlled substance prescription electronically. Most users reported that EPCS was significantly less burdensome than expected. Over half reported that EPCS was easy to use and improved work flow, accuracy of prescriptions (69.5%), monitoring of medications (59.3%), and coordination with pharmacists, though high prior expectations for improved efficiency were not met. EPCS users reported a significant decrease in the perceived frequency of medication errors and drug diversion, compared with controls. Barriers to use of EPCS included limited pharmacy participation and instances of unreliability of the technology.

Interest in adoption of EPCS is considerable among providers, pharmacies, and vendors. The results suggest that while most EPCS security features may be more acceptable to providers than expected, barriers such as the limited participation by pharmacies may also partly explain slow adoption rates for EPCS nationally.

EPCS was a better experience for many providers than they had expected, but related improvements in practice efficiency and quality of care will depend upon implementation strategies.

We conducted a mixed methods cross-sectional case study of 16 physicians at an academic-affiliated ambulatory clinic from April to June 2010. We utilized standardized prescription and chart review to identify errors. Fourteen providers also participated in interviews.

We analyzed 1905 prescriptions. The overall prescribing error rate was 3.8 per 100 prescriptions (95% CI 2.8 to 5.1). Error rates were significantly lower 2 years after transition (p<0.001 compared to pre-implementation, 12 weeks and 1 year after transition). Rates of near misses remained unchanged. Providers positively appreciated most system refinements, particularly reduced alert firing.

Our study suggests that over time and with system refinements, use of a commercial EHR with advanced CDS can lead to low prescribing error rates, although more serious errors may require targeted interventions to eliminate them. Reducing alert firing frequency appears particularly important. Our results provide support for federal efforts promoting meaningful use of EHRs.

Ongoing error monitoring can allow CDS to be optimally tailored and help achieve maximal safety benefits.

ClinicalTrials.gov, Identifier: NCT00603070.

To examine both paper- and computer-based workarounds to the use of EHR systems in three benchmark institutions.

Qualitative data were collected in 11 primary care outpatient clinics across three healthcare institutions. Data collection methods included direct observation and opportunistic questions. In total, 120 clinic staff and providers and 118 patients were observed. All data were analyzed using previously developed workaround categories and examined for potential new categories. Additionally, workarounds were coded as either paper- or computer-based.

Findings corresponded to 10 of 11 workaround categories identified in previous research. All 10 of these categories applied to paper-based workarounds; five categories also applied to computer-based workarounds. One new category, no correct path (eg, a desired option did not exist in the computer interface, precipitating a workaround), was identified for computer-based workarounds. The most consistent reasons for workarounds across the three institutions were efficiency, memory, and awareness.

Consistent workarounds across institutions suggest common challenges in outpatient clinical settings and failures to accommodate these challenges in EHR design. An examination of workarounds provides insight into how providers adapt to limiting EHR systems. Part of the design process for computer interfaces should include user-centered methods particular to providers and healthcare settings to ensure uptake and usability.

We used video to observe 163 consultations by 16 clinicians using four EPR brands. We made a visual study of the consultation room and coded interactions between clinician, patient, and computer. Few patients (6.9%, n=12) declined to participate.

Patients looked at the computer twice as much (47.6 s vs 20.6 s, p<0.001) when it was within their gaze. A quarter of consultations were interrupted (27.6%, n=45); and in half the clinician left the room (12.3%, n=20). The core consultation takes about 87% of the total session time; 5% of time is spent pre-consultation, reading the record and calling the patient in; and 8% of time is spent post-consultation, largely entering notes. Consultations with more than one person and where prescribing took place were longer (R² adj=22.5%, p<0.001). The core consultation can be divided into 61% of direct clinician–patient interaction, of which 15% is examination, 25% computer use with no patient involvement, and 14% simultaneous clinician–computer–patient interplay. The proportions of computer use are similar between consultations (mean=40.6%, SD=13.7%). There was more data coding in problem-orientated EPR systems, though clinicians often used vague codes.

The EPR system is used for a consistent proportion of the consultation and should be designed to facilitate multi-tasking. Clinicians who want to promote screen sharing should change their consulting room layout.

Quantitative and qualitative analysis of interactions between GPs, patients, and computer systems using multi-channel video recordings of 112 primary care consultations with eight GPs in three UK practices.

132 prescriptions were issued in the course of 73 of the consultations, of which 81 (61%) attracted at least one alert. Of the total of 117 alerts, only three resulted in the GP checking, but not altering, the prescription. CDSS provided information and safety alerts at the point of generating a prescription. This was ‘too much, too late’ as the majority of the ‘work’ of prescribing occurred prior to using the computer. By the time an alert appeared, the GP had formulated the problem(s), potentially spent several minutes considering, explaining, negotiating, and reaching agreement with the patient about the proposed treatment, and had possibly given instructions and printed an information leaflet.

CDSS alerts do not coincide with the prescribing workflow throughout the whole GP consultation. Current systems interrupt to correct decisions that have already been taken, rather than assisting formulation of the management plan.

CDSS are likely to be more acceptable and effective if the prescribing support is provided much earlier in the process of generating a prescription.

A cross-sectional study of 1000 adult admissions in five community hospitals from October 1, 2008 to September 30, 2010 was performed. Observed rates of preventable ADE and potential ADE were compared with scores reported by the Leapfrog CPOE evaluation tool. The primary outcome was the rate of preventable ADE and the secondary outcome was the composite rate of preventable ADE and potential ADE.

Leapfrog performance scores were highly related to the primary outcome. A 43% relative reduction in the rate of preventable ADE was predicted for every 5% increase in Leapfrog scores (rate ratio 0.57; 95% CI 0.37 to 0.88). In absolute terms, four fewer preventable ADE per 100 admissions were predicted for every 5% increase in overall Leapfrog scores (rate difference –4.2; 95% CI –7.4 to –1.1). A statistically significant relationship between Leapfrog scores and the secondary outcome, however, was not detected.

Our findings support the use of the Leapfrog tool as a means of evaluating and monitoring CPOE performance after implementation, as addressed by current certification standards.

Scores from the Leapfrog CPOE evaluation tool closely relate to actual rates of preventable ADE. Leapfrog testing may alert providers to potential vulnerabilities and highlight areas for further improvement.

The evaluation was designed as a non-randomized controlled trial with a historical control group (CG). In 2007, we consecutively recruited 1363 patients admitted for elective surgery, and evaluated the preoperative assessment. In 2008, PROP was implemented and available online. In 2009 we recruited 1148 patients preoperatively assessed using PROP (294 outpatients, 854 hospital sector). Our analysis includes full blood count, liver function tests, coagulation parameters, electrolytes, ECG, and chest x-ray.

The number of tests/patient without indication was 3.39 in the CG vs 0.60 in the intervention group (IG) (p<0.001). 97.8% (CG) vs 31.5% (IG) received at least one unnecessary test. However, we also observed an increase in recommended tests not performed/patient (0.05±0.27 (CG) vs 0.55±1.00 (IG), p<0.001). 4.2% (CG) vs 30.1% (IG) missed at least one necessary test. The guideline adherence (correctly tested/not tested) improved distinctively for all tests (1.6% (CG) vs 49.3% (IG), p<0.001).

PROP reduced the number of unnecessary tests/patient by 2.79 which implied a reduction of patients’ burden, and a relevant cut in unnecessary costs. However, the advantage in specificity caused an increase in the number of patients incorrectly not tested. Further research is required regarding the impact of PROP on perioperative outcomes.

We combined data from multiple electronic sources: the billing system, structured fields in the electronic health record (EHR), and free text physician notes using natural language processing (NLP). We also used point-of-care worksheets to capture clinical rationale.

Nephrologist billing accurately identified patients with CKD. Using an algorithm that incorporated multiple BP readings increased the measured rate of control (130/80 mm Hg) from 37.1% to 42.3%. With the addition of NLP to capture BP readings from free text notes, the rate was 52.6%. Data from point-of-care worksheets indicated that in 52% of visits in which patients were identified as not having controlled BP, patients were actually at goal based on BP readings taken at home or on that day in the office.

Building a method for clinically meaningful continuous performance measurement of BP control is possible, but will require data from multiple sources. Electronic measurement systems need to grow to be able to capture and process performance data from patients as well as in real-time from physicians.

We developed a system that automates the creation of diagnostic decision-support applications. It relies on a medical ontology to direct the acquisition of clinic data from a clinical data warehouse and uses an automated analytic system to apply a sequence of machine learning algorithms that create applications for diagnostic screening. We refer to this system as the ontology-driven diagnostic modeling system (ODMS). We tested this system using samples of patient data collected in Salt Lake City emergency rooms and stored in Intermountain Healthcare’s enterprise data warehouse.

The system was used in the preliminary development steps of a tool to identify patients with pneumonia in the emergency department. This tool was compared with a manually created diagnostic tool derived from a curated dataset. The manually created tool is currently in clinical use. The automatically created tool had an area under the receiver operating characteristic curve of 0.920 (95% CI 0.916 to 0.924), compared with 0.944 (95% CI 0.942 to 0.947) for the manually created tool.

Initial testing of the ODMS demonstrates promising accuracy for the highly automated results and illustrates the route to model improvement.

The use of medical knowledge, embedded in ontologies, to direct the initial development of diagnostic computing systems appears feasible.

To examine the use of computerized analysis of respiratory sounds, a hybrid system based on principal component analysis (PCA) and probabilistic neural networks (PNNs), to aid the detection of coexisting pneumonia in patients with COPD.

A convenience sample of 58 patients with COPD (25 patients hospitalized for community-acquired pneumonia and 33 owing to acute exacerbation of COPD) was studied. Auscultations were performed by the patients themselves on their suprasternal notch. Short-time Fourier transform analysis was used to extract features from the recorded respiratory sounds, PCA was selected for dimensionality reduction and a PNN was trained as classifier. 10-Fold cross-validation and receiver operating characteristic curve analysis were used to estimate the system performance.

Based on the cross-validation results, a sensitivity and a specificity of 72% and 81.8%, respectively, were achieved in validation data. The operating point was selected to maximize the specificity and sensitivity pair in the training set.

The results strongly suggest that electronic self-auscultation at a single location (suprasternal notch) can support diagnosis of pneumonia in patients with COPD.

A simple, cost-effective method has been proposed to aid decision-making in areas with no radiological facilities available and in resource-constrained settings, and could have a great diagnostic impact on telemedicine applications.

We developed the index using EHR data for 7463 patients ≥50 years old with ≥1 visit(s) in 2003 to a large, academic, multispecialty group practice. We extracted 980 attributes from the EHRs of the practices and affiliated hospitals. Correlation feature selection with greedy stepwise search was used to find the attribute subset with best average merit. Rotation forest ensembling with alternating decision tree as underlying classifier was used to predict 5 year mortality. Model performance was compared with the modified Charlson Comorbidity Index and the Walter life expectancy method.

Within 5 years of the last visit in 2003, 838 (11%) patients had died. The final model included 24 attributes: two demographic (age, sex), 10 comorbidity (eg, cardiovascular disease), one vital sign (mean diastolic blood pressure), two medications (loop diuretic use, digoxin use), six laboratory (eg, mean albumin), and three healthcare utilization (eg, the number of hospitalizations 1 year prior to the last visit in 2003). The index showed very good discrimination (c-statistic 0.86) and outperformed comparators.

The EHR based index successfully distinguished adults ≥50 years old with life expectancy >5 years from those with life expectancy ≤5 years. This information could be used clinically to optimize preventive service use (eg, cancer screening in the elderly).

Patient demographic data were obtained from the New York Clinical Information Exchange (NYCLIX), a regional health information organization (RHIO) representing most of the major medical centers in the borough of Manhattan in NYC. Patients’ home address zip codes were used to create a research dataset with aggregate counts of patients by US county and international standards organization country. Times Square was designated as the geographical center point of the RHIO for distance calculations.

Most patients (87.7%) live within a 30 mile radius from Times Square and there was a precipitous drop off of patients visiting RHIO-affiliated facilities at distances greater than 100 miles. 43.6% of patients visiting NYCLIX facilities were from the other NYC boroughs rather than from Manhattan itself (31.9%).

Most patients who seek care at members of NYCLIX live within a well-defined area and a clear decrease in patients visiting NYCLIX sites with distance was identified. Understanding the geographical distribution of patients visiting the large medical centers in the RHIO can inform the RHIO's planning as it looks to add new participant organizations in the surrounding geographical area.

Qualitative data were collected over 8 months from seven experts at New York State Department of Health who implemented web services and protocols for querying, receipt, and validation of electronic data supplied by regional health information organizations. Extensive project documentation was also collected. During group meetings experts described the implementation process and created reference modes and causal diagrams that the evaluation team used to build a preliminary model. System dynamics modeling techniques were applied iteratively to build causal loop diagrams representing the implementation. The diagrams were validated iteratively by individual experts followed by group review online, and through confirmatory review of documents and artifacts.

Three casual loop diagrams captured well-recognized system dynamics: Sliding Goals, Project Rework, and Maturity of Resources. The findings were associated with specific policies that address funding, leadership, ensuring expertise, planning for rework, communication, and timeline management.

This evaluation illustrates the value of a qualitative approach to system dynamics modeling. As a tool for strategic thinking on complicated and intense processes, qualitative models can be produced with fewer resources than a full simulation, yet still provide insights that are timely and relevant.

System dynamics techniques clarified endogenous and exogenous factors at play in a highly complex technology implementation, which may inform other states engaged in implementing HIE supported by federal Health Information Technology for Economic and Clinical Health (HITECH) legislation.

We developed a software tool, the Regional Healthcare Ecosystem Analyst (RHEA), that can accept user-inputted data to rapidly create a detailed agent-based simulation model (ABM) of the healthcare ecosystem (ie, all healthcare facilities, their adjoining community, and patient flow among the facilities) of any region to better understand the spread and control of infectious diseases.

To demonstrate RHEA's capabilities, we fed extensive data from Orange County, California, USA, into RHEA to create an ABM of a healthcare ecosystem and simulate the spread and control of methicillin-resistant Staphylococcus aureus. Various experiments explored the effects of changing different parameters (eg, degree of transmission, length of stay, and bed capacity).

Our model emphasizes how individual healthcare facilities are components of integrated and dynamic networks connected via patient movement and how occurrences in one healthcare facility may affect many other healthcare facilities.

A decision maker can utilize RHEA to generate a detailed ABM of any healthcare system of interest, which in turn can serve as a virtual laboratory to test different policies and interventions.

To present lessons learned about validation of EMR-based phenotypes from the Electronic Medical Records and Genomics (eMERGE) studies.

The eMERGE network created and validated 13 EMR-derived phenotype algorithms. Network sites are Group Health, Marshfield Clinic, Mayo Clinic, Northwestern University, and Vanderbilt University.

By validating EMR-derived phenotypes we learned that: (1) multisite validation improves phenotype algorithm accuracy; (2) targets for validation should be carefully considered and defined; (3) specifying time frames for review of variables eases validation time and improves accuracy; (4) using repeated measures requires defining the relevant time period and specifying the most meaningful value to be studied; (5) patient movement in and out of the health plan (transience) can result in incomplete or fragmented data; (6) the review scope should be defined carefully; (7) particular care is required in combining EMR and research data; (8) medication data can be assessed using claims, medications dispensed, or medications prescribed; (9) algorithm development and validation work best as an iterative process; and (10) validation by content experts or structured chart review can provide accurate results.

Despite the diverse structure of the five EMRs of the eMERGE sites, we developed, validated, and successfully deployed 13 electronic phenotype algorithms. Validation is a worthwhile process that not only measures phenotype performance but also strengthens phenotype algorithm definitions and enhances their inter-institutional sharing.

We use a one-time process of identifying which patients have data in multiple repositories by comparing one-way hash values of patient demographics. This enables us to partition the local databases such that all patients within a given partition have data at the same subset of hospitals. Federated queries are then run separately on each partition independently, and the combined results are presented to the user.

Using theoretical bounds and simulated hospital networks, we demonstrate that once the partitions are made, SHRINE can produce more precise estimates of the number of patients matching a query.

Uncertainty in the overlap of patient populations across hospitals limits the effectiveness of SHRINE and other federated query tools. Our technique reduces this uncertainty while retaining an aggregate federated architecture.

To create a historical controls database by leveraging internal clinical trial data at Pfizer, focusing on patients who received only placebo in randomized controlled trials.

We transformed disparate clinical data sources by indexing, developing, and integrating clinical data within internal databases and archives. We focused primarily on trials mapped into a consistent standard and trials in the pain therapeutic area as a pilot.

Of the more than 20 000 internal Pfizer clinical trials, 2404 completed placebo controlled studies with a parallel design were identified. Due to challenges with informed consent and data standards used in older clinical trials, studies completed before 2000 were excluded, yielding 1134 studies from which placebo subjects and associated clinical data were extracted.

It is technically feasible to pool portions of placebo populations through a stratification and segmentation approach for a historical placebo group database. A sufficiently large placebo controls database would enable previous distribution calculations on representative populations to supplement, not eliminate, the placebo arm of future clinical trials. Creation of an industry-wide placebo controls database, utilizing a universal standard, beyond the borders of Pfizer would add significant efficiencies to the clinical trial and drug development process.

Observational study using computerized analysis of publicly available Medline article data and clinical trial registry data. We analyzed a purposive set of five ICMJE founding journals looking at all trial articles published in those journals during 2010–2011, and data from the ClinicalTrials.gov (CTG) trial registry. We measured adherence to trial ID inclusion policy as the percentage of trial journal articles that contained a valid trial ID within the article (journal-based sample). Adherence to timely registration was measured as the percentage of trials that registered the trial before enrolling the first participant within a 60-day grace period. We also examined timely registration rates by year of all phase II and higher interventional trials in CTG (registry-based sample).

To determine trial ID inclusion, we analyzed 698 clinical trial articles in five journals. A total of 95.8% (661/690) of trial journal articles included the trial ID. In 88.3% the trial-article link is stored within a structured Medline field. To evaluate timely registration, we analyzed trials referenced by 451 articles from the selected five journals. A total of 60% (272/451) of articles were registered in a timely manner with an improving trend for trials initiated in later years (eg, 89% of trials that began in 2008 were registered in a timely manner). In the registry-based sample, the timely registration rates ranged from 56% for trials registered in 2006 to 72% for trials registered in 2011.

Adherence to URM requirements for registration and trial ID inclusion increases the utility of PubMed and links it in an important way to clinical trial repositories. This new integrated knowledge source can facilitate research prioritization, clinical guidelines creation, and precision medicine.

The five selected journals adhere well to the policy of mandatory trial registration and also outperform the registry in adherence to timely registration. ICMJE's URM policy represents a unique international mandate that may be providing a powerful incentive for sponsors and investigators to document clinical trials and trial result publications and thus fulfill important obligations to trial participants and society.