J Am Med Inform Assoc doi:10.1136/amiajnl-2011-000557
  • Research and applications

Validity of electronic health record-derived quality measurement for performance monitoring

  1. Sarah Shih
  1. Primary Care Information Project, New York City Department of Health and Mental Hygiene, New York, New York, USA
  1. Correspondence to Dr Amanda Parsons, Health Care Access & Improvement, New York City Department of Health and Mental Hygiene, 42-09 28th Street, 12th Floor, Queens, NY 11101, USA; aparsons{at}
  • Received 18 August 2011
  • Accepted 21 December 2011
  • Published Online First 16 January 2012


Background Since 2007, New York City's primary care information project has assisted over 3000 providers to adopt and use a prevention-oriented electronic health record (EHR). Participating practices were taught to re-adjust their workflows to use the EHR built-in population health monitoring tools, including automated quality measures, patient registries and a clinical decision support system. Practices received a comprehensive suite of technical assistance, which included quality improvement, EHR customization and configuration, privacy and security training, and revenue cycle optimization. These services were aimed at helping providers understand how to use their EHR to track and improve the quality of care delivered to patients.

Materials and Methods Retrospective electronic chart reviews of 4081 patient records across 57 practices were analyzed to determine the validity of EHR-derived quality measures and documented preventive services.

Results Results from this study show that workflow and documentation habits have a profound impact on EHR-derived quality measures. Compared with the manual review of electronic charts, EHR-derived measures can undercount practice performance, with a disproportionately negative impact on the number of patients captured as receiving a clinical preventive service or meeting a recommended treatment goal.

Conclusion This study provides a cautionary note in using EHR-derived measurement for public reporting of provider performance or use for payment.


  • Funding This study was supported by the Agency for Healthcare Research and Quality (grant nos R18HS17059 and 17294). The funder played no role in the study design, in the collection, analysis and interpretation of data, in the writing of the report or in the decision to submit the paper for publication.

  • Competing interests None.

  • Ethics approval This study was approved by the Department of Health and Mental Hygiene institutional review board no 09067.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Related Article

Free Sample

This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JAMIA.
View free sample issue >>

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Open Access fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.

Navigate This Article