J Am Med Inform Assoc 19:346-352 doi:10.1136/amiajnl-2011-000185
  • Research and applications
  • Focus on health information technology, electronic health records and their financial impact

A framework for evaluating the appropriateness of clinical decision support alerts and responses

  1. Josh F Peterson1,4,5
  1. 1Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
  2. 2Department of Biostatistics, University of Kansas Medical Center, Kansas City, Kansas, USA
  3. 3Division of Nephrology, Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
  4. 4Center for Health Services Research, Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
  5. 5Geriatric Research Education Clinical Center (GRECC), VA Tennessee Valley Healthcare System, Nashville, Tennessee, USA
  1. Correspondence to Dr Allison B McCoy, School of Biomedical Informatics, The University of Texas Health Science Center at Houston (UTHealth), The University of Texas at Houston-Memorial Hermann Center for Healthcare Quality and Safety, 6410 Fannin St, UTPB 1100, Houston, TX 77030, USA; allison.b.mccoy{at}
  • Received 14 February 2011
  • Accepted 6 July 2011
  • Published Online First 17 August 2011


Objective Alerting systems, a type of clinical decision support, are increasingly prevalent in healthcare, yet few studies have concurrently measured the appropriateness of alerts with provider responses to alerts. Recent reports of suboptimal alert system design and implementation highlight the need for better evaluation to inform future designs. The authors present a comprehensive framework for evaluating the clinical appropriateness of synchronous, interruptive medication safety alerts.

Methods Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI).

Results Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate.

Conclusion The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.


  • Ethics approval Vanderbilt University Institutional Review Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Related Article

Free Sample

This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JAMIA.
View free sample issue >>

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Open Access fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.

Navigate This Article