rss
J Am Med Inform Assoc 2008;15:227-234 doi:10.1197/jamia.M2493
  • Original Investigation
  • Research Paper

Individual and Joint Expert Judgments as Reference Standards in Artifact Detection

  1. Marion Verduijna,d,
  2. Niels Peeka,
  3. Nicolette F de Keizera,
  4. Erik-Jan van Lieshoutb,
  5. Anne-Cornelie J M de Pontb,
  6. Marcus J Schultzb,
  7. Evert de Jongeb,
  8. Bas A J M de Molc,d
  1. aDepartment of Medical Informatics, Academic Medical Center, Amsterdam, The Netherlands
  2. bDepartment of Intensive Care Medicine, Academic Medical Center, Amsterdam, The Netherlands
  3. cDepartment of Cardio-thoracic Surgery, Academic Medical Center, Amsterdam, The Netherlands
  4. dDepartment of Biomedical Engineering, University of Technology, Eindhoven, The Netherlands
  1. Correspondence: Marion Verduijn, PhD, Department of Medical Informatics, Academic Medical Center (AMC), P.O. Box 22700, 1100 DE Amsterdam, the Netherlands (Email: <m.verduijn{at}amc.uva.nl>)
  • Received 30 April 2007
  • Accepted 10 December 2007

Abstract

Objective To investigate the agreement among clinical experts in their judgments of monitoring data with respect to artifacts, and to examine the effect of reference standards that consist of individual and joint expert judgments on the performance of artifact filters.

Design Individual judgments of four physicians, a majority vote judgment, and a consensus judgment were obtained for 30 time series of three monitoring variables: mean arterial blood pressure (ABPm), central venous pressure (CVP), and heart rate (HR). The individual and joint judgments were used to tune three existing automated filtering methods and to evaluate the performance of the resulting filters.

Measurements The interrater agreement was calculated in terms of positive specific agreement (PSA). The performance of the artifact filters was quantified in terms of sensitivity and positive predictive value (PPV).

Results PSA values between 0.33 and 0.85 were observed among clinical experts in their selection of artifacts, with relatively high values for CVP data. Artifact filters developed using judgments of individual experts were found to moderately generalize to new time series and other experts; sensitivity values ranged from 0.40 to 0.60 for ABPm and HR filters (PPV: 0.57–0.84), and from 0.63 to 0.80 for CVP filters (PPV: 0.71–0.86). A higher performance value for the filters was found for the three variable types when joint judgments were used for tuning the filtering methods.

Conclusion Given the disagreement among experts in their individual judgment of monitoring data with respect to artifacts, the use of joint reference standards obtained from multiple experts is recommended for development of automatic artifact filters.

Footnotes

  • Dr. Peek received a grant from the Netherlands Organization of Scientific Research (NWO) under project number 634.000.020.

This Article

  1. Abstract
  2. Full text
  3. PDF
  4. Data Supplement

Responses

  1. Submit a response
  2. No responses published

Social bookmarking

Access policy for JAMIA

All content published in JAMIA is deposited with PubMed Central by the publisher with a 12 month embargo. Authors/funders may pay an Unlocked fee of $2,000 to make the article free on the JAMIA website and PMC immediately on publication.

All content older than 12 months is freely available on this website.

AMIA members can log in with their JAMIA user name (email address) and password or via the AMIA website.