Generating a Reliable Reference Standard Set for Syndromic Case Classification
- Affiliations of the authors: RODS Laboratory, Center for Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA
- Correspondence and reprints: Wendy W. Chapman, PhD, Center for Biomedical Informatics, University of Pittsburgh, Suite 8084, Forbes Tower, 200 Lothrop Street, Pittsburgh, PA 15213-2582; e-mail: <chapman{at}cbmi.pitt.edu>
- Received 31 March 2005
- Accepted 7 June 2005
Abstract
Objective To generate and measure the reliability for a reference standard set with representative cases from seven broad syndromic case definitions and several narrower syndromic definitions used for biosurveillance.
Design From 527,228 eligible patients between 1990 and 2003, we generated a set of patients potentially positive for seven syndromes by classifying all eligible patients according to their ICD-9 primary discharge diagnoses. We selected a representative subset of the cases for chart review by physicians, who read emergency department reports and assigned values to 14 variables related to the seven syndromes.
Measurements (1) Positive predictive value of the ICD-9 diagnoses; (2) prevalence of the syndromic definitions and related variables; (3) agreement between physician raters demonstrated by κ, κ corrected for bias and prevalence, and Finn's r; and (4) reliability of the reference standard classifications demonstrated by generalizability coefficients.
Conclusion Of the 27 syndromes generated by the 14 variables, 21 showed high enough prevalence, agreement, and reliability to be used as reference standard definitions against which an automated syndromic classifier could be compared. Syndromic definitions that showed poor agreement or low prevalence include febrile botulinic syndrome, febrile and nonfebrile rash syndrome, respiratory syndrome explained by a nonrespiratory or noninfectious diagnosis, and febrile and nonfebrile gastrointestinal syndrome explained by a nongastrointestinal or noninfectious diagnosis.
Footnotes
-
This work was funded by Defense Advanced Research Projects Agency (DARPA) Cooperative Agreement No. F30602-01-2-0550, Pennsylvania Department of Health grant no. ME-01-737, and AHRQ grant no. 290-00-0009.
-
The authors thank the physician raters: Karen Barnard, Amber Barnato, Peter Bulova, Rebecca Drayer, Gary Fischer, Mandy Garber, Robin Gehris, Franziska Jovin, Chris Rihn, and David Segel.
