Understanding Detection Performance in Public Health Surveillance: Modeling Aberrancy-detection Algorithms
- David L Buckeridgea,b,
- Anna Okhmatovskaiaa,b,
- Samson Tuc,
- Martin O'Connorc,
- Csongor Nyulasc,
- Mark A Musenc
- aDepartment of Epidemiology and Biostatistics, McGill University, Montreal, Canada
- bMcGill Clinical and Health Informatics, McGill University, Montreal, Canada
- cStanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA
- Correspondence: David Buckeridge, MD, PhD, McGill Clinical and Health Informatics, 1140 Pine Avenue West, Montreal, QC, H3A 1A3 (Email: david.buckeridge{at}mcgill.ca)
- Received 19 March 2008
- Accepted 25 July 2008
Abstract
Objective Statistical aberrancy-detection algorithms play a central role in automated public health systems, analyzing large volumes of clinical and administrative data in real-time with the goal of detecting disease outbreaks rapidly and accurately. Not all algorithms perform equally well in terms of sensitivity, specificity, and timeliness in detecting disease outbreaks and the evidence describing the relative performance of different methods is fragmented and mainly qualitative.
Design We developed and evaluated a unified model of aberrancy-detection algorithms and a software infrastructure that uses this model to conduct studies to evaluate detection performance. We used a task-analytic methodology to identify the common features and meaningful distinctions among different algorithms and to provide an extensible framework for gathering evidence about the relative performance of these algorithms using a number of evaluation metrics. We implemented our model as part of a modular software infrastructure (Biological Space-Time Outbreak Reasoning Module, or BioSTORM) that allows configuration, deployment, and evaluation of aberrancy-detection algorithms in a systematic manner.
Measurement We assessed the ability of our model to encode the commonly used EARS algorithms and the ability of the BioSTORM software to reproduce an existing evaluation study of these algorithms.
Results Using our unified model of aberrancy-detection algorithms, we successfully encoded the EARS algorithms, deployed these algorithms using BioSTORM, and were able to reproduce and extend previously published evaluation results.
Conclusion The validated model of aberrancy-detection algorithms and its software implementation will enable principled comparison of algorithms, synthesis of results from evaluation studies, and identification of surveillance algorithms for use in specific public health settings.
Footnotes
-
This research was supported by a grant from the Centers for Disease Control and Prevention under the BioSense Initiative to Improve Early Event Detection (5R01PH000027) and the Protégé Resource Grant from the National Institutes of Health (NLM LM007885). David Buckeridge is supported by a Canada Research Chair in Public Health Informatics.
-
↵The subtask order listed in the text and depicted in Figure 2 does not imply the order of their execution by task-decomposition methods. Task sequencing will be discussed later in this section.
-
↵† Here, the term “algorithm” is used in a more specific sense than when referring to aberrancy-detection algorithms in the surveillance literature; thus the two usages of this term should not be confused.
