Use of statistical analysis in the biomedical informatics literature
- 1Yale Center for Medical Informatics, Yale University, New Haven, Connecticut, USA
- 2Center for Outcomes Research and Evaluation, Yale-New Haven Hospital, New Haven, Connecticut, USA
- Correspondence to Dr M Scotch, Center for Medical Informatics, Yale University, 300 George Street, Suite 501, New Haven, CT 06511, USA;
- Received 12 May 2008
- Accepted 23 August 2009
Statistics is an essential aspect of biomedical informatics. To examine the use of statistics in informatics research, a literature review of recent articles in two high-impact factor biomedical informatics journals, the Journal of American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics was conducted. The use of statistical methods in each paper was examined. Articles of original investigations from 2000 to 2007 were reviewed. For each journal, the results by statistical methods were analyzed as: descriptive, elementary, multivariable, other regression, machine learning, and other statistics. For both journals, descriptive statistics were most often used. Elementary statistics such as t tests, χ2, and Wilcoxon tests were much more frequent in JAMIA, while machine learning approaches such as decision trees and support vector machines were similar in occurrence across the journals. Also, the use of diagnostic statistics such as sensitivity, specificity, precision, and recall, was more frequent in JAMIA. These results highlight the use of statistics in informatics and the need for biomedical informatics scientists to have, as a minimum, proficiency in descriptive and elementary statistics.
Statistical analysis is an essential component of all biomedical research including research in informatics. Much of clinical informatics research involves implementing new methods and technologies, and evaluating their effectiveness. Use of descriptive and inferential methods enables researchers to summarize findings and conduct hypothesis testing. Despite its importance, a recent study showed that medical residents lack the knowledge to understand the most common statistics found in clinical journals.1 This deficiency limits their ability to critically analyze scientific papers, extrapolate key findings, apply the new knowledge in practice, and ultimately advance the science.
The International Medical Informatics Association (IMIA) includes knowledge of statistics as part of their recommendations for medical informatics education.2 3 Most, if not all, National Library of Medicine (NLM) degree-granting programs in biomedical informatics4 require at least an introductory course in biostatistics. In addition, one of the authors (RS) recently participated in a committee tasked with developing core content for a curriculum in applied informatics,5 which included a consensus recommendation that some level of biostatistical competence be demonstrated.
While most of the popular biomedical informatics textbooks contain elements of statistics, the actual use of statistics in the biomedical informatics literature is unclear. To examine the current use of statistical methods, we analyzed a sample of recent investigations published in two leading informatics journals. We propose that this work will help to illuminate the use of statistics and consequently the educational and training needs of biomedical informatics professionals.
Original investigation articles published in the Journal of the Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI) from 2000 to 2007 were reviewed (except for supplements). For JAMIA, this includes volumes 7–14, and for IJMI, volumes 57–76. JAMIA and IJMI were selected because they have high impact factors6 among informatics journals. For our analysis, we considered only original investigations. In JAMIA, this includes research papers, case reports, methods papers and model formulation papers.7 JAMIA articles not considered included literature reviews, application development, whitepapers, viewpoints and editorials. In IJMI, we concluded that research papers and “practice of informatics” were original research.
For both journals, one of the authors (MS for JAMIA and MD for IJMI) examined each paper, including abstract, figures, tables and the body of text, and recorded all statistics used in the paper. Each statistical method was recorded only once per paper, no matter how often it was used. Other authors re-examined a sample of about 10% of the papers (ZQ and CB for JAMIA and CB for IJMI). During this initial round of review, more categories for classifying statistical methods were added. Because of this, a second round of review was conducted.
We adapted the categories of Windish et al1 for analyzing statistical methods in journal articles. The main categories include: descriptive statistics, elementary statistics, multivariable statistics, other regression analyses, and other. Descriptive statistics includes mean, median, frequency, SD, and IQR. Elementary statistics includes χ2, t test, Kaplan–Meier, Wilcoxon rank sum, Fisher exact, ANOVA, and correlation. Multivariable statistics includes Cox proportional hazard, logistic regression and linear regression. The category other regression analyses includes weighted logistic regression, unconditional logistic regression, conditional logistic regression, longitudinal regression, Poisson regression, pooled logistic regression, nonlinear regression, negative binomial regression, and generalized estimating equations. Another category, machine learning and data mining (not included in Windish et al), includes statistical classifiers such as Bayesian networks, decision trees, artificial neural networks and support vector machines. This group also includes unsupervised learning methods such as clustering. Finally, the category other statistics includes mostly classification and diagnostic test analyses such as relative risk/risk ratio, sensitivity/specificity and precision/recall.
From 2000 to 2007, we identified 305 JAMIA papers and 532 IJMI papers that met our inclusion criterion. For the JAMIA articles, articles were also stratified by article type: research papers, case reports, methods papers, and model formulation papers (table 1). IJMI papers were not stratified because the type of the original investigation was not always indicated by the journal. Table 2 shows the number (and percentage) of articles in which each type of test was applied for each journal.
A sample of 10% (31) of the JAMIA papers had an observed agreement of 0.95 (κ=0.85) among two of the reviewers (MS and CB). A sample of 10% (53) of the IJMI papers had an observed agreement of 0.90 (κ=0.63) among two of the reviewers (MD and CB).
Descriptive statistics such as mean and SD were by far the most frequently used in both biomedical informatics journals. Elementary statistics including parametric and non-parametric tests were used in 42% of the JAMIA studies, while only 22% of the IJMI papers used these types of statistics. Statistics that are often used for clinical reasoning and decision-making, such as sensitivity, specificity, precision and recall, were more frequent in JAMIA than IJMI. For multivariable statistics including regression, JAMIA had 12% of these, while IJMI had 6%. Finally, data-mining and machine-learning methods such as support vector machines, decision trees, and Bayesian networks were 9% in JAMIA and 6% in IJMI.
Original investigations frequently include statistical analysis. In our results, the use of descriptive and elementary statistics was high. In addition, diagnostic statistics such as sensitivity, specificity, precision, and recall, a popular approach for those original studies using statistics, was frequently used in JAMIA articles. This is not surprising, since much of the informatics field revolves around use of computers to assist decision-making (in medicine, public health, etc) as well as evaluation of different methods for retrieving biomedical information. Clinicians are taught methods for reasoning under uncertainty and the use of sensitivity, specificity and other diagnostic statistics. Biomedical informatics trainees without clinical backgrounds are taught these statistics through introductory biomedical informatics coursework and textbooks such as Medical Informatics: Computer Applications in Health Care8 and Evaluation Methods in Biomedical Informatics.9
Descriptive or elementary knowledge of statistics is needed for informatics research such as decision-support system evaluation, understanding the barriers to Electronic Medical Record implementation, information retrieval, summarization of phylogenetic analysis, or spatial clustering for outbreak detection. In fact, many studies in clinical settings (eg, clinical trials) require even more sophisticated techniques. Knowledge of statistics, therefore, is important not only for those conducting research studies, but also for understanding the findings in the biomedical informatics literature and scientific presentations.
Development of biomedical informatics training requirements and competencies in statistics must be done with the consideration that the current statistical methods used in these journals might not always represent the most appropriate methods to analyzing data. There is the potential that statistical tests not often used in the field would greatly enhance the analysis of biomedical informatics research. Careful consideration must be done when finalizing training requirements and competency guidelines.
Our study focused on articles in the JAMIA and the IJMI. These journals publish articles from all different foci within biomedical informatics. This includes bioinformatics, which is one of the most popular and growing disciplines within the field. JAMIA is more clinically oriented, and the number of bioinformatics studies is small. Thus, our results likely do not account for the breadth of statistics in these studies which, because of the nature of this discipline, can be more sophisticated.
As core competencies and credentialing in biomedical informatics are developed, scientists in this field should have, as a minimum, proficiency in descriptive and elementary statistics.
Funding This project is supported in part by TI5 LM007056 and R01 LM007199 from the National Library of Medicine (NLM).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.