Evaluation of an Intelligent Tutoring System in Pathology: Effects of External Representation on Performance Gains, Metacognition, and Acceptance
- Rebecca S Crowleya,b,c,
- Elizabeth Legowskia,
- Olga Medvedevaa,
- Eugene Tseytlina,
- Ellen Rohd,
- Drazen Jukica,c,d
- aDepartment of Biomedical Informatics, University of Pittsburgh School of Medicine, Pittsburgh, PA
- bIntelligent Systems Program, University of Pittsburgh School of Medicine, Pittsburgh, PA
- cDepartment of Pathology, University of Pittsburgh School of Medicine, Pittsburgh, PA
- dDepartment of Dermatology, University of Pittsburgh School of Medicine, Pittsburgh, PA
- Correspondence and reprint requests to: Rebecca Crowley, MD, MS, Department of Biomedical Informatics, University of Pittsburgh School of Medicine, UPMC Shadyside Cancer Pavilion, Room 307, 5230 Centre Avenue, Pittsburgh, PA 15232; e-mail: <crowleyrs{at}upmc.edu>)
- Received 5 August 2006
- Accepted 5 December 2006
Abstract
Objective Determine effects of computer-based tutoring on diagnostic performance gains, meta-cognition, and acceptance using two different problem representations. Describe impact of tutoring on spectrum of diagnostic skills required for task performance. Identify key features of student-tutor interaction contributing to learning gains.
Design Prospective, between-subjects study, controlled for participant level of training. Resident physicians in two academic pathology programs spent four hours using one of two interfaces which differed mainly in external problem representation. The case-focused representation provided an open-learning environment in which students were free to explore evidence-hypothesis relationships within a case, but could not visualize the entire diagnostic space. The knowledge-focused representation provided an interactive representation of the entire diagnostic space, which more tightly constrained student actions.
Measurements Metrics included results of pretest, post-test and retention-test for multiple choice and case diagnosis tests, ratios of performance to student reported certainty, results of participant survey, learning curves, and interaction behaviors during tutoring.
Results Students had highly significant learning gains after one tutoring session. Learning was retained at one week. There were no differences between the two interfaces in learning gains on post-test or retention test. Only students in the knowledge-focused interface exhibited significant metacognitive gains from pretest to post-test and pretest to retention test. Students rated the knowledge-focused interface significantly higher than the case-focused interface.
Conclusions Cognitive tutoring is associated with improved diagnostic performance in a complex medical domain. The effect is retained at one-week post-training. Knowledge-focused external problem representation shows an advantage over case-focused representation for metacognitive effects and user acceptance.
Footnotes
-
Supported by the National Library of Medicine through grant R01-LM007891.
-
The authors thank Maria Bond for her assistance with manuscript preparation. This work was conducted using the Protégé resource, which is supported by grant LM007885 from the United States National Library of Medicine. SpaceTree was provided in collaboration with the Human-Computer Interaction Lab (HCIL) at the University of Maryland, College Park.
-
Preliminary findings from this study were reported in the Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005).
