Telehealth

The Need for Evaluation Redux

Generally, we do not publish papers describing the background and methods for a research project until the results are available for inclusion. In the case of the papers by Shea and Starren in this issue, we decided to make an exception. Our decision reflects the following factors. First, the literature does not contain examples of adequate evaluation of telemedicine despite years of application of the technology and several calls to action. Second, this trial is a major effort, and the results will not be available for some time. Third, it is unlikely that multiple large-scale trials are underway in this area. Accordingly, we decided that access to the methods would inform the community of the type of research that is needed, regardless of the outcome of this specific trial. The urgent need for more evaluation argues against publication delay. The lack of competing parallel efforts limits the chance of introducing bias by early publication.—William W. Stead, MD

Five years ago, the journal published a set of five articles describing telehealth applications. Accompanying those papers was an editorial lamenting the lack of adequate evaluation for these studies.1 This editorial appeared shortly after an Institute of Medicine report was published that reached the same conclusions about telemedicine in general.2

Unfortunately, this situation has not changed in the ensuing half-decade. Two years ago, concerned with political pressure to reimburse telemedicine services through Medicare despite an unclear picture about efficacy and cost-effectiveness, the Health Care Financing Authority, along with the Agency for Healthcare Research and Quality, awarded a contract to the Evidence-based Practice Center at Oregon Health & Science University, to prepare an evidence review on the efficacy of telemedicine interventions in terms of diagnosis, clinical outcomes, satisfaction, access to care, and cost. The original review assessed telehealth applications for the Medicare population,3 while a supplemental study analyzed pediatric and obstetric population applications.4

Our conclusions from these reviews, which were exhaustive analyses of the peer-reviewed literature in telemedicine, echoed the previous observations—that while telemedicine research has led to novel and creative uses of the technology, the quality of the evaluation studies is poor. It is important to note that the major problems we found were with the methodologies of the studies. Thus, we were careful not to conclude that telemedicine technologies were not efficacious but rather that the low quality of studies assessing them precluded any conclusions about their efficacy. Examples of the problems we found included:

• Diagnostic efficacy studies in which the telemedicine and in-person assessments were performed by the same individuals

• A paucity of clinical outcome studies in clinical areas in which telemedicine is widely used

• Satisfaction studies with extremely low response rates and use of nonstandardized instruments

• Access studies that failed to use appropriate measures of access to care

• Cost studies that focused solely on the tradeoff of system cost vs. patient travel or emergency transport cost, ignoring the effects of adverse outcomes or the cost of the whole episode of care

In our reviews, we noted that medical informatics investigators have demonstrated for many years the capability to carry out well-designed studies assessing the application of information technology in the health care setting.5 6 7 To the potential concern that the technology is changing too rapidly to achieve adequate research control conditions, we also stated that techniques (e.g., “tracker trials”) have been developed to cope with clinical trials of changing technologies.8 We also took journal editors to task for publishing papers that had well-written descriptions of systems and issues in the use of technologies but were marred by poor evaluation designs. Thus, we concluded that the telemedicine community still had not met the a challenge of defining the efficacious use and cost-effectiveness of their technologies.

This issue of the Journal features two papers describing a large-scale telemedicine project in New York.9 10 The ongoing evaluation study is naturally of great interest to us. We are pleased that a large-scale evaluation has been incorporated into this project from the outset and note that its methodology is vastly superior to those of most of the studies we reviewed for our evidence report. However, we do have two concerns about the methodology that we hope the authors will address in this or future studies. The first is that the control intervention appears to consist of no intervention at all. Since the experimental intervention consists of both a telemedicine intervention and intensive nursing case management, a positive outcome of the study will not enable us to discern whether benefit accrued from the telemedicine intervention, the extensive case management, or both. Somewhat ironically, this issue has also plagued the various studies of the Diabetes Control and Complications Trial.11 Although these experiments were purported to demonstrate the beneficial effects of tight blood sugar control in diabetes mellitus, they may in reality have demonstrated the value of nursing case management. A better control group in the Shea et al. study would be one in which comparable nursing case management was also delivered by non-telemedicine means. This would enable us to assess the added value of telemedicine per se.

The second concern is a hope that Shea et al. will look beyond intermediate patient outcome measures of glycosylated hemoglobin and blood pressure to actual outcomes, such as development of complications, morbidity, and mortality. Although the measurement of intermediate outcomes provides more statistical power, the ultimate aim of health care interventions is to directly benefit patients, not improve their test results. A related problem with the intermediate outcome measures is their use in cost-effectiveness calculations. Although the current evaluation will allow this intervention to be compared with other interventions that affect the intermediate measures (i.e., other diabetes interventions), it will not allow comparisons that a policy maker might wish to make, such as comparison with treatments for osteoporosis. A better cost-effectiveness measure would be quality-adjusted life years.

Nonetheless, we applaud this large-scale trial and eagerly await the results. We hope the findings will not only enable us to get a better picture of the value of this form of telemedicine application for diabetic care but also serve as a means of expanding our knowledge about the contribution of control conditions for future evaluations. Such high-quality studies should also serve as a standard and raise the bar for the methodology of future evaluations in telemedicine and telehealth applications.