In reply

In reply:—We appreciate this opportunity to clarify the methodology and conclusions of our paper “Indicators of Accuracy of Consumer Health Information on the Internet,”1 in response to the letter from Drs. Carroll, Saluja, and Tarczy-Hornoch.

Carroll et al. may be correct that consulting a physician is the best way for people to obtain accurate health information. However, the reality is that millions of people are using the Internet to obtain health information.2 And it is unlikely that they are restricting themselves to sites “run by large professional or governmental organizations.” In addition, the information that consumers find on the Internet frequently has an effect on treatment decisions.2

Given that people are using the Internet to obtain health information, it seems reasonable to provide them with guidelines for evaluating this information. Furthermore, such guidelines should be empirically grounded. A number of individuals and organizations have published guidelines for evaluating the quality of health information on the Internet.3 However, at the time we began our study, no empirical work had been done to test these guidelines. In other words, no one had checked to see whether any of the proposed indicators of accuracy really are indicators of accuracy.

Since ours was one of the first empirical studies to test these guidelines, we would not want to claim that our methodology is the ideal one. As more researchers address this important issue, better methodologies will undoubtedly be developed. Even so, we do not believe that our study is invalidated by the concerns raised by Carroll et al.

To determine whether a proposed indicator of accuracy really is an indicator of accuracy, one has to be able to measure the accuracy of information on a particular health topic. As Carroll et al. suggest, this is an extremely difficult task. In designing our study, however, we took encouragement from the fact that a number of studies (e.g., Berland et al.,4 Impicciatore et al.,5 Griffiths and Christensen,6 and Sandvik7) have undertaken to measure the accuracy of information on the Internet on a variety of health topics. In fact, our accuracy protocol (which is available on request) is based on the procedure used in a previous study5 on the accuracy of information on the treatment of fever in children.

The standard metholodology in these studies has been to measure accuracy using authoritative sources and expert opinion. The authoritative sources that we used to develop our accuracy protocol are given in our paper. Of course, as Carroll et al. suggest, there are some disagreements among experts on the treatment of fever in children. In fact, there are going to be some disagreements among experts on most health topics. However, we found that several authoritative sources were in agreement on most details on this topic. The greater the consensus among experts, the more reliable the measure of accuracy.

The main reason that people need accurate health information is so that they do not come to harm. Of course, as Carroll et al. correctly point out, a measure of accuracy is not necessarily a measure of safety. Unfortunately, measuring the safety of health information is even more difficult than measuring accuracy. For example, in addition to determining whether some information is inaccurate, one has to be able to predict what the likely consequences of basing a treatment decision on that information will be. As a result, we chose—as have all the other researchers in this area—to focus first on measuring accuracy. In addition, as the title of our paper attests, we were very explicit about this choice.

With regard to the selection of Web sites, we used essentially the same technique as other studies on the accuracy of health information on the Internet. The specific search engines and search terms that we used are given in our paper. All the Web sites on the treatment of fever in children that were retrieved by use of these search engines and these terms were included in our study.

Setting aside concerns about our methodology, Carroll et al. go on to claim that our conclusions go beyond what our data support. In particular, they claim that our data do not show that “the presence of the HONcode logo on a Web site is a fairly good indication that a Web site contains accurate information on the treatment of fever in children.”1 They point to the fact that the HONcode logo was found on three Web sites that were judged to be inaccurate. However, this is simply to misunderstand our assertion. As we are careful to point out in the paper, the presence of the HONcode logo is not a guarantee that a Web site contains accurate information. In fact, no one has ever claimed that any of the proposed indicators of accuracy are guarantees of accuracy. However, our data do show that a Web site that displays the HONcode logo is significantly more likely to contain accurate information on the treatment of fever in children.

Although our data suggest that the presence of the HONcode logo may really be an indicator of accuracy, we do make another important proviso in our paper; that is, our study looked at only one particular health topic. Proposed indicators of accuracy should certainly be tested on a variety of health topics before we can be confident that any of them really are indicators of accuracy.

Finally, we would like to emphasize an important part of our paper that Carroll et al. neglected to mention. That is, we found that “several of the proposed indicators from published guidelines for evaluating the quality of health information on the Internet are not correlated with accuracy.”1 In fact, this result has been confirmed by the relatively few studies that have tested these published guidelines.

For example, in a study on the accuracy of information on depression, Griffiths and Christensen6 found—just as we did—that “currently popular criteria for evaluating the quality of Web sites were not indicators of content quality.” (Griffiths and Christensen combined several proposed indicators of accuracy into a single index. They looked for a correlation between this index and content quality.) Also, in a study on the accuracy of information on incontinence, Sandvik7 found—just as we did—that “the number of links to a site was not an indication of the quality of its contents.” (Sandvik looked only for a correlation between this particular proposed indicator of accuracy and content quality.)—DON FALLIS, PHD, MARTIN FRICKÉ, PHD