Computer-generated Patient Education Materials: Do They Affect Professional Practice?
A Systematic Review
- Affiliations of the authors: Department of Health Services Research, Oslo, Norway (SPT, CG, ADO); Wairau Hospital, Blenheim, New Zealand (AP)
- Correspondence and reprints: Shaun P. Treweek, PhD, Department of Health Services Research, Norwegian Directorate for Health and Social Welfare, P.O. Box 8054 Dep, N-0031 Oslo, Norway; e-mail: <streweek{at}online.no>
- Received 21 December 2001
- Accepted 11 March 2002
Abstract
A systematic search of seven electronic databases was done to identify randomized controlled trials that assessed the effect of computer-generated patient education material (PEM) on professional practice. Three studies met the authors' criteria.
All three studies involved preventive care. All used a complex intervention of which computer-generated PEM was a major component. Improvements in practice were seen in all studies, although these gains were generally modest. One study showed improvement in patient outcomes. Mann-Whitney statistics calculated for the studies' outcome measures ranged from 0.48 to 0.66, equivalent to risk differences of −4 to 32 percent.
Computer-generated PEM seems to have a small, positive effect on professional practice. The small number of included studies and the complex nature of the interventions makes it difficult to draw conclusions about the ability of computer-generated PEM to change professional practice. Future work should involve well-defined interventions that can be clearly evaluated in terms of effect and cost.
The traditional consultation between a health professional and a patient suffers from the flawed memories of both participants. A study by Kitching1 found that, on average, patients forget half of what they were told by a doctor within 5 min of leaving the consultation room. In addition, health professionals can easily forget to pass on important information to the patient. There is, therefore, scope for improvement in communication between professionals and patients.
This is especially true if patients are to be more active participants in their own care. One solution proposed by Coulter et al.2 is to ensure that patients have access to written or audiovisual material that supplements the material discussed during the consultation, which can also be used to guide questioning during later consultations.
This material could be delivered in several ways. Pre-printed booklets or leaflets are the most widely used method,3 but some authors4 5 suggest that computerized clinical information systems should routinely include education materials. These materials could also be tailored to individual patients. Another method would be to use interactive educational software packages that allow the patient to direct the delivery of information. These packages have been used to combine audio, video, text, and graphics to provide information on a wide range of topics.6 The Internet is also an increasingly important source of health information, although the quality of this information is currently variable.7
Are health professionals willing to make use of education materials and packages in their practice? Other reviews have focused on patient behavior and outcomes.8 9 The health professional is, however, often the point of access to these materials, and an education package should, at the very least, be good enough for a professional to want to make it available to patients.
Two potential barriers to increased use of education materials by professionals are storage and access problems (booklets and leaflets take up space and are not always to hand) and the need to keep the materials up to date.10 Wilson10 suggests computer-generated materials as a solution, because these materials are not subject to storage limitations and can be easily updated. Moreover, these materials are immediately available to the health care professional during a consultation, which is often not the case for preprinted leaflets. The computer system can also be used to remind the clinician about the patient information.11 However, use of computer-based materials may be limited by financial, administrative, and attitudinal barriers in a health care organization or among its health care professionals.
As Mayberry and Mayberry12 point out, it is important that the effect of patient education material (PEM) should be assessed with the same rigor as other interventions. In this review, we aim to systematically determine the effects of computer-generated PEM on professional practice (our primary outcome) and on patient outcomes.
Objectives
To examine the effect of computer-generated PEM as a way of changing the practice of health care professionals and patient outcomes. Three primary comparisons will be considered:
-
Comparison 1—providing computer-generated material compared to alternative material (preprinted leaflets, booklets, etc.).
-
Comparison 2—providing computer-generated material compared to no material.
-
Comparison 3—providing patient-specific computer-generated material compared to generic computer-generated material.
Methods
Our review is based on searching the following databases, from their start date to the end of 2000:
-
Medline (1966–2000)
-
Embase (1980–2000)
-
Cinahl (1982–2000)
-
Best Evidence (1991–2000)
-
Cochrane Collaboration EPOC specialized register (1966–2000)
-
Cochrane Controlled Trials Register (1947–2000)
-
Science Citation Index (1987–2000)
Individual search strategies were developed for each database (Appendix A). The search strategies were very liberal and designed to pick up as many potential articles as possible. Publications in any language were considered. The reference lists of review articles identified through the searches were also checked. The initial search identified 1,147 articles. A further 14 articles were identified by checking the reference lists of the review articles, giving a total of 1,161 articles.
Study Selection
Studies were considered eligible for inclusion in the review if they met inclusion criteria in four categories—types of intervention, types of studies, types of participants, and types of outcome measures.
Types of Intervention
The intervention must be PEM that involves the use of a computer in its storage and delivery, including interactive packages. The information in such material may be general or specific to individual patients. Education materials used with and without the health professional being present were considered eligible.
To be included in this review, education interventions must be delivered in one of three ways—before a consultation (e.g., information available on computers in the waiting room), during a consultation (e.g., a leaflet printed out and handed to the patient), or after a consultation (e.g., a computer-generated tailored leaflet, based on the consultation, that is sent to the patient)
Interventions that were simple patient reminders (e.g., a simple computer-generated letter suggesting that a patient come in for a health check) were excluded, since we considered these to have no education content. Moreover, a number of reviews have already considered these interventions13 14 15 or are in progress.16
Types of Studies
Studies must be randomized controlled trials or controlled trials. The unit of allocation must be the health professional or practice, since we are primarily interested in professional practice.17
Types of Participants
A study must involve health care professionals who are responsible for patient care. Studies involving only research staff without clinical responsibility for patients were excluded.
Types of Outcome Measures
All objective measures of health care professional practice and patient outcome were considered.
The primary outcome measure for this review is effect on professional practice, including delivery of information, cost, time spent with patients, prescribing, referrals, and other clinical activities. Studies with no measure of effect on professional practice were excluded.
Abstracts for the 1,161 articles were scanned for relevance by at least two reviewers. Full-text copies of all potentially relevant studies were obtained. These articles were then considered for inclusion in the review by at least two reviewers. Any discrepancies between reviewers arising from the inclusion assessment were resolved by discussion.
Analysis
The Mann-Whitney statistic was used to compare outcomes for intervention and control groups. This statistic estimates the probability that a subject chosen at random and given the intervention would have a better outcome than a subject similarly chosen and given the control. The statistic ranges from 0 to 1, with 0.5 meaning that both groups perform or respond equally well.
For dichotomous data, the Mann-Whitney statistic was calculated using:
where pi and pc are the proportions of intervention “successes” (e.g., a screening test was done) and control successes, respectively. The quantity pi−pc is also called the risk difference.
For continuous data, a Z statistic was calculated, and normal tables were then used to find the Mann-Whitney statistic. The Z statistic was calculated using the equation:
where
is the difference between the intervention and control means, and SDi and SDc are the standard deviations for the intervention and control means, respectively. More details on the use of the Mann-Whitney
statistic to compare the results of intervention studies can be found in Colditz et al.18 and Moses et al.19
Results
General Information
Of the 1,161 articles identified in our search, only three met all our inclusion criteria—Williams et al.20 (referred to here as “Williams”), Lowensteyn et al., 21 (“Lowensteyn”), and McPhee et al.22 (“McPhee”). Most articles were excluded because they discussed studies that were randomized by patient and not by professional, even if the study aimed to effect a change in professional practice. Articles that appeared relevant but were rejected after consideration of the full article text are listed in Appendix B.
Table 1 gives an overview of the Williams, Lowensteyn, and McPhee studies. All three are based in primary care, Williams and McPhee aiming to improve use of cancer screening, Lowensteyn aiming to support primary prevention of coronary heart disease. The Williams and McPhee studies were done in the United States, the Lowensteyn study in Canada.
Overview of the Three Included Studies
Physicians participating in the McPhee and Lowensteyn studies were not paid for their participation. Physicians in the Williams study who already subscribed to the Virginia Insurance Reciprocal malpractice insurance scheme received a one-time 6 percent reduction in their annual premium in return for their participation. Physicians involved in the McPhee study were all based at fee-for-service practices. Reimbursement arrangements at practices involved in the two other studies were not clear, although one of the five strata in the Williams study involved federally funded practices. None of the studies adequately described the randomization process.
Interventions
No study used an intervention involving only computer-generated PEM. Williams asked patients to use a touch-sensitive computer system to complete a questionnaire on personal and family medical history and lifestyle. The computer then produced PEM, chart organizers, order sheets, and patient-specific reminders for the physician, the PEM being given to the patient during the consultation. A nurse was also available to help practices during implementation of the system. Control practices did not receive the computer system until the end of the 1-year study period.
Lowensteyn collected risk factor information from the patient, and this information was then mailed to a central collection point. This center returned two risk profiles to the practice, one of which was given to the patient at a return visit about 2 weeks after the initial visit. Control practices did not receive the two risk profiles unless the patient was clinically re-evaluated at a follow-up visit following a minimum 3-month delay.
McPhee combined computer-generated PEM with booklet-based PEM and a physician reminder system. The physician reminder system generated information on appropriate screening, assessment, and counseling and other test information. Two copies of this information were generated, one each for the physician and patient. Control practices received nothing.
All three studies considered Comparison 2 (computer-generated PEM vs. no material) and provided estimates of the effects of computer-generated PEM on professional practice. One study (Lowensteyn) also provided estimates of effect on patient outcomes.
Williams reported a significant increase in the completion of screening mammography (8.8 percent difference between intervention and control) and clinical breast examinations (8.3 percent) in women 50 years of age and older. These results were significant at the p≤0.05 level. There were no significant changes for the five remaining screening tests.
Among patients who did not have a health maintenance examination (HME), use of the touch-screen system showed a significant increase (p≤0.05) in the number of patients who had fecal occult blood tests (3.9 percent increase). Patients who did have an HME were associated with higher (p≤0.01) proportions of screening mammographies (30.3 percent increase) and clinical breast examinations (32.4 percent). Williams also reported that patients who used the touch-screen system had a significantly higher rate of completion of screening tests than did non-users for all seven screening tests. However, not all non-users were in the control group, which may lead to a biased result in favor of the intervention (see Discussion).
Lowensteyn found that the coronary heart disease risk profile given to both patients and physicians led to a significant increase in the proportion of high-risk patients being reassessed at 3 months, compared with low-risk patients. The difference in the likelihood ratio (ratio of high-risk patients to low-risk patients returning for follow-up) between the risk profile group and the control group was 0.46 (95% confidence interval, 0.08–0.87). Lowensteyn had a difference in the unit of allocation and analysis, which is a methodological weakness, although the analysis of patient outcome does take this into account. This was not done for the risk likelihood calculation, making it unclear whether statistical significance was actually achieved for this result.
McPhee developed a percentage “performance score” for completion of screening tests and assessments, and the intervention package led to increases of between 8 and 34 percent in completion. These increases were significant for five tests or assessments (rectal examination, Papanicolaou smear, smoking assessment, smoking counseling, and diet assessment) at the p≤0.05 level and for three (stool occult-blood test, pelvic examination, and diet counseling) at the p≤0.01 level. The results for sigmoidoscopy, breast examination, and mammography were not significant.
The results were then adjusted to remove the effects of patient characteristics, using patients as the unit of analysis. As mentioned above, using patients as the unit of analysis when allocation was by physician is a methodological weakness. The adjusted results were broadly similar to the unadjusted results, but with higher levels of significance generally being claimed.
Neither Williams nor McPhee reported patient outcomes. Lowensteyn found that patients in the intervention group demonstrated greater reductions (p<0.05) in total cholesterol, low-density lipoprotein cholesterol, and the total cholesterol/high-density lipoprotein cholesterol ratio. This resulted in a significant improvement (p<0.01) in both cardiovascular age and predicted 8-year coronary risk, compared with the control group. As mentioned earlier, the units of allocation and analysis are different in this study. The authors did, however, include physician as a variable in an analysis of variance model when considering patient outcomes.
Table 2summarizes the Mann-Whitney statistics for the three studies, together with the published p values for each outcome. A Mann-Whitney statistic has not been calculated for the Lowensteyn “high-risk likelihood ratio,” because different units of allocation and analysis were used for this outcome. The spreadsheet used to calculate the Mann-Whitney statistics is available on request from the authors of this article.
Summary of the Mann-Whitney Statistics for Each of the Three Included Studies
Figure 1 shows a plot of the Mann-Whitney statistics for each study's major outcome measures. The point size is proportional to the sample size (actually 1/variance; see Figure 1). To make differentiation between the points easier, the x-axis has been started at 0.4, not 0.
Mann-Whitney statistics for each of the three included studies. The size of each point is proportional to 1/variance, where
variance is given by
and where ni and nc are the sample sizes for the intervention and control groups, respectively. The crosshair represents the mean Mann-Whitney
statistic for each study.
The Mann-Whitney statistics were largely in favor of the intervention (i.e., greater than 0.50), ranging from 0.48 (equivalent to a risk difference [RD] of −4 percent) to 0.66 (RD, 32 percent). The mean Mann-Whitney statistics were all in favor of the intervention: Williams had a mean of 0.52 (RD, 4 percent), McPhee 0.57 (RD, 14 percent), and Lowensteyn 0.54 (RD, 8 percent).
Discussion
This review has examined the effects of computer-generated PEM on the practice of health professionals. The small number of studies included in this review is a clear indication that the effect of computer-generated PEM on professional practice is an aspect of PEM that is rarely evaluated. In addition, several studies that did assess the effect of computer generated PEM on professional practice failed to randomize by professional or practice, a methodological error that tends to overestimate any effect.17 23 One included study that was randomized by practice used the patient as the unit of analysis, which can lead to over-narrow confidence intervals and spuriously significant findings.23
It is, therefore, difficult to draw conclusions about the effects of computer-generated PEM on professional practice. The Williams, Lowensteyn, and McPhee studies report positive effects, and most of the Mann-Whitney statistics given in Table 2 favor the intervention. Table 2 and Figure 1 suggest that for these three North American studies, involving similar groups of patients and health professionals and using broadly similar interventions, PEM can contribute to a small improvement in professional and patient outcomes. Taking the Mann-Whitney statistics together, the risk difference between PEM intervention and control is around 8 to 10 percent.
Williams found a significant increase (about 8 percent) in the proportion of women older than 50 years who received screening mammography and clinical breast examinations. These authors also compared patients who had used the touch-screen system with patients who had not. Patients who used the system showed a significant increase in the completion of all seven cancer screening tests. However, this result is likely to be optimistic, since not all non-users were in the control group.
At practices in which the touch-screen system was placed in the waiting area (20 of 29), patients who used the system were self-selected. At the remaining nine practices, the system was used by a group of patients selected by the practice. Both methods are likely to result in bias regarding use of the system, and non-users at intervention practices should be considered when assessing the intervention's effect. The intervention also involved a physician reminder, and it is difficult to disentangle the effect of the reminder from the computer-generated PEM. The positive effect of physician reminders is well documented.13 14 This problem is perhaps more acute with the McPhee study.
The Lowensteyn study, which combined computer-generated PEM with a physician reminder system for managing coronary risk, achieved a significant increase in the ratio of high-risk to low-risk patients returning for follow-up. However, the unit of allocation was different from the unit of analysis. It is likely that the result would have been less favorable had the data been analyzed (correctly) using the 129 physicians who actively participated instead of the 958 patients. An alternative for the original authors would have been to use an intra-cluster correlation coefficient, which would describe the extent to which patients within a cluster (a practice) are truly independent of each other.
We were interested primarily in the effect of PEM on professional practice. Eight studies were therefore excluded from our review because patients rather than professionals were randomized. For studies that aim to measure this effect, randomization by professional is appropriate because randomizing by patient would lead to the same professionals being in both the experimental and the control groups.
Randomizing by patient would be appropriate only if the PEM was delivered independent of the professional and the aim of a study was to measure changes in patient rather than professional behavior. Internet-based PEM obtained without professional input would ensure delivery independent of professionals and prevent contamination in that respect, but the same professionals would still be exposed (via the patients) to both the experimental and control interventions. Thus, randomization at the level of the professional is the only way to reliably measure effects on professional practice.
To illustrate other computer-based PEM systems, however, we thought it would be interesting to consider these eight articles in the Discussion; they are listed in Table 3. Of these, four considered Comparison 1,24 25 26 27 five considered Comparison 2,27 28 29 30 31 and two considered Comparison 3.25 27 Four studies reported a significant positive change in an outcome related to professional practice,24 26 27 28 whereas the four remaining studies reported no significant change in these outcomes. None of the eight studies reported objective measures of patient outcome; patients' knowledge of their condition was the most common measure reported.
Overview of the Eight Studies that Did Not Meet the Randomization Criteria but Did Meet the Three Other Inclusion Criteria
It must be emphasized that these studies are not included to provide quantitative evidence to support or refute the use of particular techniques. As mentioned earlier, studies that assess the effect of an intervention on professional practice should be randomized by professional or practice, not by patient.17 Even if the focus is patient outcome, if patients are drawn from several clinical sites, then clustering effects must be considered. These eight studies do, however, provide further (qualitative) food for thought.
None of the three studies meeting our full inclusion criteria considered cost. It is therefore uncertain whether the effects of the interventions are worth the costs involved. One of the studies that did not meet our randomization criteria, that by Jones et al.,25 did mention cost. These authors found that, in the absence of an electronic medical record system, their personalized computer-based PEM system would cost more than nine times that of their general computer-based PEM. With an electronic record, the cost of the personalized system was similar to that of the general system. The cost of the general computer-based PEM system was about 40 percent the cost of full access to booklets and would be less than the cost of booklets within the first year. This suggests that the cost of computer-based PEM may be modest and that even small positive effects may be worth the investment, particularly in the presence of an electronic medical record.
Personalized systems were used in all three of the studies that met our full inclusion criteria. In addition, four of the six studies that did not meet our randomization inclusion criteria24 25 27 28 used personalized systems. Only two studies25 27 compared these with general computer-based PEM, and both studies reported benefits from use of the personalized system. As Jones et al.25 point out, in the presence of an electronic medical record, such systems are of similar cost to general systems, and a personalized approach may be worth considering in future computer-based PEM systems. The evidence supporting a positive effect on professional practice is, however, currently weak.
Making computer-based PEM part of an intervention package, rather than the sole intervention, may lead to a greater effect on practice. Combining PEM systems with reminder systems may be especially useful, since reminders have been found to have a significant positive effect on professional practice.13 14 15 This approach was taken in all three studies that met our full inclusion criteria, although it is difficult to wholeheartedly endorse the approach on the basis of the results of these three studies.
Conclusions
Implications for Practice
Computer-generated PEM appears to have a small, positive effect on professional practice. However, there is currently scant evidence to support this conclusion. Combination with other interventions, particularly patient and physician reminder systems may also be promising, but further work is required to ascertain which interventions should be combined and in which circumstances. The cost effectiveness of computer-generated PEM has been inadequately assessed.
Until further evidence becomes available, purchasers should be cautious about using computer-generated PEM as a vehicle for large-scale change in professional practice. Other considerations, however, such as improved patient participation in decision making and increased patient satisfaction, may make such investment worthwhile. In decisions about whether investment in computer-generated PEM meets a provider's health care strategy, the current review should be considered with reviews that take a more patient-centered focus.8 9
Implications for Future Research
It is surprising that only three studies can be said to adequately address the effects of computer-generated PEM on professional practice. More and better PEM is a feature of many national health policies, and the attitudes of health professionals toward this, and the potential effect of PEM on their practices, is still largely unknown. Future work should address these issues, should be of higher methodological quality, should have a well-defined intervention package that can be clearly evaluated, and should consider cost. How such systems can be implemented into routine care should also be considered.
Internet-based services are already an important source of PEM, and the importance of the Internet as a source of PEM is likely to increase. The effect of these systems on professional practice and patient outcomes should be evaluated. Since computer-generated PEM is likely to have, at best, a modest effect on professional practice and patient outcome, developers should aim to make their systems cheap, preferably linked to existing electronic medical record systems.
Appendix A
Search Strategies for the Review
Effective Practice and Organization of Care (EPOC) specialized register search strategy:
comput* [text term]
Cochrane Controlled Trials Register (CCTR) search strategy:
-
comput* [text term]
-
Patient-education [exploded MeSH term]
-
1 and 2
MeSH terms for medline search strategy (through Internet Grateful Med):
-
Patient Education
-
Health Education
-
Delivery of Health Care
-
Patient Participation
-
Health Promotion
-
Preventive Medicine
-
Professional-Patient relations
-
Consumer Participation
-
Computers
-
comput* [text term]
-
Computer-Assisted Instruction
-
Therapy, Computer-Assisted
-
Medical Informatics Applications
-
Medical Informatics
-
Clinical Trial [exploded Publication Type]
-
(1 or 2 or 3 or 4 or 5 or 6 or 7 or 8) and (9 or 10 or 11 or 12 or 13 or 14) and 15
MeSH terms for EMBASE search strategy (through Ovid):
-
exp Patient Education/
-
exp Health Promotion/
-
exp Patient counseling/ or Patient guidance/ or Patient information/ or Patient education
-
exp ‘automation, computers and computer applications’/ or Computer/
-
exp Clinical trial/
-
(1 or 2 or 3) and 4 and 5
Best Evidence search strategy (through Ovid):
Comput$ [text terms]
MeSH terms for Cinahl search strategy (through Ovid):
-
exp Patient education/
-
exp ‘Computers and computerization’/
-
exp Experimental studies/
-
1 and 2 and 3
Science Citation Index search strategy (through BibSys):
-
computer (text word)
-
patient (text word)
-
education (text word)
-
1 and 2 and 3
Appendix B
Studies Excluded from the Review
