Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial
- Annie Y S Lau1,
- Vitali Sintchenko1,2,3,
- Jacinta Crimmins4,
- Farah Magrabi1,
- Blanca Gallego1,
- Enrico Coiera1
- 1Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia
- 2Sydney Medical School, University of Sydney, Sydney, Australia
- 3Centre for Infectious Diseases and Microbiology, Institute of Clinical Pathology and Medical Research, Westmead Hospital, Sydney, Australia
- 4University Health Service, University of New South Wales, Sydney, Australia
- Correspondence to Dr Annie Y S Lau, Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, UNSW, Sydney NSW 2052, Australia;
Contributors Study conceptualization: AL, EC. Study design: AL, EC, JC, VS. Journey design: JC, AL. Data collection: AL, JC. Data analysis: AL, VS, BG. First draft: AL, VS. Draft revision and approval: AL, VS, JC, FM, BG, EC.
- Received 17 June 2011
- Accepted 9 April 2012
- Published Online First 12 May 2012
Objective To assess the impact of a web-based personally controlled health management system (PCHMS) on the uptake of seasonal influenza vaccine and primary care service utilization among university students and staff.
Materials and methods A PCHMS called Healthy.me was developed and evaluated in a 2010 CONSORT-compliant two-group (6-month waitlist vs PCHMS) parallel randomized controlled trial (RCT) (allocation ratio 1:1). The PCHMS integrated an untethered personal health record with consumer care pathways, social forums, and messaging links with a health service provider.
Results 742 university students and staff met inclusion criteria and were randomized to a 6-month waitlist (n=372) or the PCHMS (n=370). Amongst the 470 participants eligible for primary analysis, PCHMS users were 6.7% (95% CI: 1.46 to 12.30) more likely than the waitlist to receive an influenza vaccine (waitlist: 4.9% (12/246, 95% CI 2.8 to 8.3) vs PCHMS: 11.6% (26/224, 95% CI 8.0 to 16.5); χ2=7.1, p=0.008). PCHMS participants were also 11.6% (95% CI 3.6 to 19.5) more likely to visit the health service provider (waitlist: 17.9% (44/246, 95% CI 13.6 to 23.2) vs PCHMS: 29.5% (66/224, 95% CI: 23.9 to 35.7); χ2=8.8, p=0.003). A dose–response effect was detected, where greater use of the PCHMS was associated with higher rates of vaccination (p=0.001) and health service provider visits (p=0.003).
Discussion PCHMS can significantly increase consumer participation in preventive health activities, such as influenza vaccination.
Conclusions Integrating a PCHMS into routine health service delivery systems appears to be an effective mechanism for enhancing consumer engagement in preventive health measures.
Trial registration Australian New Zealand Clinical Trials Registry ACTRN12610000386033. http://www.anzctr.org.au/trial_view.aspx?id=335463.
- Personal health record
- preventive health
- consumer informatics
- human behavior modeling
- social influence
- decision support
- internet intervention
- patient safety
- incident reporting
- patient safety
- decision support
Background and significance
Personal health records (PHRs) offer an opportunity to directly engage consumers in preventive healthcare.1 Consumers are increasingly using a range of online systems to inform their decisions and manage their care.2–5 However, few studies have measured the impact of these systems on consumer behavior or the uptake of preventive measures.
In a recent review, only three of 10 randomized controlled trials (RCTs) involved electronic implementations of PHRs,6 and none demonstrated significant differences in health service utilization rates or the uptake of preventive measures.7–9 The review called for further trials to evaluate the effectiveness and sustainability of PHRs, and to examine the role of primary care service providers. Indeed, there is now a broad call for more RCTs to evaluate all classes of healthcare information systems.10
We describe an RCT of a personally controlled health management system (PCHMS) called Healthy.me, developed at the University of New South Wales (UNSW).11 This PCHMS integrates (i) an untethered PHR, (ii) consumer care pathways called ‘journeys,’ and (iii) social forums and messaging that allow consumers to interact with each other and with healthcare professionals.
Our PCHMS aims to provide an integrated platform for consumers to manage their health, in partnership with their health service providers. It seeks to minimize knowledge-based (eg, lack of awareness) and system-based (eg, inconvenience) barriers associated with accessing health services, making consumers more likely to engage in preventive health measures such as influenza vaccination.12
Influenza is an important contributor to loss of workforce productivity and is a significant cause of seasonal mortality (eg, among the elderly13). Influenza vaccination has been estimated to save US$46.85 per person vaccinated,14 which from a population perspective affords substantial community benefit. A meta-analysis of interventions to increase adult vaccination and cancer screening rates concluded that involving patients in self-management through reminders can positively improve uptake.15
The hypotheses tested in the current RCT are (i) that consumers using a PCHMS are more likely to comply with public health recommendations, as measured by rates of seeking and obtaining influenza vaccination, and (ii) that providing online facilities within a PCHMS to schedule encounters with a health service provider will increase the utilization of those services.
Trial design and participants
We designed a 2010 CONSORT-compliant two-group (6-month waitlist control vs PCHMS) parallel RCT (allocation ratio 1:1).16
Box 1 outlines participant selection criteria. The study recruited 855 students and staff in an Australian university setting from May 2010, where participants were either randomized to the PCHMS, or a wait–list control arm that could use the system after 6-months, when the trial had concluded. Written material advertising the study did not mention influenza, vaccination, or service utilization.
Eligibility criteria for participants
Aged 18 or above
Access to the internet, and email at least on a monthly basis
Did not complete the registration process (excluded before randomization)
Self-reported having obtained an influenza vaccination in 2010 prior to enrolment in the study (excluded from analysis at post-study)
Self-reported to be influenced by other participants during the study to obtain (or not obtain) influenza vaccination (excluded from analysis at post-study)
Written informed consent was sought from each participant. Ethics approval was obtained from the UNSW ethics committee.
Students and staff were approached via mailing lists and advertisements in online print publications, which described the study and invited participants to complete an online pre-study survey. Participants randomized to the intervention group then completed a 5-min mandatory online tutorial about Healthy.me prior to using the site. All participants (control and intervention) then received an email in the first week of each month inviting them to complete a 1-min survey (four questions) about influenza-like illness (ILI) symptoms and health activities from May to October 2010. At study completion, all participants received an email asking them to complete a post-study survey (20 questions). Two follow-up emails 5 days apart were sent to non-completers. Those who completed all surveys entered into a draw for one $A500 prize. Supplementary online appendix table A gives the completion rates for each survey.
A researcher was available via a dedicated telephone line and email to answer participant concerns and address any unintended effects during the study. Participants could also provide feedback via the monthly surveys.
The intervention was a web-based PCHMS called Healthy.me (figure 1).11 Central to the system's design are consumer specific care pathways called ‘journeys’ that provide disease or task specific knowledge in an actionable way. A journey can be viewed as a health service engagement protocol for patients and consumers, outlining the steps and activities associated with a specified service or task. For example, at the point that a consumer encounters advice to seek influenza vaccination, they can immediately book an appointment with a doctor from the journey page, or set themselves a reminder to do so. Journeys are computationally active and can personalize other PCHMS elements like the PHR to reflect the specific content of the journey. For example, commencing a vaccination journey can trigger the creation of a vaccination record in the PHR.
Influenza vaccine journey
The influenza vaccine journey in Healthy.me contained two elements:
A consumer vaccination care pathway (figure 2), which described (i) the types of influenza vaccine currently available (ii) steps to obtain vaccination at the University Health Service (UHS; the university primary care service) or elsewhere, and (iii) vaccine costs, adverse effects, and contraindications
Online appointment booking (figure 3), whereby participants could click a ‘Book now’ button on the journey page, thus sending an email to the UHS to book an appointment for influenza vaccination or other medical issues. A dedicated UHS administrative staff member would telephone participants by the next working day to confirm appointments.
The journey was designed in consultation with UHS primary care physicians, utilizing government-endorsed evidence-based consumer education material, and was tested in the previous year for seasonal and pandemic H1N1 influenza.17–20
Table 1 outlines study outcome measures. The protocol and outcomes were not changed after study commencement.
Influenza-like illness case definition
ILI symptoms were based upon case definitions of influenza (fever with cough or a sore throat) issued by the state health department (NSW Health) and the US Centers for Disease Control and Prevention (CDC) as of 26 March 2010.21 ,22 Febrile upper respiratory tract illnesses occurring during the peak influenza period was identified as the most specific clinical case definition with the highest positive predictive value for true influenza.23
Six hundred participants with 300 in each arm were needed to detect a 10% difference in vaccination rate between the waitlist control and the PCHMS groups, calculated at 5% level of significance, 80% power (two-sided test), with an anticipated participant dropout rate of 10%.24 The effect size estimate is based on a review of patient reminder systems showing improved immunization rates of 5%–20%.25 A baseline estimate of influenza vaccination at the UHS showed 16% of patients received FluVax or PanVax between March 5, 2009 and November 5, 2009. The literature reports that 18%–30% of university students and healthy adults (18–49 years old) obtain an influenza vaccination each year.26 ,27
Eligible consumers were randomly assigned to the PCHMS or waitlist control by a random number sequence, pre-generated externally to the research team with a computerized random-number generator with randomly assigned blocks (block sizes 2, 4, and 8) and an intervention allocation ratio of 1:1.28
Allocation concealment, mechanism, and implementation
Allocation occurred automatically at enrolment. Participants received the next consecutive allocation in the random number sequence, assigning them to the PCHMS or waitlist control.
Since a PCHMS is a behavioral intervention, it was not possible to completely blind participants. Group allocation was revealed to participants after obtaining their consent to participate and completion of the pre-study survey. Investigators and clinicians were blinded to group allocation. To minimize contamination of the control group, the intervention group was asked to not share their PCHMS access details with others. Four participants who reported being influenced by other participants were excluded from the analysis.
Baseline comparisons to assess randomization effectiveness were conducted using the Student t tests and χ2 tests. Adjustments for baseline characteristics and potential confounders were made using sequential logistic regression31 to provide a stratified estimate of intervention effect.32 ,33 Primary analysis examined differences in the proportion of participants obtaining influenza vaccination during the study in the waitlist and PCHMS groups using the χ2 test, including participants who had the opportunity to use the PCHMS but did not do so. Secondary outcomes were assessed using the χ2 test (table 1). Differences in average number of days of absence per participant were compared using the Student t test.
A total of 855 participants were recruited and 604 followed up between May and October 2010 (figure 4; supplementary online appendix table A). Of 742 participants who met the inclusion criteria, 372 were randomly allocated to the 6-month waitlist and 370 to the PCHMS.
Randomization resulted in an even distribution of the pre-study baseline measures across the waitlist and PCHMS groups (p>0.05), and at post-study primary analysis (p>0.05) (table 2).
Primary and secondary analyses were conducted on 470 participants who met eligibility criteria. Data for ancillary analyses were available for 86.8% (644/742) of participants who completed at least one monthly survey, and subgroup analyses were conducted on 345 participants who accessed the PCHMS. No significant differences in post-study survey completion rates were found between the waitlist and PCHMS groups (χ2 (1, n=604)=1.6, p=0.207).
Influenza vaccination (primary analysis)
Primary analysis comparing influenza vaccination between the waitlist and PCHMS recipients is presented in Figure 5 and supplementary online appendix table C.
In absolute terms, participants assigned to the PCHMS were 6.7% (95% CI 1.5 to 12.3) more likely than waitlist recipients to receive an influenza vaccine (χ2 (1, n=470)=7.1, p=0.008; waitlist: 4.9% (12/246, 95% CI 2.8 to 8.3) vs PCHMS: 11.6% (26/224, 95% CI 8.0 to 16.5)). Relative to the waitlist control, the proportion of participants receiving an influenza vaccine was 137% higher for the PCHMS group (RR: 2.4 (95% CI 1.2 to 4.6)). Overall, the PCHMS had a small but significant effect (Φ=0.123) on influenza vaccination rates.
Sequential logistic regression assessed whether baseline characteristics (box 1) or other post-study factors (ie, contact with children during the study, past history of influenza vaccine) affected vaccination rate. The final logistic regression model explained between 8.3% (Cox and Snell R square) and 19.3% (Nagelkerke R square) of variance in the uptake of influenza vaccine, and correctly classified 92.1% of influenza vaccination cases. All three independent variables (past history of influenza vaccine, group allocation to PCHMS, and age) made a unique statistically significant contribution to the model (χ2 (3)=40.6, p<0.001) (supplementary online appendix table D).
Health service utilization (secondary analysis)
Secondary analysis of visits to the UHS is outlined in figure 5 and supplementary online appendix table C. Access to PCHMS attracted 11.3% (17/150, 95% CI 7.2 to 17.4%) of participants who were not current patients to visit the service during the study.
In absolute terms, participants assigned to PCHMS were 11.6% (95% CI 3.6 to 19.5) more likely than those assigned to the waitlist control to visit the UHS during the study (χ2 (1, n=470)=8.8, p=0.003; waitlist: 17.9% (44/246, 95% CI 13.6 to 23.2) vs PCHMS: 29.5% (66/224, 95% CI 23.9 to 35.7)). Relative to the waitlist control, the proportion of participants visiting the UHS was 65% higher for the PCHMS group (RR: 1.6 (95% CI 1.2 to 2.3)). Overall, PCHMS use had a small but significant effect (Φ=0.137) on visits to the UHS.
Influenza-like illness symptoms
Cumulative rates of monthly ILI symptoms and health activities from May through October 2010 are summarized in supplementary online appendix table B. No significant differences in ILI symptoms were detected between PCHMS and waitlist participants (p>0.05). During the study, 27.0% (174/644) of participants across the waitlist control and PCHMS groups had ≥1 episode of ILI symptoms (fever/fever-like, with coughs or sore throat), 14.0% (90/644) had ≥1 episode of ILI symptoms with vomiting or diarrhea, and 80.7% (520/644) experienced at least one of these symptoms (fever/fever-like, cough, sore throat, vomiting or diarrhea).
Impact on health activities according to frequency of PCHMS access
There appears to be a dose–response effect associated with the use of the PCHMS on health service engagement behaviors (figure 6, supplementary online appendix table E). Compared to accessing the PCHMS only once, accessing the PCHMS more than once was associated with:
Significantly higher use of the PHR to add, update, or delete health data (χ2 (1, n=345)=66.5, p<0.001; used PCHMS once only: 16.5% (41/248); used PCHMS more than once: 60.8% (59/97))
Significantly greater use of the online feature to book appointments with the UHS (χ2 (1, n=345)=68.2, p<0.001; used PCHMS once only: 1.2% (3/248); used PCHMS more than once: 29.9% (29/97)).
Among those who also reported their health service usage, accessing the PCHMS more than once was associated with:
Significantly higher rates of influenza vaccination (χ2 (1, n=184)=11.7, p=0.001; used PCHMS once only: 8.5% (11/130); used PCHMS more than once: 27.8% (15/54)).
Significantly higher rates of visiting the UHS (χ2 (1,n=184)=9.1, p=0.003; used PCHMS once only: 23.8% (31/130); used PCHMS more than once: 46.3% (25/54)).
Reasons for influenza vaccination
Reasons for obtaining (or not obtaining) an influenza vaccine were categorized using past history of influenza vaccination and post-study vaccination status (Past–Post). The most frequently cited reason for vaccination among Yes–Yes participants was ‘don't want to get sick’ (70.8%, 17/24); Yes–No participants: ‘forgot’ (30.9%, 25/81); No–No participants: ‘low-risk or low impact of getting ill’ (27.1%, 95/351); and No–Yes participants: ‘PCHMS’ (57.1%, 8/14) (supplementary online appendix tables F and G).
No harms or unintended effects were reported by participants during the study. The trial did not end earlier than planned.
To our knowledge, this is the first trial of a PCHMS to have shown a statistically significant increase in vaccination rates and health service utilization by consumers. The intervention's effect size (an absolute difference of 6.7% in influenza vaccination rate between PCHMS and waitlist control) is similar to the 5%–20% increase in vaccination rates reported for recall and reminder systems targeted at clinicians.25 It does appear that PCHMS can significantly increase consumer participation in preventive health activities, such as influenza vaccination.
We have previously tested the suitability of our PCHMS to support women undergoing fertility treatment. In that clinical setting, patients were highly motivated to engage in their care, and required complex scheduling of procedures and tests over a relatively short period (eg, see Stone et al15). In contrast, the present study focuses on a preventive clinical setting which has a much lower intensity of activity, where building motivation for engagement with clinical services is one of the challenges. Demonstrating the effectiveness of the PCHMS across these diverse settings and patient groups provides initial evidence that this approach is indeed generalizable and of broad utility.
A previous study using a web-based PHR incorporating reminders, weekly influenza risk maps, and provision of respiratory illness advice, did not report significant improvements in vaccination rates.9 That study did not include online booking, or a consumer specific health service engagement protocol, which are possible reasons why the present intervention was more effective. Online consumer systems in practice offer a ‘bundle’ of e-health services and features well beyond the PHR, and we now need to focus our efforts to identify the right mix of these features that motivate consumer behavior change in different clinical settings.
One approach to bundling e-health services and features is to recognize the ability of a PCHMS to overcome the barriers consumers experience when accessing primary care services.34 This study provides evidence that a PCHMS which (i) addresses knowledge barriers (eg, via disease and task specific service description) and (ii) reduces system barriers (eg, embedding action tools like online booking within these service descriptions) is a promising approach to engage consumers in health service utilization for preventive health activities.
A one-size-fits-all strategy is thus unlikely to result in sustained change in consumer behavior.12 PCHMS service bundles will need to accommodate varying levels of consumer health literacy, motivation, and willingness to engage in preventive behaviors. For example, effective interventions to influence health behaviors in young people will undoubtedly require the use of social networks, social influences, viral communication, and social recommendations,35 ,36 given the ubiquitous nature of such channels in this population.
In this study, the relative simplicity of the PCHMS design, along with the use of immediately actionable tools (such as the ‘Book now’ button), embedded within consumer specific content (ie, the journey), may account for the improved uptake of influenza vaccination and increased visits to the health service. The Health Belief Model, in particular, discusses how cues in an environment can motivate action.37 Developed in the early 1950s, the HBM remains one of the most widely used and cited conceptual frameworks for understanding why individuals did or did not engage in a variety of health-related actions.37 The HBM uses perceived susceptibility, perceived severity, perceived benefits and perceived barriers to explain and predict individuals' acceptance of health and medical care recommendations.37 With our web-based PCHMS, by providing informational cues that are directly linked to action, we may have overcome some of the perceived barriers that participants experience when deciding to obtain an influenza vaccine, and thus increased the likelihood that consumer intention translated to action.
There are several explanations for the dose–response effect we observed between increased PCHMS access and increased health service utilization. There are two broad causal interpretations—either that increased exposure leads to increased service utilization, or second that those who are high service utilizers are also likely to use the PCHMS more frequently, reflected in higher motivation for engagement.38 ,39 If increased exposure indeed leads to higher service utilization, then the combination of increased information through content in journeys and the use of actionable tools in the PCHMS, are likely explanations for this effect. However, the dose–response effect between use of web interventions and health behaviors is complex.38 Having more features on a website does not necessarily increase participant engagement, and may sometimes create adherence challenges as more features require more effort from participants.39
Research is now at the stage where we have evidence that web interventions can indeed trigger health behavior changes, but the empirical and theoretical basis for e-health service design is still weak, especially concerning which ‘bundles’ of website features would lead to behavior change. Future studies should employ more theoretical approaches to designing e-health services, recognizing that uptake and outcome changes may be highly dependent on population, disease group, and socio-economic factors.
Not much is currently known about the cost effectiveness of PCHMS as preventive health strategies. Using results from this trial to calculate the underlying return on investment in PCHMS to improve vaccination rates is difficult for a number of reasons. First, the return on investment depends upon an assessment of the costs of deploying and maintaining a system in a working clinical setting, and the current trial uses a research grade system rather than one designed for routine use. Second, a system like this is likely to concurrently support multiple clinical conditions and tasks across a variety of settings, meaning that the cost of system operation would be distributed across all these parallel uses. We would not anticipate a system like ours would be used exclusively to improve vaccination rates.
The study relied on self-reports by participants, which has been shown to be acceptably accurate in studies of days of absence,40 influenza symptoms, and vaccination status for diverse patient cohorts.41–46 We minimized the risk of recall bias by conducting short 1-min monthly follow-up surveys during the first week of each month. In addition, we validated influenza vaccination and health service utilization rates by matching self-reports from study participants with their medical records at the UHS (supplementary online appendix table C).
In 2010 the uptake of vaccination and the number of confirmed influenza diagnoses were lower than expected due to a relatively mild winter in Australia,47 and the controversies around the adverse effects of the 2010 seasonal influenza vaccine. It is possible that in a more severe season of influenza, the impact of PCHMS on vaccination rates and health service utilization could be higher than observed in our study. Integrating PCHMS into routine health service delivery systems does appear to be an effective mechanism for enhancing consumer engagement in preventive health measures.
This RCT provides evidence that a web-based PCHMS improves the uptake of influenza vaccination and utilization of health services. The nature of the intervention is sufficiently general that it is likely to be applicable to a wide variety of preventive health tasks, and future work should focus on understanding how to design the right “bundle” of E-Health services and features for different clinical settings and different health tasks.
Worldwide, governments have made multi-billion dollar investment in e-health to modernize health services delivery, with many questions still unanswered about the uptake, benefits, and cost effectiveness of these investments.48 ,49 Finding approaches that effectively engage consumers in e-health, with minimal attrition rates, remains a high priority.50–52 There is lack of a systematic approach to guide the design of consumer E-Health systems that would encourage the uptake and engagement amongst patients and clinicians. For this reason, unpacking the features and factors that drive effective online engagement remains a crucial area for future research.
The authors thank Farshid Anvari, Jingbo Liu, Vitaliy Kim, and Jay Liu for their contributions to software development, and the staff and students at UNSW who assisted or participated in the study.
Funding This research is supported in part by grants received from the National Health and Medical Research Council (NHMRC) Centre for Research Excellence (1032664), and the HCF Health and Medical Research Foundation. The funding bodies did not have a role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation, review, or approval of the manuscript.
Competing interests The university and some of the researchers involved in this project could benefit from any commercialization of the personally controlled health management system (PCHMS).
Ethics approval Ethics approval was provided by the University of New South Wales Human Research Ethics Committee (HREC10109).
Provenance and peer review Not commissioned; externally peer reviewed.