Social but safe? Quality and safety of diabetes-related online social networks
- 1Children's Hospital Informatics Program at the Harvard-MIT Division of Health Sciences and Technology, Children's Hospital Boston, Boston, Massachusetts, USA
- 2Division of Adolescent Medicine, Children's Hospital Boston, Boston, Massachusetts, USA
- 3Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA
- 4Division of Emergency Medicine, Children's Hospital Boston, Boston, Massachusetts, USA
- 5Manton Center for Orphan Disease Children's Hospital Boston, Boston, Massachusetts, USA
- Correspondence to Dr Elissa R Weitzman, Children's Hospital Informatics Program, One Autumn Street, Room 541, Boston, MA 02215, USA;
- Received 9 November 2010
- Accepted 17 December 2010
- Published Online First 24 January 2011
Objective To foster informed decision-making about health social networking (SN) by patients and clinicians, the authors evaluated the quality/safety of SN sites' policies and practices.
Design Multisite structured observation of diabetes-focused SN sites.
Measurements 28 indicators of quality and safety covering: (1) alignment of content with diabetes science and clinical practice recommendations; (2) safety practices for auditing content, supporting transparency and moderation; (3) accessibility of privacy policies and the communication and control of privacy risks; and (4) centralized sharing of member data and member control over sharing.
Results Quality was variable across n=10 sites: 50% were aligned with diabetes science/clinical practice recommendations with gaps in medical disclaimer use (30% have) and specification of relevant glycosylated hemoglobin levels (0% have). Safety was mixed with gaps in external review approaches (20% used audits and association links) and internal review approaches (70% use moderation). Internal safety review offers limited protection: misinformation about a diabetes ‘cure’ was found on four moderated sites. Of nine sites with advertising, transparency was missing on five; ads for unfounded ‘cures’ were present on three. Technological safety was poor with almost no use of procedures for secure data storage and transmission; only three sites support member controls over personal information. Privacy policies' poor readability impedes risk communication. Only three sites (30%) demonstrated better practice.
Limitations English-language diabetes sites only.
Conclusion The quality/safety of diabetes SN is variable. Observed better practice suggests improvement is feasible. Mechanisms for improvement are recommended that engage key stakeholders to balance autonomy, community ownership, conditions for innovation, and consumer protection.
Internet engagement centered on health is expanding beyond online search activity, reported for example by 66% of healthy US adults and 51% of those with a chronic illness.1 Now, it extends into ongoing relational communication undertaken on social networking (SN) websites, recently reported by 47% of online adults in the USA.2 Driven by steep uptake of general purpose social media, online health SN is emerging as a vibrant communication technology through which a distributed population of persons with a shared health problem or interest exchanges information and support through personal or community-broadcast channels.3 Engagement may offset the time and attention constraints typical of clinical encounters and provide important avenues for health communication, factors related to improved outcomes.4 5
Little is known about the quality and safety of health-related SN sites.6 Despite recognition that the internet in general and social media specifically are viewed by consumers as crucial resources for consumer health information,7 data are lacking on the value or impacts of these resources. No formal assessments have been done of alignment of site-published content with medical science and/or clinical practice recommendations (CPRs), accuracy of site- or member-published information, transparency of advertising content, and effectiveness of protections against misinformation and deceptive advertising. Recent studies of member posts and communications within the Facebook community suggest strong demand for emotional support and unsolicited exposure to promotional information including from advertisers.8 These risks are traditionally protected against by professional peer review and publication of formally sourced findings. Resources may not be available to support these checks within health-focused SN sites, and sites may eschew these controls in favor of organic peer exchange.9 A related concern is that health-focused SNs cater to vulnerable populations who may poorly understand or discount privacy risks under conditions of countervailing need for information and support. Thus, the accessibility and readability of site privacy policies and practices for centralized sharing of member data and opportunities for members to control data are vital. Poor practice in these areas may imperil patients and foreclose opportunities to develop SN for health.
To foster informed decision-making by patients and the clinicians advising them about SN engagement, we sought to characterize the quality of online health-related SN. We focused on sites organized around diabetes, a health problem of global significance,10 which imposes on patients heavy daily demands for monitoring and management that could be offset by SN.11
We undertook an observational study of the quality of diabetes-oriented SN websites. We defined SN sites as internet-based social systems that allow a distributed community of individuals to connect, communicate, and share information, and establish a stable personal identity or profile by which individuals can be recognized and connect to other individuals, groups, or collectives. We used a strict approach of reviewing and categorizing site contents using internet published information without requesting additional information or clarification from site administrators. This approach was deemed to result in review of information that would be most reflective of information generally available to users in a fashion that was least invasive to the community and members. Because the study design involved site masking with anonymity of data sources reported at the system level, the IRB approved it and exempted it from full review.
SN sites were identified through the Google search engine using ‘diabetes’ as a disease term and ‘social networking,’ ‘community,’ ‘virtual community,’ and ‘forum’ as network function terms. Site inclusion criteria in addition to a keyword match were: site falls within top 20 sites identified through the Google search; English language; no prohibitions on adult access; members can define a unique personal profile or self-description that persists over time; members can interact with other members directly. Sites that focused on multiple illness/health concerns (termed ‘pan health’ sites) were excluded. The review period was October 2008 to July 2010.
Review activities and analyses
Two trained research assistants reviewed sites using a structured protocol covering 28 quality indicators in four domains:
Alignment of content with diabetes science and clinical practice recommendations (CPRs): these indicators were included to characterize the extent to which site content was complementary to content of traditional authoritative sources and as a means of assessing the potential for synergy and reinforcement of information across sources.
Safety practices for auditing content, supporting transparency and moderation: these indicators were included to characterize strategies used by sites to monitor content for accuracy, misinformation and potential conflict of interest, recognizing that monitoring could take the form of external and/or internal review of centralized, member (consumer) authored, and commercial (advertising) information.
Accessibility of privacy policies and the communication and control of privacy risks: these indicators were included to ascertain the extent to which sites prioritized effective understanding of privacy risks among a broad consumer audience and informed consumers about site use of common safeguards for protecting personal information.
Centralized sharing of member data and member control over sharing: these indicators were included to foster comparison of site practices for sharing information and approaches taken by sites to providing members with opportunities to personally control sharing in the context of social network engagement.
Domain and measure specification served a goal of balancing quality indicators that reflect diabetes science with indicators that reflect the design choices, strategies, and opportunities within the SN medium. Indicators were selected that could be reviewed across sites, dichotomously coded and summed in thematic indices to facilitate within and across site review. Domains and indicators were specified prior to site observation activities, and operational definitions of indicators were developed through team consultation and review of published literature. Wherever possible, indicators were chosen that allowed the team to assess site content relative to standard metrics and measures (eg, definitions of A1c, or CPRs) and internet practice (eg, tools for protecting personal information). To minimize bias from secular changes in diabetes science or temporal variability in site content, thematic areas were reviewed near simultaneously across sites.
The first domain reviewed pertained to alignment of centralized site content with diabetes science and CPRs.12 We searched site areas designated for persons ‘new to diabetes’ or ‘newly diagnosed’ plus all centrally published content using each site's search function for presence of nine indicators: a medical disclaimer or text presented alongside editorial content that advised site users to consult with a healthcare professional about their care; clear central and correct definitions of glycosylated hemoglobin (A1c) and prediabetes; and text referring to each of six CPRs for diabetes care (routine checkups, annual eye exams, biannual A1c, yearly flu vaccine, annual lipid profile, and smoking-cessation recommendation).
The second domain reviewed pertained to safety practices for auditing site content and supporting transparency of advertising. Use of external resources to audit content was assessed (eg, the Health On the Net Foundation (HON) service,13 which charges sites for certification and checks sourcing but not accuracy of published content) as was presence on the home page of a link to an accredited or professional association at which users might reasonably expect to see vetted information. We assessed internal safety practices and resources considering support for formal or informal moderation of user-generated content and its effectiveness for addressing misinformation. We characterized the presence of moderation, clarity of the number of moderators and their credentials, and use of credentialed moderators. Moderation effectiveness was assessed by searching for the presence of member-published content related to availability of a ‘cure’ for diabetes and formal or informal moderation through removal or rebuttal. Once a week for three consecutive weeks, research assistants searched for the term ‘cure’ in the discussion section of each site and surveyed the presence and persistence of content promoting a cure for, or reversal to, diabetes within the five most recent discussions. Sites with member-posted content that did not have or removed/rebutted misinformation about a cure within the 3-week window were deemed ‘free’ of such content, while sites hosting discussions in which member-posted misinformation describing a diabetes cure was found that persisted (ie, was not removed or rebutted) were deemed ‘not free.’ Member-published content that referred to a wish for or anticipation of a cure or a similar out-of-context use of the term was not counted as misinformation. Given the potential for advertising activities to influence perception of disease risk or treatment choice,14 15 we assessed the presence of pharmaceutical advertising on the site home page, the transparency of advertising content as distinct from editorial content, and the safety of advertising content. The latter item was assessed by searching for the presence of misleading or deceptive advertising promoting a diabetes ‘cure’ or ‘reversal’ to diabetes using the above 3-week schedule to allow a review of time-varying safety risks.
All measures were coded dichotomously to facilitate within- and across-site review such that evidence of medical alignment, use of various mechanisms to ascertain safety and transparency of content, availability and readability of privacy policies, use of measures to protect disclosure of identified data, avoidance of centralized sharing of identified data, and options for member control over sharing were positively scored (coded as ‘1’), and the absence of these measures or counter practices was neutrally scored (coded as ‘0’). In this fashion, higher index values connote better practice. Findings are reported in thematically organized tables with by- and across-site values given for discrete indicators and thematic indices. A by-site comparison of quality considering all four thematic indices is reported as a summary graph to identify best-practice patterns.
General characteristics of diabetes-related online SN sites
Eleven sites met the study inclusion criteria. One closed during the review period and was excluded from the report for incomplete information. Sites have founding dates between 2001 and 2008. All address multiple diabetes subtypes (ie, Type 1, Type 1.5, Type 2, prediabetes, and gestational diabetes). All target adults, and one focuses on youth ages 13 and older and their adult care givers. The average community membership was n=6707. Member activity varied within a series of randomly monitored 24 h periods: four sites published over 100 new messages per 24 h period, while two sites were minimally active and published fewer than five new messages in that time. The level of site activity and membership size were not correlated. Seven sites were for profit. Three sites belonged to a professional association or organization. Across sites, several types of promotions were available to members, including coupons for diabetes supplies (four sites), information about clinical trials (five sites), and information about diabetes-related events (seven sites).
Alignment of site content with diabetes science and clinical practice recommendations
Across sites, there was a bimodal pattern of alignment of site content with diabetes science and CPRs, reported in table 1. Five sites had alignment indices with values in the top third of the nine-point scale, and three sites had values in the bottom third of the scale. Where a mixed pattern was evident (sites G, H, I), no clear pattern was present. Across sites, the lowest areas of alignment were for presence of a medical disclaimer or reminder to discuss care with a provider which was present on three sites; and publication on site of a definition of A1c, present on five sites, of which none defined A1c levels according to standards for effective disease management. Four sites provided recommendations for all six CPRs assessed; three sites had no information on these CPRs.
Safety of diabetes SN sites
A bimodal pattern was observed for safety practices (table 2); three sites' safety index values fell in the top third of the 10-point scale, and the remainder fell in or near the bottom third. With respect to external checks on the quality of published editorial content, two sites indicated that they subscribe to an outside agency that audits site content by ensuring content is sourced; however, the accuracy of published information was not verified under that service. Five sites publish a link to an outside diabetes organization on the home page, which may facilitate members' access to formally vetted diabetes information or guidelines. Internal checks on user-generated or member content through lay- or professional moderation were suggested in seven sites. Specific information about numbers or credentials of moderators was available for three of these sites. One site indicated use of professionally accredited moderation (ie, moderator has professional grounding in diabetes clinical science, nutrition, nursing, other). Member-posted misinformation about a cure for diabetes was found on six sites; four of these were moderated sites. Four sites were free of member-posted misinformation about a cure, of which three were moderated sites. Nine sites allowed advertising. One site had pharmaceutical advertising on its home page. Four sites clearly distinguished advertising from editorial content, and six sites were free of deceptive advertising about a ‘cure.’ Sites A, D, and H were identified as better-practice sites; they had transparency of content and no pharmaceutical advertising on their home page or deceptive advertising. Two sites lacked a clear distinction between advertising and editorial content and had advertisements promoting a ‘cure’ for diabetes.
Characteristics of privacy policies governing diabetes SN sites
Practices for protecting privacy of member data
Consistency of better practice
Across-site analysis of each of the four quality domains showed a clustering of better practice at three sites (A, D, and H) and is reported in figure 1. Within the observed range of site quality measures, sites that have high levels of alignment with diabetes science and clinical practice recommendations also have high levels of content safety, transparency, and protection from misinformation. None of the sites was completely aligned with diabetes science or published information on all six CPRs assessed, few used standard technical tools to protect personal information, and communication of risk and transparency of advertising were weak.
In this first study of the quality of online SN for health, we characterized the 10 most frequently referenced diabetes sites along 28 quality indicators pertaining to four practice domains. We documented variation in practice by domain and across sites to characterize better practice. Variability was evident, as was the ability of a minority of sites to offer greater quality and safety protection. With respect to medical alignment, safety, and transparency, notable gaps even among better practice sites were around publication of a medical disclaimer or recommendation reminding users to discuss their care with a provider, and provision of a link to a specific disease or professional association on their home page. Gaps may be oversights or intentional omissions. Lay-led and community-based networks may be choosing to amplify the community voice above the professional citation or resource. Alternatively, professional associations and the medical infrastructure may be ignoring these communities. Another notable gap area was around moderation, found among half of the sites. Information about moderation was scarce, and the presence of misleading content about a cure for diabetes raises questions about its effectiveness. Advertising content and pharmaceutical industry presence were observed on nine sites, although eight of these did not have pharmaceutical advertising on their home pages. Transparency around advertising content was missing on five sites, a possible red flag for misinformation, commercial influence, and even conflict of interest.17 Indeed, three sites contained advertisements promoting a ‘cure’ for diabetes.
The voluntary nature of SN, its disconnection from institutional authorities, and its alignment with patient/consumer-driven models and philosophies may seem to reduce our obligation for protective action to improve quality and support safety. Moreover, the diffusion of SN memberships over countries and cultures obscures locus of responsibility for doing so. Nevertheless, actionable steps for improving quality and protecting populations made vulnerable by illness are warranted and include:
ensure easy identification of where to get help, guidelines for care, and professionally recognized content within the larger set of broadcast information;
post information about the use of credentialed moderators and their number so users can be informed about protection;
enlist periodic external review of samples of member discussions to protect from misinformation and support effective moderation;
clearly and consistently flag commercial content and commercial members;
ensure that privacy policies are easy to find, clearly readable by the majority of healthcare consumers;
provide for member control over sharing of personal health information within and beyond the network;
use industry standard approaches to protecting individual health information and sharing defaults that emphasize protection of personal health information.
An effective system of site quality review is needed that encompasses the study domains and extends to periodic safety monitoring of published content given that this information is time-varying. Patient, patient advocacy, and medical communities should give consideration to the type of mechanisms that may be available to support cost-efficient peer or professional review. To be effective, a quality-improvement mechanism will need to balance values of individual autonomy and community ownership with conditions for innovation and consumer protection. Traditional institutional and regulatory approaches to overseeing patient safety and privacy poorly fit the online SN medium, and there is no clear authority or agency empowered to protect patients within this medium. Voluntary review by members of professional associations is one option that might be pursued as a service requirement for association membership. Another option is a peer-based review system in which members and leaders of a collection of SN communities ‘site visit’ each other to offer support and guidance around quality criteria. Sites that support these efforts and meet criteria might be flagged with a badge indicating participation in a quality-monitoring system. Such a system might inform users, contribute toward quality, and improve use of SN to ameliorate the effects of diabetes, a debilitating chronic illness with an expanding global footprint.
Strengths and limitations
This study adds to the growing literature on health-related SN by characterizing quality and safety issues at the system level. This is a new area with limitations. We report on observed practice and did not contact site administrators to obtain unpublished practice information. This allowed us to characterize apparent risks and protections as a user might encounter them in weighing possible benefits and harms. Actual protections and practices may differ and could be more rigorous if sites under-reported protective tools. We reviewed the most frequently referenced English language sites centered on diabetes using specific keywords. The full universe of diabetes sites or other health sites is not represented. We identified sites using the Google search engine, a site with broad consumer recognition and reach. Use of alternative search engines could result in identification of additional sites that were not included in this study, especially in light of the potential that websites could promote the visibility of their site on the Google platform through financial arrangements with Google. We used multiple measures to indicate practice along different dimensions, drawing on recognized standards where possible. This should protect against threats to the reliability and validity of findings; however, no gold standard for measuring the quality of SN practice for health yet exists.
Funding National Institutes of Health; Centers for Disease Control and Prevention (CDC); National Institute on Alcohol Abuse and Alcoholism (NIAAA). This work was supported by P01HK000016 and P01HK000088-01 from the CDC, by R21 AA016638-01A1 from the NIAAA, and by 1U54RR025224-01 from NCRR/NIH.
Competing interests None.
Ethics approval This study was conducted with the approval of the Children's Hospital Boston.
Provenance and peer review Not commissioned; externally peer reviewed.