Desiderata for Personal Electronic Communication in Clinical Systems

Methods

We used an electronic-mail version of the Delphi technique.23 This was done in part because of our panelists' disparate geographic locations and the convenience of electronic mail messaging.23 We contacted 16 people who were affiliated with the IDEATel project; 10 (one nurse, one administrator, and eight physicians) agreed to participate. These 10 people made up our expert panel. Four of the panelists had significant experience in medical informatics, in that they were either faculty in informatics departments or were actively involved in informatics projects. Neither author was a member of the panel.

The study was part of a system selection process. It was begun prior to selection of the IDEATel PCEC system and was completed after selection, but prior to installation, of the system.

The elicitation process began with a request that each panelist, independent of all the others, present a list of features that he or she believed were important in the selection of a PCEC system, based on their personal experience. To maximize the generalizability of the resulting features, we asked the panelists to consider general clinical practice and to not limit themselves to the context of the IDEATel project. We also asked the panelists to not worry about any of the technical limitations that may currently exist for selection of what they thought of as an ideal system.

For this and each subsequent round, the panelists had two weeks to respond, and all responded on time for all rounds. We intentionally did not “seed” the list with features from existing products or prior published guidelines. The goal was to product a user-centric list that could be compared with lists derived by different methods.

We combined all the responses into a single, unaccredited list, which we then returned to all the panelists. We asked the panelists to add new features—and refine any of their own features—that they felt needed to be included on the list. We recompiled the results and again returned them to the panel.

During the third round, we asked the panelists to rank the features. Using a ten-point Likert scale,24 ranging from −5 (most detrimental) to +5 (most essential), they were asked to quantify each feature in the set in terms of its benefits or detriments to a hypothetical, ideal PCEC system.

In the fourth round, we sent the panelists a list that included, for each feature, a mean score and that panelist's individual score; we asked them to adjust their individual scores if they wished, based on the overall means. As is standard in the Delphi method, direct discussion among panel members was discouraged, to mitigate the influence of forceful or highly respected persons. After we updated the mean scores, we again sent the list to each panelist, with the mean and most recent individual score and a request that he or she update the score, if desired. No panelist changed his or her individual score in this round, so the collection process ended.

We removed duplicate items from the final list. The criteria for concluding that two items were duplicates included not only that they appeared to describe the same feature but also that the were scored the same by the panel. Remaining features were grouped into thematic categories. Within each category, the items were ranked in descending order first by mean, then by standard deviation. The categories were then collectively ranked also by mean and standard deviation.

Further inspection of the feature set lead to the development of three categories of features, based on mean rank and inter-rater agreement. Inter-rank agreement was ascertained by examining the standard deviations of each feature. The “critical” features shared a mean greater than or equal to 3.5 and a minimum score greater than or equal to 0. The equivocal features shared mean less than 2.25 and a minimum score less than 0. Between these two groups are the “desirable” features.

We made no attempt to enforce uniform syntactic structure on the submitted features; i.e., some features were phrased in the positive (“Strong Authentication) and some in the negative (“No SPAM”). We considered the effect of such phrasing on the scoring by looking at the relative scoring of positive and negative features. In particular, we looked for phrases in which a statement was negatively set and then the grouping it was placed in. Because of the nearly bimodal distribution of scores for one negatively phrased feature, “Recipient Disclosure,” we suspected possible misunderstanding. We contacted our panel for written descriptions of their answers. Eight panelists responded, and the “Recipient Disclosure” scores were recalculated on the basis of just these responses.

The features and associated scores were converted to a numeric evaluation method for PCEC systems. The evaluation method was based on two components—a weighting value, based on the median score from the panel, and an implementation value from 0 to 1.

Each feature was individually considered for a candidate PCEC system. If a feature was fully supported, an implementation score of 1 was assigned. Those that were partially supported were given an implementation score of 0.5. The score for each feature of the candidate system was determined by the implementation score (0 to 1) multiplied by the weighting score (0 to 5). Only critical and desirable features were used for the score.

As a proof of concept, we applied the evaluation to two potential implementations of conventional electronic mail and to three existing Web-based PCEC systems—Healinx2 (Healinx Corporation, Alameda, California), the PCEC supplied by the American Medical Association to its members4 (Medem Inc., San Francisco, California), and the PCEC system selected for the IDEATel project (Siemens Health Services, Malvern, Pennsylvania).

Results

In the initial round of the Delphi method, our panelists submitted 29 criteria; they added 5 criteria in the second round. We then removed two duplicate entries, for a total of 32 criteria. These criteria were sorted into 21 categories, which are shown in Table 1.

A total of 318 scores were received for the 32 criteria, across the full possible range (−5 to +5). The overall mean, median, and mode were 3.3±0.7 (SD), 4, and 5, respectively. Using the rank definitions and the inter-rater agreements, our final list consisted of 16 critical features (mean, 4.3±1.0), 13 desirable features (mean, 2.7±1.8), and 3 equivocal features (mean, 1.5±2.1). The scoring for each feature is summarized graphically in Figure 1.

The numeric evaluation based on the feature list revealed that conventional e-mail scored between 19 and 33 percent overall. The range of scores for conventional e-mail was dependent on the number of features that were available for different implementations of e-mail. This scoring assumed partial credit for features, like digital signature, that are not supported by all e-mail readers. Without these features, the electronic mail scores dropped to 19 percent, for both the critical and overall scores. In comparison, all three Web-based PCEC systems scored markedly better. Scores ranged between 43 and 62 percent for all features and between 49 and 75 percent for critical features.

Discussion

The most striking finding was the level of consensus among panelists in different specialties from different institutions. Across all the features, the average standard deviation of the group was 1.3 units on a 10-point scale. This consensus was especially strong for the 16 features in the “critical” group (1.1 to 1.6, 2 to 8.3, and 9.1), in which the average standard deviation was 1.0. Nine features had median scores of +5.

Further evidence of consensus was the fact that no feature submitted by an individual panelist was ranked as undesirable (average score less than 0) by the panel. In spite of this agreement among clinicians, the highest-rated PCEC system scored only 75 percent for these features. More important, conventional e-mail, which is probably the most widely used PCEC tool today, scored only 40 percent for critical features in an optimal implementation and only 19 percent in a more typical implementation.

Although the number of implementations evaluated was limited, every Web-based PCEC evaluated out-scored conventional e-mail.

These features are not the first collection of recommendations for electronic messaging. The features selected by the panel compare closely with those included in previous guidelines on e-mail use, including the importance of message archiving, encrypted transport, and documentation of when messages are actually read.11

Of the 16 criteria ranked as critical in our study, 13 were also relevant to the HIPAA requirements.25 These criteria (1.1 to 5, and 8.1 to 8.3) all deal with the security and reliability of messaging between various health care players, such as clinicians among themselves and with their patients. This was a reflection of how the critical elements, on the whole, are essential to a PCEC system.

Strikingly absent from the list of features are many that are typically discussed in relation to Web-based sites for patients. These include statements regarding the limits of liability3 or limits on the types of messages that are permitted.16 Also absent are features relevant to developers and implementers. For example, there is no mention of open application program interfaces or the use of standards, even though both are critical for the integration of PCEC into complex clinical computing environments.22

Similarly, disclaimers and controls to limit legal liability are not mentioned.16 This is because these features reflect the views of individual clinical users rather than the views of developers or institution administrators. It also reflects the focus on messaging among individuals, not on broad communication (e.g., Web sites). As a result, these features complement, but are not a substitute for, good system and user interface design. The requirements of local clinical practice, pre-existing electronic medical record systems, HIPAA, and other regulations (such as federal Section 508 guidelines26) will influence the final design of a PCEC implementation.

Although this list of features represents a start at addressing criteria that should be used in the selection of a PCEC system, this study is not without its limitations. The number of panelists included in the study was small. With such a small sample, it is conceivable that areas of disagreement were not fully expanded and areas of agreement were not properly extracted from the data.

The panelists came from two separate institutions but were all affiliated with a similar project, and most had some experience with electronic medical record systems. However, because the IDEATel PCEC was not functional at the time of this survey, this is not likely to have induced significant bias. Also, this study focused on clinician users. Further studies will need to assess the preferences of other groups, such as patients, vendors, and other non-clinician users.

Finally, in the numeric evaluation, the weighting values for individual features may not translate directly to the aggregate value of classes of features. For example, the large number of security features (6 of 16 critical features) may lead to overestimation of the aggregate importance of security in PCEC systems.

The “critical” features may be the most important for system developers. However, equally interesting are the “desirable” and “equivocal” features, which elicited more disagreement among the panel. The desirable group (features 1.7, 9 to 17, and 18.1) is distinguished from the critical group by both lower mean scores and higher average deviations. Com-pared with the critical features, these features share the characteristic that their value to a clinician may be dependent on the functionality of other computer systems at the institutions.

For example, a calendaring system may be essential at one institution but of no particular use at another, where an efficient scheduling system is already in place. Structured messaging, which provides templates for certain message types (e.g., medication renewal and appointment request), could act as a surrogate for a forms or order entry system if one did not already exist. Threaded message viewing would allow the messages pertaining to a specific patient to act almost as an electronic medical record. Thus, the importance of these features could be expected to vary among individuals and among institutions.

Institutions selecting PCEC systems will want to pay particular attention to those features that receive both beneficial and detrimental ratings from the panel. This group included all three equivocal features, as well as five features from the desirable group, including automatic timeout, due-date escalation, chat, levels of importance, priority preservation, separation of roles, asynchronous alerting, and VIP messaging. Each of these features was also rated +4 or +5 by at least one panelist, indicating that some panelists felt these features to be highly desirable. One feature, levels of importance, received scores ranging from +5 (must have) to −5 (must never have). This level of disagreement did not decrease in subsequent Delphi rounds.

Overall, these features tended to involve additional effort, interaction, or liability by clinicians in the way they practice medicine. Both chat and asynchronous alerting involve the use other communication means, such as paging or pop-up windows, that may interrupt the clinician. Paging systems are often seen as the bane of life by hospital clinicians, and many clinicians may be hesitant to permit computers to page them directly. Computerized paging may also affect clinical care, when the sensitivity is set too high and an excessive number of alerts are sent to clinicians.27

Four features in this subset involved the creation of special high-priority messages (due-date escalation, levels of importance, priority preservation, and VIP messaging). Levels of importance and priority preservation involve the ability of a sender to flag a message as high priority and retention of the flagging by the reply message. Most clinicians have encountered coworkers who label everything “STAT.” Consequently, some may wish to limit that behavior in a PCEC system.

With the VIP feature, messages pertaining to persons designated VIPs are associated with additional security and auditing procedures. In practice, most hospitals already have special procedures for protecting the security of both the paper and the electronic records of VIP patients. These may include admitting the patient under an alias or removing the name label from the paper chart. This practical need conflicts with the ethical imperative to treat all patients equally, regardless of wealth or status.

The case of receipt disclosure/no BCC (blind copying) is also of note. Originally, the panel generated a nearly bimodal distribution of strongly positive and negative scores. Subsequent queries to the panel revealed that there was confusion about whether a negative score meant that blind copying was bad or that preventing blind copying was bad. After clarification, the scores still had a somewhat bimodal distribution, but they were now clustered about 0 and +5. Panelists who gave high scores indicated that full disclosure was imperative if patients were to trust the system. Panelists who gave low scores indicated a conflict between the need for disclosure and the fear that full disclosure might also disclose patient names and thereby violate patient privacy.11 This is an example of a feature that is very different when viewed from clinician and patient perspectives.

It is clear from these desiderata that no single PCEC system will satisfy all the wants and needs of all clinicians or institutions. However, they also show that a high level of consensus about the critical features that should be included in future PCEC systems.