Development and Psychometric Evaluation of the Impact of Health Information Technology (I-HIT) Scale

Introduction

Nurses have traditionally functioned in the role of communication coordinator in acute care settings.1 2 In this role, nurses have access to an abundance of data and information. The all-encompassing activities of hospital based nurses as direct providers of nursing care as well as integrators of care entail continuous contact with patients, family members and interdisciplinary providers and position nurses at the hub of communication. Point-of-care data and information are recorded in the patient record and are often transferred through other formal or informal channels.3 4 Nurses serve as information repositories that store, maintain and communicate data and information within and across processes of care.

The use of Health Information Technology (HIT) applications and tools in acute care settings to support communication processes and data and information access is a relatively new phenomenon. The adequacy of HIT for capturing and transferring data and information in hospital settings is undetermined at this time. A means to identify and describe the ways in which HIT influences communication and workflow patterns was needed. The purpose of this research was to design and psychometrically evaluate the Impact of Health Information Technology scale (I-HIT). I-HIT is an instrument designed to measure nurses’ perceptions about the ways in which HIT influences interdisciplinary communication and workflow patterns as well as nurses’ satisfaction with HIT applications and tools currently available in hospitals.

Background and Conceptual Framework

Evidence suggests that face-to-face, “real-time” or synchronous communication is the preferred mode of communication among health care providers.5 6 However, the interruptive nature of synchronous communication may contribute to communication breakdown and medical error.7 8 Communication breakdown is a principal impediment to patient safety in hospitals. Lack of effective communication to mitigate errors is the most common root cause of adverse events in hospitals. Approximately 70% of the 3,661 Joint Commission on Accreditation of Healthcare Organization (JCAHO) reported sentinel events for the 5-year timeframe of 2000–2005 were attributable to communication failure.9 Barriers to effective communication can precipitate errors of omission (when actionable information is not transferred) and errors of commission (when information is incorrectly perceived). Sutcliffe et al. (2004) found that communication failures were identified by medical residents as a primary contributor to medical mishaps.10 In focus group interviews with physicians and nurses, significant communication barriers were found to exist in acute care settings, leading to diminished access to information whereby incomplete or inaccurate information resulted in medical errors.11 In fact, only a small fraction of preventable adverse events in hospitals are attributable to procedures or specific treatments for which the patient is hospitalized.12 A majority of adverse events occur in the context of general care processes and poor communication between providers was identified as a probable contributing factor.12 In addition, one-third of poor outcomes in the operating room (delays and inefficiencies, team tension, and errors) were related to communication failure.13

Because communication failures are a common cause of unintentional patient harm, it is important that systems are developed with the natural limitations of human performance in mind. A large and ever present cultural barrier in health care is the belief that quality of care and error free clinical performance is the result of being well trained and trying hard. Health care professionals, including nurses, using this model may view inevitable mistakes as an occurrence of personal failure. Human factors research tells us that the inherent limitations of human memory, the complexity in which nurses make decisions about patient care, the effects of stress and fatigue, the risks associated with distractions and interruptions, and limited ability to multitask ensure that even skilled, experienced providers will make mistakes.14

Cognitive artifacts15 are physical objects that nurses create and use to manage their work environment. Artifacts within nursing are human-made objects that reflect both professional and workplace characteristics such as values, norms, symbols, and rituals. This includes the use and placement of communication tools such as checklists, shift report logs and documentation templates that are employed by nurses and used within typical care delivery routines. As HIT adoption becomes more common in hospital settings, technology is now employed within nursing workflows to represent the physical artifacts traditionally used to promote best practice. In this regard, the contemporary inclusion of HIT in nursing practice holds potential to be a natural, seamless phenomenon. Therefore, it is important that design and intended use of new technologies address and support functional requirements of physical artifacts within nursing practice that are replaced or disrupted by new HIT.16 One way for new technology to support collaborative work is by embedding information and communication technologies (ICT) into the existing infrastructure of physical artifacts so that they support both interdisciplinary communication and the complexity of acute care workflows.

Well-designed and well integrated electronic systems reduce communication errors through decision support features and ubiquitous access to data and information at the point-of-care.17 However, medical errors associated with poorly designed systems are well documented and serve as a reminder that effective support of interdisciplinary communication requires electronic systems crafted to facilitate access to information, interdisciplinary communication, and dynamic patient care workflows.18 19

The effect of health information technology (HIT) applications and tools on the role of nurses and interdisciplinary communication is not well understood. As health care organizations embark on the transition from paper-based to electronic systems, it is necessary to develop ways to explore the impact of HIT on nursing practice and interdisciplinary communication in general and specifically on the nursing role of communication coordinator.

In addition to providing bedside care to patients, the registered nurse has traditionally served as patient care coordinator and integrator.2 Within these roles, nurses function as patient advocate and intermediary between other professionals. The ongoing assessment and monitoring work that nurses do and the immediate connections that nurses have with patients, family members, and providers serve to protect patients through early detection of changes in health status20 and through medical error recovery.21 22 Well-designed HIT applications and tools with decision support and interdisciplinary communication features have the potential to improve patient safety by assisting nurses with improved monitoring, capacity for early detection of potential problems, and ease in communicating changes in status to the interdisciplinary team.

The Communication-Information Continuum

Coiera’s Communication-Information Continuum (C-IC) (2000) provided the conceptual underpinnings for this instrument development and psychometric evaluation study (see Fig. 1).23 The key components of the C-IC are the Communication Space (vertical axis) and the Task Space (horizontal axis). Information exchange takes place in the Communication Space. The shared understanding or “common ground” that exists between communicating agents determines Communication Space. Factors likely to increase common ground are similar educational preparation, domain of practice, and socio-economic status. When common ground shared by communicating agents is increased, formalized or structured communication interactions are improved. Therefore a higher degree of common ground enables communication processes that can be simulated in HIT applications and tools.

Communication Space includes the continuum from individual space (minimal shared understanding exists between communicating agents) to shared space (high degree of understanding between agents). Communication tools (e.g., face-to-face or telephone communication) support effective communication in the individual space where shared understanding is limited. Therefore, the interactive dialogue that occurs in individual space contributes in a significant way to understanding complex patient care and treatment issues. In contrast, information tools (i.e., HIT applications and tools) facilitate communication in shared space where a high degree of shared understanding exists and personal interaction is unnecessary.

The Task Space includes the continuum in which tasks range from communication tasks to information tasks. Communication tasks are unstructured (informal) and not mediated by electronic systems. Rather, face-to-face and other forms of synchronous communication are the typical method of information exchange. Representing communication tasks in information systems is not feasible because the interactive dialogue associated with communication tasks plays an important role in communication and developing a shared understanding of the problem. However, information tools, such as wireless voice messaging systems, can effectively support (not replace) communication tasks within the context of acute care workflows.

Information Tasks are structured (formal) and easily mediated by computer systems using a combination of synchronous (real-time) and asynchronous methods (e.g., intermittent or not real-time). An overarching goal of Coiera’s model is to help HIT developers to determine the best tool set to facilitate communication and information transfers at specific stages of individual and shared understanding. The C-IC provides a useful framework for evaluating the adequacy of HIT in acute care settings and offers a means to predict the types of communication-information interactions most likely to benefit from automation.

Procedures for Instrument Development: Methods and Results

Methods

Item Development

Items for the Impact of Health Information Technology (I-HIT) scale were generated from a critical review of the literature and focus group style interview of nursing informatics content experts (n = 5). The literature review included papers found in MEDLINE (1966 to present) and CINAHL (1982 to present) databases using the following search terms: HIT applications and tools, healthcare communication, and the professional nursing role. In addition, bibliographies from selected papers were reviewed for additional papers of interest. The review of literature was summarized and analyzed by two members of the research team (PCD, MC). Key themes and sub-themes were identified by team members separately and then confirmed jointly. The following key themes emerged from the literature review and were confirmed through analysis and discussion:

• 1. HIT as a facilitator/barrier to interdisciplinary communication

• 2. HIT as a facilitator/barrier to the nursing roles of communication coordinator and integrator of care

• 3. HIT as a facilitator/barrier to the nursing process

• 4. Unintended consequences of HIT

Using Coiera’s work and the C-IC as a conceptual framework, the key themes were employed to develop a semi-structured interview guide (see Fig. 2).

Telephone Focus Group

Five content experts associated with the national Health Information Management System Society (HIMSS) Nursing Informatics Task Force and with 15 or more years of experience in health care informatics, participated in the focus group interview conducted via conference call. Informed consent was obtained from participants and the interview guide was distributed via email one week before the scheduled call. The group leader (PCD) began the focus group interview by reviewing the purpose, ground rules, and operational definitions. To facilitate, the group leader asked a question and then gave each participant a chance to respond. The order in which participants were asked to respond was rotated with each question. The session was audio taped and transcribed verbatim.

Transcript Analysis

Two nurse informaticians with training in qualitative methods (PCD, MC) participated in analysis of the transcripts. The interview data were transcribed verbatim into a word processing package, reviewed, corrected for transcription accuracy and removal/masking of any identifying characteristics, and converted into NVivo for coding and support of qualitative analysis. Coding allowed us to retrieve, organize, and ultimately classify segments of text24 for developing descriptive categories. We then followed a process of basic content analysis of narrative data25 in which an independent review of the transcript was conducted by two research team members who identified potential scale items. Units of information were compared to other new pieces of information in the same and different categories. Findings were then integrated into common themes and sub-themes with descriptions and sent out to the content experts for review, clarification, and comment. Comments from experts were reviewed and incorporated into the common themes, sub-themes, and descriptions so that content was consistent with expert feedback.

Results

Item Development

Using the C-IC as a framework, five key themes emerged from qualitative analysis of focus group transcripts. Operational definitions confirmed by the content experts follow. (1) Interdisciplinary communication is an interactive process characterized by sharing and interpretation of information between and among disciplines. (2) Information access is the degree to which information is available to nurses at the point-of-care. (3) Practice effectiveness and efficiency means the degree to which HIT applications and tools facilitate patient-centered care. (4) Interdisciplinary relationships mean perceived ability of HIT applications and tools to augment or support patient-centered communication across disciplines. (5) Workflow implications of HIT are positive or negative impact of HIT on caregiver ability to provide acute inpatient care.

Item Validation

Fifty survey items were abstracted from transcript analysis sub-themes, were grouped under the five key themes, and subject to content validity assessment by the expert panel. Following an established method,26 27 five content judges (the original focus-group participants) used a two-point scale to rate each item and its description for relevance with the operational definition of the theme under which it was placed. Forty-three of the 50 items were scored as relevant by all five judges and were retained to provide a content validity index (CVI) of 1.0. In terms of the quantification of content validity,27 the 43 items achieved a CVI beyond the 0.05 level of significance and were retained.

Fifteen hospital-based nurses from the HIMSS Nursing Informatics Task Force read and responded to each item and provided feedback regarding face validity. For the I-HIT to be a valuable measure, it must be considered reasonable by nurses who would respond (or not) to individual items. We used the criterion of credibility28 to learn empirically if the items were acceptable to nurses and how they could be improved.

Recommendations for simplification of language and additional examples to clarify items were incorporated. The resulting 43-item scale was related to perceptions of nurses regarding the ways in which HIT influences interdisciplinary communication and workflow patterns and nurses’ satisfaction with HIT applications and tools. The 43-items under the five key themes were arranged as follows: 1) Interdisciplinary Communication: 10-items, 2) Information Access: 8-items, 3) Practice Effectiveness and Efficiency: 11-items, 4) Interdisciplinary Relationships: 5-items, and 5) Workflow Implications of HIT: 9-items.

Psychometric evaluation: Methods and Results

Methods

The I-HIT scale targets nurses practicing in acute care settings. The study received institutional review board approval and the I-HIT was posted to the Internet using the Survey Monkey software package. Survey Monkey is web-based survey development software that offers two levels of service subscription; basic (no charge) and professional (monthly fee for access). Both options are available at http://www.surveymonkey.com. The professional subscription was required for access to the advanced design and logic features necessary for construction of the I-HIT survey. The functionality of the Survey Monkey Software offered several benefits to this project: 1) The predefined questionnaire templates were easy to use and shortened development time. 2) The Web-based survey and response tables were hosted by the vendor, therefore, a separate server was not needed. 3) The survey was made available and thoroughly tested and evaluated by research team members before dissemination. 4) A central dashboard facilitated tracking and downloading of responses into a format that can be used with most statistical software programs (i.e., .xls, .sav, .txt). The main disadvantage of the software was that the basic survey templates constrained the survey format and lessened the visual appeal of the survey. However, some of these limitations can be overcome with a basic knowledge of HTML, as additional code can be inserted at the time of survey construction. The disadvantages of using Web-based surveys in general are the potential limitations on the generality and validity of the results.29 Web-based surveys limit respondents to those who have access to the Internet and those with sufficient computer literacy skills and time to complete a survey online. In addition, a single respondent is able to complete the survey multiple times. While Survey Monkey software does include functionality to limit responses to one response per IP address, we did not activate this feature, because it would have prevented different respondents from completing a survey from the same computer at a given hospital.

Sample and Procedures

Participants were recruited via email and healthcare-related listservs using a non-probability snowball sampling technique whereby HIMSS NI community members e-mailed contacts working in acute care settings or who managed healthcare-related listservs and asked the contacts to pass the link on to others with contacts in acute care settings. An inherent disadvantage of using this sampling method is that we were unable to determine how many individuals received a request to complete the survey and therefore we were unable to calculate a response rate.29 The I-HIT was formatted as a 43-item web survey organized under the five key themes. Respondents were asked to self-report their beliefs on scale items, reflecting their own experiences with HIT applications and tools currently available in their work setting. A 6-point Likert rating scale was used with values ranging from 1 = “Strongly Disagree” to 6 = “Strongly Agree.” Negatively worded items were reverse scored. Higher I-HIT scores indicate perceived agreement regarding the impact of HIT applications and tools on nurses’ care of hospitalized patients, on the bedside work of nurses, and on interdisciplinary communication. An additional item measuring overall satisfaction with HIT applications and tools currently available in hospital setting (OSI-HIT) was added to examine concurrent validity. We hypothesized that a high rating on the OSI-HIT, would correlate positively and highly with high scores of the I-HIT. OSI-HIT is rated on a 9-point Likert-type scale ranging from 1 = “Completely Dissatisfied” with a midpoint of 5 = “Neither Satisfied Nor Dissatisfied” to 9 = “Completely Satisfied.”

Analytic Methods

Data were downloaded from the Survey Monkey Web site as an Excel file and uploaded into the Statistical Package for Social Sciences Software (SPSS), version 14.0. Psychometric evaluation of the Impact of I-HIT included: 1) item analysis, 2) principal components analysis, 3) internal consistency reliability using Cronbach’s alpha coefficient, and 4) construct validity evaluation.

Results

Sample

Seventeen hundred and sixty (1,760) repondants (all disciplines) answered the web survey from July 15 through September 30, 2005. The sample for the PCA consisted of all registered nurse respondents who had less than 10% missing data. Therefore, of the total sample of 1,760, 681 were dropped at this point, resulting in 1,079 cases. An independent samples t-test was conducted to evaluate differences in total satisfaction with HIT scores for respondents with no data missing and for respondents with less than 10% missing data. No significant differences in overall satisfaction were found between the groups (t = 0.03; p = 0.98), suggesting that occurrences of missing data are random. In the statistical analysis, the “exclude cases listwise” options were selected in SPSS so that all cases with a missing value on any of the test variables or on the grouping variable were excluded from the analysis. Survey respondents were mostly female (91.3%), over 40 years old (73.1%), direct care providers (48.5%), employed in nursing over 20 years (51.1%), currently employed at a medical teaching hospital (68.1%), and were well educated (72.6% held a bachelor’s degree or higher). The sample size of 1,079 provided more than 10 respondents per item, which was adequate to undertake a principal component analysis.30 31 32

Psychometric Evaluation

Internal consistency reliability analysis using Cronbach’s alpha was conducted on the 43-item scale to initiate the item reduction process. Item-total statistics were examined and items with an item total correlation of less than 0.35, indicative of not contributing to an internally consistent scale, were deleted (N = 11). The Cronbach’s alpha analysis was repeated on the remaining 32 items and revealed one additional item with an inter-item correlation of less than 0.35 and two items with inter-item correlations greater than 0.8 (indicating redundancy). These three items were removed, leaving a total of 29 items. Data were found to be suitable for PCA with a Kaiser-Meyer-Olkin value of 0.96, exceeding the recommended value of 0.60. Bartlett’s Test of Sphericity was also very significant (p < 0.000), supporting the factorability of the correlation matrix.32

Principal components analysis (PCA) with Varimax rotation and Kaiser Normalization was applied to data with a cutoff difference at < 0.30 for determining which items make a significant contribution to the factor. The PCA revealed the presence of five components with Eigenvalues exceeding one, explaining a total of 62% of the variance, consistent with the theoretical prediction. However, upon examination, the 5-component solution, although parsimonious, was not easily interpretable. Thus, the scree plot for 4-components was explored. The scree plot leveled off after the fourth factor, suggesting a four-component solution. A subsequent Varimax rotation restricted to four factors revealed the presence of strong loadings on each of the four components. Seventeen items had significant side loadings and those items were kept with the factor with the strongest loading that was conceptually congruent with the statement. Factors with side loadings are indicated by a double asterisk in Table 1. The four factors that emerged from PCA explained 58.5% of the total variance and are consistent with the definitions of components that make the Coiera model operational. Thus, the four-factor solution was accepted as parsimonious and interpretable (see Table 1). The reliability estimate using Cronbach’s alpha coefficient to measure internal consistency for the confirmed 29-item scale was 0.95. The reliability of the subscales ranged from 0.80 to 0.89. Psychometric properties and variance are outlined in Table 1.

The first subscale, Workflow Implications of HIT, accounts for 16.1% of the variance. Items are related to facilitating patient-centered communication through ubiquitous access to data and information within the context of acute care workflows. The second subscale, General Advantages of HIT, explaining 14% of the variance, includes items that reflect the degree to which Communication Space (i.e., common ground) and Task Space (i.e., type task) inform development of existing HIT applications and tools that support nurses in the communicator and integrator roles. The Information Tools to Support Communication Tasks subscale, explaining 12.6% of the variance, includes items that reflect the availability and use of HIT to optimize communication and support the work of nurses in the Communication Space where less common ground exists and therefore electronic tools are used to support more traditional synchronous approaches. Finally, the Information Tools to Support Information Tasks subscale accounting for 12.1% of variance includes items that relate to the appropriate use of information tools to promote efficient communication where a high degree of common ground exists among communicating agents.

Additional Validity Evidence

Concurrent Validity Analysis

Concurrent validity was assessed by running correlations between mean total subscale scores and OSI-HIT. The research team hypothesized that higher scores on the four subscales would indicate perceived agreement that HIT applications and tools have a positive impact on the work of nurses and interdisciplinary communication. Moderate to large correlations were noted between OSI-HIT scores and the four I-HIT subscales (Table 2), providing support for criterion related validity.

Discussion

Measurement is an important step in achieving quality.33 The I-HIT scale was developed to provide a means to measure nurses’ perceptions about the ways in which HIT influences interdisciplinary communication and workflow patterns as well as nurses’ satisfaction with HIT applications and tools currently available in hospitals. The I-HIT scale is consistent with the C-IC conceptual model, performed well in initial testing and has adequate psychometric properties for continued use in acute care settings to measure the impact of HIT. The internal consistency estimates, PCA and validation data of the I-HIT scale and the four subscales support the psychometric adequacy of the I-HIT for use in both research and performance improvement activities. The four subscales further support factorial validity of the I-HIT scale and the internal consistency reliability estimates suggest that the subscales are suitable as independent measures of this construct. Due to the unequal numbers of items in each of the I-HIT subscales, we advise using average scores so that each of the subscale scores have equal weighting.

I-HIT items were generated through a critical review of the literature and analysis of focus group interview transcripts of nursing informatics experts using Coiera’s C-IC as a conceptual framework. Content and face validity were established through evaluation of items by nursing informatics experts and pilot testing of I-HIT by bedside nurses. Criterion-related validity was established in relation to an overall measure of satisfaction with HIT (OSI-HIT) administered along with the I-HIT scale.

Psychometric evaluation is an important precursor to using the I-HIT scale for research. Demonstrating the I-HIT’s psychometric adequacy provides a foundation for future work where the I-HIT scale may be used to learn about the ways in which HIT influences interdisciplinary communication and workflow patterns as well as nurses’ satisfaction with HIT applications and tools currently available in hospitals. Research on the impact of HIT at the organizational and global level is critically needed. Despite potential benefits, a recent national survey of information technology executives found that only 13% reported that a fully functional electronic medical record (EMR) was in place at their facility.34 The slow rate of EMR adoption suggests that existing HIT applications and tools may not yet be compatible with and integrated into hospital communication patterns and dynamic patient care workflows.

The literature on HIT and Coiera’s C-IC model both demonstrate that the preferred mode of communication in health care has been face-to-face and other synchronous methods.3 4 5 6 8 Therefore, incorporation of electronic methods of communication must be accompanied by a careful assessment of the role that personal interaction plays in information transfer and problem solving. Coiera’s C-IC model provides a useful framework when evaluating the adequacy of existing HIT applications and tools in acute care settings and offers a means to predict the types of communication-information interactions most likely to benefit from automation.

The I-HIT scale should be administered to nurses working in direct care positions and will offer useful information regarding the adequacy of existing HIT applications and tools in acute care settings. Over time, it is hoped that I-HIT will provide a means to track improvement and progress with HIT applications and tools with regard to the degree to which the HIT infrastructure is supportive of the role of nurses and interdisciplinary communication. Because the theoretical foundation of the I-HIT is the C-IC, this instrument provides a means to evaluate where on the Communication-Information Continuum organizational HIT strengths and weaknesses lie. Therefore, the I-HIT scale could be used, not only for research, but the items could provide useful performance improvement data related to specific areas along the Communication-Information Continuum within organizations or hospitals that might need improvement. These data may assist organizations with evaluating nurses’ perceptions of the degree to which changes in the HIT infrastructure are supportive of the role of nurses and interdisciplinary communication. In addition, the I-HIT scale may be valuable for HIT planning and improvement because it will provide health care executives with data that can be used to negotiate with vendors and promote improvement of existing HIT applications and tools.

Caution should be exercised when interpreting the findings of this psychometric evaluation because there are several limitations of the I-HIT scale. The survey is self-report and non-probability sampling methods were employed to recruit participants. While respondents who answered the Internet survey reported that they were acute care or hospital-based nurses, only about half (48.5%) reported that they were direct care providers. The sample may not be representative of all bedside nurses. Therefore, continued testing is warranted to evaluate the stability of I-HIT and its component structure over time and across various acute care settings.

Conclusion

The use of health information technology (HIT) for the support of communication processes and data and information access in acute care settings is a relatively new phenomenon. A means of evaluating the impact of HIT in hospital settings is needed. The I-HIT scale is designed to measure the perception of nurses regarding the ways in which HIT influence interdisciplinary communication, workflow patterns and the degree of satisfaction with HIT applications and tools. Findings from this study suggest that the I-HIT and its four subscales are a promising measure of attitudes towards the impact of HIT on the role of nurses and interdisciplinary communication in acute care settings.