Table 1
Methodologic Issues and Recommendations for Health Informatics Trialists
| Methodologic Issue | Recommendations |
| 1. Intervention design | Use a pragmatic randomized controlled trial design if the study aims to determine effectiveness of a health informatics intervention. The following factors will determine the choice of specific design, such as two-group parallel, cross-over, ‘early versus delayed’, factorial, or cluster-randomized trial: 1) research question (HI intervention versus control, one intervention versus another), 2) type of intervention (CDSS, computer-generated reminder, computerized physician order entry), 3) outcome measures (patient-related endpoints, provider performance, economic analysis), 4) participants, 5) setting (primary care, tertiary care, university-affiliated), 6) length of a follow-up period (short-term versus long-term). |
| 2. Choice of randomization | Select individual-level versus cluster-level randomization considering the following factors: potential and magnitude of contamination, unit of analysis in the study (e.g., patients, physicians, hospital wards), feasibility (e.g., ability to recruit a sample size large enough to adjust for clustering, cost, ethical considerations), and existing workflow (e.g., availability of electronic prescribing in one unit versus the whole hospital). Sample size calculations and statistical analysis should be adjusted for clustering effect; also, special care should be taken to prevent possible selection bias with cluster randomization. |
| 3. Allocation concealment | Allocation concealment at the time of randomization should be done by using adequate methods (e.g., by an individual not otherwise involved in the trial). This maneuver is critical for ensuring the validity of study results, and is always feasible. |
| 4. Blinding of subjects | Determine whether designs such as ‘early versus delayed’ or factorial can be utilized to blind study participants. ‘Partial blinding’ can also be used. If blinding of participants is not possible then incorporate blinding of outcome assessors, data collectors, statisticians, or other strategies to ensure comparable experience between study groups. Use of information technologies can be used to assist with blinding. |
| 5. Components of a complex intervention | Identify and describe the active components within your HI intervention (e.g., computerized physician decision support, patient alerts, provider education), and predict the mechanisms by which these will contribute to the overall success of the study. Make sure all components are described in sufficient details. |
| 6. Sample size and power | Plan ahead and calculate sample size and power for your HI trial, using the results of a pilot study or past similar interventions. Generally, patient-related clinical outcomes require larger sample size or power, compared to process or composite endpoints. Accounting for clustering will lead to increase in sample size. Allow adjustments of sample size for protocol violations, participants drop out, and design considerations. |
| 7. Outcome measures | Choose outcome measures that are clinically relevant, sensitive, and measurable. Consider the trial duration and outcome prevalence in your sample population. Estimate or calculate minimal clinical important difference for each primary outcome. Consider validation of newly developed outcomes or scores. Evaluation of potential harm or negative effects, as well as economic analysis of health informatics intervention should be among the measured endpoints. |
| 8. Statistical analysis | Primary statistical analysis should always follow the intention-to-treat principle, preserving the power of randomization. Sensitivity analysis (“per protocol” or “on treatment”) can be used as a secondary approach, especially in explanatory HI studies. Statistical analysis should reflect the study design (e.g., factorial, cluster-randomized). |
| 9. Follow-up and missing data | Ensure complete patient follow-up, including situations when they participate through virtual mechanisms. Unless this violates consent or privacy agreement, participants who drop out of study should be followed up as well (this could be negotiated in advance). There are many ways to prevent or minimize missing data in study, and investigators have to make a pragmatic decision about which strategy to choose. Missing values should be appropriately reported and handled, preferably using multiple imputation techniques. |
| 10. Reporting | Use the CONSORT guidelines to report the results of HI intervention and overall trial logistics to allow readers to interpret and generalize the trial findings to different settings, systems, or populations and to help them better understand the study's strengths and limitations. |
