Summary of 30 Critiques on 15 Evaluation Plans Submitted by AHRQ Implementation Grantees
| Domain | Domain Score (1 = Best, 5 = Worst) N = 15 | Standard Deviation |
| Implementation goals | 1.2 | 0.5 |
| Evaluation goals | 1.5 | 1.0 |
| Measure Selection | 2.3 | 1.2 |
| Study Design | 2.4 | 0.8 |
| Feasibility of Plan | 2.9 | 0.9 |
| Quantitative measures | 3.3 | 1.1 |
| Qualitative measures | 3.3 | 1.0 |
| Overall impression | 2.7 | 0.8 |
-
Key for Overall Impression:
1 = Excellent, 2 = Good, 3 = Fair, 4 = Poor, 5 = No Plan.
1. Excellent: The plan has well-defined goals that were explicitly stated. The plan has measures to evaluate if those goals have been met. These measures appear to be feasible and important to evaluate for the health IT implementation. Behind each measure is a well-designed study, data collection and analysis plans.
2. Good: The plan has well-defined goals that either were explicitly stated or could be easily inferred. The plan has measures to evaluate if those goals have been met. These measures appear to be feasible and important to evaluate for the health IT implementation. Behind most measures are well-designed studies, data collection and analysis plans, although there may be minor deficiencies that need to be addressed.
3. Fair: The plan has goals that could be inferred. The plan has some measures to evaluate if those goals have been met. For those well defined metrics, most have reasonable and feasible study designs, data collection and analysis plans, although there may be deficiencies in these areas that need to be addressed.
4. Poor: The plan has unclear goals. Measures chosen were either inappropriate or impossible to measure given investigators' resources. There were significant deficiencies in the study design, data collection and analysis plans.
5. No plan: No goals could be inferred from the plan, nor were there measures to evaluate.
