Full impact of laboratory information system requires direct use by clinical staff: cluster randomized controlled trial
- Joaquín A Blaya1,2,
- Sonya Shin2,
- Carmen Contreras3,
- Gloria Yale4,
- Carmen Suarez5,
- Luis Asencios6,
- Jihoon Kim7,
- Pablo Rodriguez3,
- Peter Cegielski8,
- Hamish S F Fraser1,2
- 1Partners In Health, Division of Global Health Equity, Brigham and Women's Hospital, Boston, Massachusetts, USA
- 2Division of Global Health Equity, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, USA
- 3Socios en Salud Sucursal Peru, Lima, Peru
- 4DISA V Lima Ciudad, Lima, Peru
- 5DISA IV Lima Este, Lima, Peru
- 6Instituto Nacional de Salud, Lima, Peru
- 7Division of Biomedical Informatics, University of California, San Diego, California, USA
- 8Centers for Disease Control and Prevention, Atlanta, Georgia, USA
- Correspondence to Dr Joaquin Blaya, Partners In Health, Division of Global Health Equity, Brigham and Women's Hospital, 641 Huntington Avenue, Boston, MA 02115, USA;
- Received 9 July 2010
- Accepted 3 November 2010
- Published Online First 27 November 2010
Objective To evaluate the time to communicate laboratory results to health centers (HCs) between the e-Chasqui web-based information system and the pre-existing paper-based system.
Methods Cluster randomized controlled trial in 78 HCs in Peru. In the intervention group, 12 HCs had web access to results via e-Chasqui (point-of-care HCs) and forwarded results to 17 peripheral HCs. In the control group, 22 point-of-care HCs received paper results directly and forwarded them to 27 peripheral HCs. Baseline data were collected for 15 months. Post-randomization data were collected for at least 2 years. Comparisons were made between intervention and control groups, stratified by point-of-care versus peripheral HCs.
Results For point-of-care HCs, the intervention group took less time to receive drug susceptibility tests (DSTs) (median 9 vs 16 days, p<0.001) and culture results (4 vs 8 days, p<0.001) and had a lower proportion of ‘late’ DSTs taking >60 days to arrive (p<0.001) than the control. For peripheral HCs, the intervention group had similar communication times for DST (median 22 vs 19 days, p=0.30) and culture (10 vs 9 days, p=0.10) results, as well as proportion of ‘late’ DSTs (p=0.57) compared with the control.
Conclusions Only point-of-care HCs with direct access to the e-Chasqui information system had reduced communication times and fewer results with delays of >2 months. Peripheral HCs had no benefits from the system. This suggests that health establishments should have point-of-care access to reap the benefits of electronic laboratory reporting.
Evaluations of laboratory reporting systems to date have measured their impact on those health establishments that have point-of-care access to the system via internet, local area networks, or other forms. In these settings, they have been shown to decrease result delivery times (RDTs) of laboratory results,1–3 reduce redundancy in resource utilization,2 4 5 and provide faster and more complete notification for public health purposes.6–8 Shorter RDTs have been associated with decreased treatment time, mortality, morbidity, and length of hospital stay.9 10
In many resource-poor settings, public health systems are complex networks in which some institutions have direct access to electronic systems while others receive information indirectly, often relying on central health establishments for transmission of patient data. As laboratory networks become more decentralized in resource-poor settings,11 effective communication strategies must address the needs of a heterogeneous group of users, and overcome challenges in relaying timely information to even the most peripheral sites. To our knowledge, there are no reports of evaluations of the impact of information systems on complex public health systems in resource-poor settings.12
To evaluate the impact of a laboratory information system on both point-of-care and peripheral health centers (HCs), we implemented the e-Chasqui tuberculosis (TB) laboratory system13 as part of a larger project to decentralize laboratory capacity within the Peruvian National Tuberculosis Program.14 Here, as in many other countries, communication of results between central and local laboratories and clinical facilities is still problematic. A baseline assessment found that 10% of results take >2 months to arrive, and patients could still experience risky delays unless programmatic aspects are also addressed.15 Therefore, e-Chasqui was developed and implemented in Lima, Peru to communicate data between two district laboratories, 14 point-of-care HCs with internet that had direct access to e-Chasqui, and 17 peripheral HCs without internet whose results were relayed through a point-of-care HC (figure 1). This system has been shown to decrease the number of reporting errors to HCs by up to 87%, most importantly eliminating missing results.16
We conducted a cluster randomized controlled trial (RCT) to evaluate the effectiveness of the e-Chasqui laboratory information system in reducing communication delays to both point-of-care and peripheral HCs in the National Tuberculosis Program in Peru.
A cluster RCT tested the effect of the laboratory information system, e-Chasqui, in reducing the time to communicate patients' test results. The trial is reported according to the CONSORT statement,17 and was performed within a larger observational study evaluating the impact of expanded laboratory capacity in the district laboratories.15 This study was approved by the Partners Healthcare Human Research Committee and the Peruvian National Institute of Health.
This study was carried out in two health districts of Lima, Peru: Lima Ciudad and Lima Este. Lima Ciudad includes 45 health establishments (24 HCs, nine health posts, and 12 hospitals) serving a population of 1 577 090 in an area of ∼100 km2. Lima Este includes 134 health establishments (42 HCs, 87 health posts, and five hospitals) serving a population of 1 088 515 in an area of ∼6340 km2. Smear microscopy is used to diagnose active TB, while culture and drug susceptibility testing (DST) are reserved for patients with confirmed TB and at least one risk factor for multidrug-resistant (MDR)-TB according to National Tuberculosis Program (NTP) Norms.18 Smear microscopy is performed in level I laboratories in HCs and hospitals. Health posts send sputum samples to their closest HC for smear microscopy. For patients with MDR-TB risk factors, smear-positive samples are sent to district laboratories for culture and/or DST to first-line drugs (table 1). Strains resistant to isoniazid or rifampicin or both are sent to the National Reference Laboratory (NRL) for DST to second-line drugs. Paper results are emitted from the NRL and district laboratories and transmitted, directly or indirectly, to health establishments. The patient is then routinely seen by a pulmonologist at the local hospital to review the DST results and, if necessary, modify the TB regimen. In patients with drug-resistant isolates, a district expert committee must review the case and approve initiation of MDR-TB therapy (figure 1).
The two health districts organize transmission of paper results to HCs differently. In Lima Ciudad, all 24 HCs are point-of-care HCs that receive results directly from the district laboratory. In Lima Este, health establishments are organized into ‘micro-networks.’ Seventeen point-of-care HCs serve as the micro-network heads and use an identical process to HCs of Lima Ciudad. The other 25 HCs and 87 health posts are peripheral HCs, which receive test results via the head of its micro-network (figure 1).
In March 2006, e-Chasqui was first implemented at the two district laboratories and the NRL. These laboratories served all of the health establishments. After full implementation in the laboratories, 12 of 34 point-of-care HCs were randomized to utilize e-Chasqui. We randomly assigned e-Chasqui to six of the 20 highest TB incidence HCs in Lima Ciudad and to six of the 12 Lima Este micro-networks within Lima city limits (figure 1). In Lima Este, the six micro-networks assigned to e-Chasqui consisted of six point-of-care HCs with 17 peripheral HCs. The six control micro-networks comprised six point-of-care HCs with 27 peripheral HCs (table 2).
We performed intent-to-treat analysis based on the original HC randomization. During the study (October 2006), the Lima Este health district reorganized their micro-networks, which had the following effect on study assignments: one point-of-care control HC became a peripheral intervention HC; three peripheral intervention HCs became peripheral control HCs; and three peripheral control HCs became peripheral intervention HCs. Data from these ‘cross-over’ HCs only affected 17 data points for the primary outcome.
All individuals who lived within the catchment area of participating health centers—the 20 HCs with highest TB incidence in Lima Ciudad and establishments within the 12 micro-networks within Lima Este city limits—and had at least one MDR-TB risk factor as defined by the Peruvian NTP Norms were included in this study.18 There were no exclusion criteria for enrollment into the study. Since sputum samples of patients in the public health sector with at least one MDR-TB risk factor should all be submitted to the district laboratory for DST, subjects eligible for enrollment into the study were identified using only this referral.
The primary outcome of the study was the laboratory RDT, defined as the number of days between a test result date and the date that result was received by the HC. For the electronic system, the date received at the HC was the earliest date of reception of the paper result as shown in the reception stamp by the HC or when the result was viewed online by a TB staff member. For the paper system, the date received at the HC was the reception of the paper result as shown in the reception stamp by the HC. This primary outcome was calculated for both cultures and DSTs. The secondary outcome was the proportion of DST results with a laboratory RDT >60 days.
We designed and implemented a web-based laboratory information system, ‘e-Chasqui’, in Lima, Peru to improve the timeliness and quality of laboratory data.13 It was deployed in the NRL, two district laboratories, and 12 intervention HCs. The core of the e-Chasqui interface is a single patient page containing the history of all tests performed for the patient on a left sidebar and the details for any single sample on the main part of the page (figure 2). Tools built for the laboratory include quality control, reports on tests performed, warnings of delayed reporting of results, and a user directory to control any individual's access. Tools for clinicians included email notification of new results, a consolidated list of results for their jurisdiction, and a list to track the status of all pending samples. All intervention HC staff were trained at their HC in an initial visit for ∼1 h. The data administrator would then visit or call the HC at least twice a month and could be contacted via cell phone or email during business hours.
e-Chasqui was built as a stand-alone module on the open source Partners In Health Electronic Medical Record (PIH-EMR), a web-based system designed for TB and MDR-TB treatment in resource-poor settings.19 The PIH-EMR provides the ability to register patients, order medications,20 display chest x-rays, generate monthly reports for funders, and predict future drug requirements.21 This system is built using Java, HTML and an Oracle database. The PIH-EMR has been adopted as the official system of the Peruvian NTP and is being used in the Philippines. A second version of the system to support HIV/AIDS treatment has been deployed in rural Haiti and Rwanda.22 23 The functionality of e-Chasqui is being ported to the open source EMR, OpenMRS.24–26
Baseline data were collected 15 months prior to the implementation of e-Chasqui (Jan 1, 2005 to Mar 30, 2006 for Lima Ciudad, May 1, 2005 to Aug 18, 2006 for Lima Este). However, the Lima Este district laboratory did not perform DSTs before the implementation of e-Chasqui, hence there are no pre-implementation data on DSTs for that district.
Data were prospectively abstracted by a team of trained collectors who used standardized forms. For the RCT, the study started on the date of implementation of e-Chasqui and ended on August 31, 2008. We also used data from the e-Chasqui database, including the date of electronic receipt of test result. If the end date for any RDT was missing, we censored that time using the date the patient finished the study.
We examined the effect of the intervention, adjusting for the impact on variance of the clustering in the study design. We used multivariate regression models (marginal model with generalized estimating equations) to investigate the effect of the intervention on the RDT outcomes as a function of covariates and to account for clustering at the HC level.27 To investigate whether the intervention was associated with a reduction in the number of DST results with laboratory RDT >60 days, we used a generalized linear mixed model,28 29 with HC as a random effect, patient as a nested random effect within HC due to repeated samples, and health district and period (pre- and post-implementation) as fixed effects. This model did not converge for point-of-care HCs, therefore for this analysis we used the model without patient as a nested random effect.
To adjust for possible HC differences that may have been unequally distributed despite randomization, we included the median pre-intervention RDT per HC for both cultures and DSTs (as a proxy for HC variance) and number of HC staff changes. We used SAS V.9.1 for all analyses and checked all models built using R.30
During the trial, 89% (1671/1888) of all eligible patients were enrolled (figure 3). The intervention HCs had a significantly greater number of study participants per HC. If separated by district, these differences existed only in Lima Ciudad. The intervention HCs also had younger patients, a higher proportion of female patients, and a larger number of personnel changes in the TB clinician per HC value. There were no significant differences in the number of patients per peripheral HC, number co-infected with HIV, number of TB nurse changes during the study, number of patients who had a smear or culture positive or had a drug-resistant DST by study arm (table 2). Most (98%) of all culture results and all DST results available in e-Chasqui were viewed by the intervention HCs.
For point-of-care HCs, the intervention group took significantly less time to receive both DST (median 9 vs 16 days, p<0.001) and culture (4 vs 8 days, p<0.001) results (table 3) than the control group. However, when peripheral HCs were compared, the intervention group did not have significantly different RDTs for DSTs (22 vs 19 days, p=0.294) or cultures (10 vs 9 days, p=0.10) from the control HCs (table 3).
Laboratory RDT >60 days
For point-of-care HCs, the intervention group had fewer DSTs with a laboratory RDT of >60 days than control HCs (6.0% vs 17.7%, p<0.001) as seen in table 3. However, for peripheral HCs, the intervention group had a non-significantly higher proportion (30.2% vs 26.1%, p=0.568) than the control group.
The e-Chasqui laboratory information system considerably reduced the time to communicate results of cultures and DST to point-of-care HCs and the proportion of results that had an excessive delay of >60 days or never arrived. This decrease in delays could have a profound impact on the patient, since DSTs are needed to ensure that the regimen being provided has sufficient drugs to which the strain is susceptible to make it effective. Monthly culture results are required to monitor the patient to ensure the treatment is working. Large delays or missing results, especially in DSTs, could not only have a clinical impact on the patient, but also keep them in an infectious state and thus promote transmission of the strain.
These positive effects, however, were not seen in peripheral HCs where there was no difference in outcomes between the intervention and control groups. This suggests that the effect of this laboratory information system is only seen with direct clinician access to the data and cannot be expected to ‘trickle down’ to other health establishments where there is no direct access.
Apart from another study published by our group, no one has published information about the impact of laboratory reporting systems in decreasing communication and treatment times in developing countries31 to see if they are a worthwhile investment of scarce resources. We have previously published the resources required to develop and implement e-Chasqui to allow decision makers to analyze whether such systems were appropriate in their regions.13 The data provided by this paper should provide further quantitative data to help decide if funds should be assigned to health informatics projects such as this one.
This study shows that placing the electronic systems closer to clinical users, but not providing direct access does not change anything. This ‘last mile’ problem is a common theme in bridging the digital divide or providing connectivity to rural areas and also seems to apply in the delivery of laboratory results to clinicians. Complex networks including central and district laboratories, and point-of-care and peripheral HCs, are common in many public health systems. Decision makers must make a decision about how far to extend technologies such as the internet. Our findings suggest that the greatest benefit can be achieved by providing all establishments with the internet so that they can reap the advantages of the information system, even if the organizational structure remains the same. Although internet access is becoming more accessible worldwide, many health establishments (often the most peripheral ones) do not have access. One possibility would be to deploy well-designed mobile systems using personal digital assistants or smart phones in such sites, in order to ensure point-of-care access to data. There are already several groups working and implementing systems in developing countries that could be used for this purpose.32–34
There are limitations to this study. There were fundamental baseline differences between the intervention and control HCs despite the randomized nature of this trial. These differences could introduce bias into the analysis, but in all cases we used pre-implementation values in our models to account for this. The study was conducted in the two most populous health districts in Peru. Therefore the generalizability of these results should be treated with caution. Being in an urban area provided the project with mostly consistent power and internet, as well as geographic proximity to technical support, which is not the case in many resource-poor settings. Therefore groups implementing these systems should ensure that the appropriate infrastructure is in place. Further, patients in the study were assigned to the HC where they were first captured. If a patient transferred or left a sample at another HC, their data were assigned to the initial HC. If a patient transferred from a control to an intervention HC, the intervention HC staff would have access to all of that patient's bacteriological history in e-Chasqui. Both of these situations may have the effect of weakening the impact of the intervention. Finally, this was a formative, rather than summative, evaluation since the developers were involved.
This is the first RCT to show that a laboratory information system can reduce communication delays in a resource-poor setting, and yet this impact is attenuated among HCs that do not have direct access to the system.
A carefully designed and implemented web-based TB laboratory information system reduced the time to communicate results between laboratories and health establishments with point-of-care access to the system; however, it had no impact on peripheral health establishments that relied on these point-of-care centers for communication of the results. These results suggest that, when electronic systems to communicate results to clinical staff are implemented, the last mile problem of providing personnel with direct access to the electronic results must be addressed.
We acknowledge the dedication of the laboratory and health center users. We thank Betty Palma, Michael Seaton, Darius Jazayeri, Ruben Alvarado, and Ellen Ball for designing and maintaining these systems, and Leonid Lecca, Jaime Bayona, and Carole Mitnick for assistance in the study. We also thank the Peruvian Ministry of Health and Socios en Salud Sucursal Peru for their help.
Funding Harvard Global Infectious Diseases Program, David Rockefeller Center for Latin American Studies. JAB received a MIT Public Services Center grant, the MIT Hugh Y Hampton Fellowship, and a National Research Service Award from the NIH.
Competing interests JAB is cofounder of eHealth Systems, a Chile-based company providing health informatics consulting and implementation work.
Ethics approval This study was conducted with the approval of the Brigham and Women's Hospital and the Peruvian National Institute of Health.
Provenance and peer review Not commissioned; externally peer reviewed.