Does Feedback Improve the Quality of Computerized Medical Records in Primary Care?
- Affiliations of the authors: St. George's Hospital Medical School, London, (SdL); IMS Health, Pinner, Middlesex (PNS); PMSI UK Ltd., High Wycombe, Buckinghamshire (NA); University College, London (AM), United Kingdom
- Correspondence and reprints: Simon de Lusignan, BSc, MBBS, Senior Lecturer, Primary Care Informatics, General Practice and Primary Care, St. George's Hospital Medical School, London SW17 0RE, United Kingdom; e-mail: <slusignan{at}sghms.ac.uk>
- Received 27 September 2001
- Accepted 13 February 2002
Abstract
Objective The MediPlus database collects anonymized information from generalpractice computer systems in the United Kingdom, for research purposes. Data quality markers are collated and fed back to the participating general practitioners. The authors examined whether this feedback had a significant effect on data quality.
Methods The data quality markers used since 1992 were examined. The authors determined whether the feedback of “useful” data quality markers led to a statistically significant improvement in these markers. Environmental influences on data quality from outside the scheme were controlled for by examination of the data quality scores of new entrants.
Results Three quality markers improved significantly over the period of the study. These were the use of highly specific “lower-level” Read Codes (p=0.004) and the linkage of repeat prescriptions (p=0.03) and acute prescriptions (p=0.04) to diagnosis. Clinicians who fall below the target level for linkage of repeat prescriptions to diagnosis receive more detailed feedback; the effect of this was also statistically significant (p<0.01.)
Conclusions The feedback of four of the ten markers had a significant effect on data quality. The effect of more detailed feedback appears to have had a greater effect. The lessons learned from this approach may help improve the quality of electronic medical records in the United Kingdom and elsewhere.
Although the majority of NHS (National Health Service) general practitioners in the United Kingdom are now computerized1 and the computer systems they use have can record structured data (Read Coded2; see box on next page), high-quality coding of clinical data is not yet universal.3 4 5 There are a number of reasons for this. Until recently, general practitioners were required to keep written as well as computerized medical records.6 Using computers in primary care also results in longer consultations.7 8 Despite these obstacles, an increasing amount of clinical data is now being recorded electronically.9
Many recent NHS policy documents have promoted the use of computerized records. These include the NHS information strategy,10 the National Service Frameworks,11 and the NHS Plan.12 The more recent “Building the Information Core” document,13 from the NHS Information Policy Unit, provides the most up-to-date milestones. A key target is that half the primary care trusts will have implemented electronic patient records by 2004. To improve the usefulness and accuracy of these electronic records, primary care trusts will need to implement programs that improve data quality. Evaluation of such interventions is lacking, however.14
Primary Care Computerization in the UK National Health Service
In the United Kingdom, the delivery of health care is based on the patient's general practitioner, who is responsible for organizing both primary care services and referrals for specialist care. Almost the entire population of the United Kingdom is registered with general practitioners. As there are no charges for consulting a general practitioner, there are no consulting a general practitioner, there are no financial barriers to access to primary health care. General practitioners in the United Kingdom have much in common with family practitioners in the United States; both groups have undergone higher professional training and see a broad mix of patients. Specialty practice, as in pediatrics, gynecology, and internal medicine, however, is almost entirely based in secondary care in the United Kingdom, unlike in the United States.
Because the United Kingdom has a very strong gatekeeping system, general practitioners have access to complete medical histories of their patients. Almost all general practices in the United Kingdom are now computerized. This has largely occurred because the UK government has underwritten the cost of computerization, so general practitioners have been reimbursed for most of the costs.
Where general practitioners work in practices that are fully computerized, these data provide an important resource for clinical management, auditing, quality assurance, and research. Clinical systems in the United Kingdom use the Read Codes, a coding system developed in the United Kingdom to code clinical data. The UK government has ambitious plans for further developing the use of computerized medical records in primary care: it is hoped that, by 2005, most general practices will be fully computerized and will have stopped using paper medical records.
Until now, only clinicians who volunteered have been part of data quality schemes and received feedback on the quality of their coding. There is no clear evidence about whether such feedback can improve data quality. For example, the effectiveness of PCO (primary care organization)-wide feedback on data quality has yet to be shown by the Primary Care Information Services (Primis) project,15 and only limited published data from three PCOs has come out from the program Primary Care Data Quality (PCDQ) program.5
We examined the feedback of data quality markers within the MediPlus database to see whether this led to a more rapid improvement in data quality than that generally occurring in primary care. We hoped that the experience gained from data quality feedback over an 8-yr period could be applied more generally to raising the standards of computerized medical records in primary care.
Methods
Data Source: The MediPlus Database
The MediPlus database was established in 1992; it contains information on almost two million patients and more than 53 million prescriptions.16 The database is based on information drawn from more than 500 representative general practitioners across the United Kingdom using the Torex-Meditel System 5 computer package.17 This computer system allows the linkage of diagnosis or problem title to the acute or repeat (long-term) prescriptions issued to patients. This makes clearer the diagnoses for which prescriptions are being issued. This is particularly useful among groups such as the elderly, who often suffer from several chronic diseases.
Data quality markers are used to ensure that only doctors supplying data that reaches specified quality standards are included in the database used by researchers. In total, ten data quality scores are used. These are calculated at individual doctor level and fed back to the participating practices quarterly. Newsletters are also sent every six months, addressing issues around coding highlighted by a panel of expert general practitioners. Doctors are given a small incentive (about £400 per doctor per year) to reach the target levels across the ten quality scores used (Table 1).
Data Quality Markers Used by the MediPlus Database
Literature Review
We carried out a literature review to establish whether there was a consensus on what data quality parameters should be fed back. PubMed (National Library of Medicine) was searched using “Data Quality” and “General Practice” as search terms. This identified 848 abstracts, each of which was examined to identify articles relating to either the membership of a data quality scheme or the effectiveness of feedback of clinical markers. There were many descriptions of the potential,18 19 the need for,20 21 22 23 or the actual use of data quality feedback,24 25 26 but no evidence about what elements should be fed back and in what way. Where feedback techniques and audit have been used, they have focused on clinical outcomes rather than on changes in data quality.27 28 29 Individual feedback appears to be better than group feedback,30 and feedback from a peer-group in general practice appears to be more effective than feedback from non-clinicians.31 Feedback focused on a particular clinical area also seems to be more effective than generalized feedback.32
Data Analysis
If feedback had a positive effect on quality, then the longer a general practitioner had been in the scheme, the higher would be their score.
Calculating Whether Data Quality Is Related to Time
General practitioners contributing data in the first quarter of year 2000 were placed into groups representing the year in which they joined. The mean scores on each quality marker for each group in the first quarter of 2000 were then calculated and regression analysis was used to determine whether length of time in the scheme affected quality scores and, if so, in which areas.
Excluding the Effect of External Environmental Factors
Improvements in the quality marker scores may have been due to various NHS initiatives, such as Collection of Health Data from General Practice.33 If general practitioners were improving “naturally,” this would be reflected in an increase in the starting scores of general practitioners who joined the scheme over time. General practitioners were therefore grouped according to year of joining, their starting scores on each marker extracted, and regression analysis on the means for each group used to see whether their starting scores improved over time on any marker.
Excluding the Effect of Differential General Practitioner Drop Out
Results may be biased by a greater proportion of the poorly performing doctors dropping out during the early years of the scheme. To check this, doctors were first grouped according to the length of time they had spent in the scheme, regardless of start date. For example, two general practitioners who started in 1992 and in 1994 but who remained in the scheme for three years would be placed in the same group. The difference in each general practitioner's first and last scores was calculated and from these the mean scores were found for each group and marker. Regression analysis was again used to determine whether time in scheme affected data quality.
Effect of Specific Feedback to Those with a Below-average Score for Linkage of Diagnosis and Prescription
One particular form of feedback was also investigated, an initiative to improve the linkage of diagnosis to prescription. Its effectiveness was assessed by comparing the mean score for all the general practitioners in the second quarter of 1999 who received the report, with their mean score in the first quarter of 2000.
Results
Three Markers Show Improvement with Time
The quality markers showing a significant improvement with time at the 5 percent level were:
-
Percentage of acute prescriptions linked to a diagnosis
-
Percentage of repeat prescriptions linked to a diagnosis
-
Percentage of problems defined by a Read Code of level 3 or lower.
The mean starting scores and the results of the regression analysis are shown in Tables 2 and Tables 3.
Mean Score for Each Quality Marker, by Year in Which General Practitioners Joined MediPlus Database
Change in Data Quality Markers, and Their Regression Scores, Ranked in Order of Significance
External Environmental Factors Do Not Explain Improvement
None of the markers that improved over time showed any evidence that external factors had played a part. Tables 4 shows the results of the regression analysis on the general practitioners' starting scores. The only marker showing any significant improvement in starting score with time was the number of prescriptions issued per 1,000 registered patients (p<0.05).
The Influence of External Factors on Data Quality, Ranked in Order of Significance
Differential General Practitioner Drop Out Cannot Explain Improvement
It was possible to do this analysis for two data quality markers—the percentage of repeat prescriptions and the percentage of acute prescriptions linked to a diagnosis. The repeat prescription linkage showed significant improvement with time in the scheme, regardless of the general practitioner start date (p<0.05).
Acute prescription linkage did not show any such trend. However, whether time spent in the scheme was short or long, acute prescription linkage improved (range, 2.5–15.1 percent). These two analyses suggest that the improvements seen in coding quality were not due to differences in the rate at which poorly performing general practitioners dropped out of the scheme.
Significant Effect of Specific Feedback for Linkage of Diagnosis and Prescription:
The feedback of the detailed repeat prescribing reports had a significant effect on the percentage of repeat prescriptions linked to diagnosis. This increased from 64.6 percent in the second quarter of 1999 to 79.5 percent in the first quarter of 2000 (difference, 14.9 percent; 95% confidence interval, 3.7–26.1 percent; p=0.005).
Discussion
The main finding from this study is that the feedback of four of the quality markers improved data quality. All these markers were fed back over a long period; one marker was also fed back over a shorter period, with the specific aim of increasing the linkage of diagnosis or problem to repeat prescriptions. The scheme members were also offered a small financial incentive, but this was dependent on meeting target scores for all markers, not specific ones. To receive this payment, general practitioners would have needed to focus on those markers for which they performed least well.
The results of this study are mixed. Feedback of half the markers achieved significant improvement, while feedback of others did not. Feedback of this nature is not, therefore, in itself an effective mechanism, but it may represent a low-cost tool that can be used alongside other tools. The explanations for why the short-term feedback was so successful also needs to be explored further.
The findings from this study have potentially important implications for electronic patient records. First, they can inform those seeking mechanisms to improve data quality about the effects of a long period of feedback to a large group of practices. Second, those feeding back data quality indexes may wish to critically examine whether the feedback had any effect on data quality. Third, they indicate the importance of further research to describe the context in which feedback may contribute toward improvement in data quality.
Some potential confounding factors need to be considered. The members of the MediPlus database all use the Meditel computer system and are volunteers. They may have made considerable efforts to raise their data quality standards before joining the scheme. Some general practitioners had data quality markers that were already at levels over 90 percent before they joined the scheme. For markers with such high scores, it would be difficult to show a statistically significant improvement in score over time. Finally, some markers have been overtaken in their usefulness by advances in technology. Automated registration links with the Health Authority (GP-Links Project) has almost eliminated the number of patients without full demographic details. Similarly, patients who die or move away are now more likely to be automatically removed from a practitioner's list. In the past, unremoved patients may have artificially increased the list size, thereby increasing the denominator population used to calculate the data quality scores. Technical solutions like GP-Links have clearly had a major influence on some aspects of general practitioner data.
Further research is needed to ascertain what data quality markers should be fed back and by whom. From the literature review, individual feedback on a narrow clinical focus seems to offer the best approach to feedback of data quality markers. However, this was not the mechanism used for the most successful data quality marker fed back within the MediPlus database. Research is also needed to ascertain whether the personal feedback, the token financial rewards, or some other factor was responsible for this change. The role of practice staff may also need to be carefully examined, as primary care support staff are responsible to varying degrees for issuing repeat prescriptions.
Conclusions
We found that four data quality markers, all relating to the linkage of diagnosis to prescription and the use of more specific Read Codes, improved at a significantly higher rate in MediPlus practices. The personalized feedback to those general practitioners with below-average scores and the token financial incentives may have been important motivating factors and should be tested elsewhere. However, the role of practice support staff and the improvements made to the accuracy of the denominator through the GP-links project show that factors other than coding by clinical staff may have profound effects on data quality. If general practice computer records are to become the cornerstone of the electronic patient and health records promised by the NHS information strategy, research is urgently needed to define how feedback on data quality should be given.
Acknowledgments
Simon de Lusignan heads a Primary Care Informatics Group within the Department of General Practice and Primary Care at St. George's Hospital Medical School. Peter Stevens and Naeema Adal are employees of IMS, which runs the MediPlus database. Azeem Majeed holds a Primary Care Scientist Award and is funded by the NHS Research and Development Directorate. Neither Simon de Lusignan nor Azeem Majeed received any payment for this study, and they have no financial interest in IMS.
