Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa
W Mphatswe a, KS Mate b, B Bennett b, H Ngidi a, J Reddy a, PM Barker b & N Rollins c
a. Department of Paediatrics and Child Health, University of KwaZulu-Natal, Durban, South Africa.
b. Institute for Healthcare Improvement, 7th Floor, 20 University Road, Cambridge, Massachusetts, MA 02138, United States of America.
c. Department of Maternal, Newborn, Child and Adolescent Health, World Health Organization, Geneva, Switzerland.
Correspondence to KS Mate (e-mail: firstname.lastname@example.org).
(Submitted: 29 June 2011 – Revised version received: 28 October 2011 – Accepted: 02 November 2011 – Published online: 05 December 2011.)
Bulletin of the World Health Organization 2012;90:176-182. doi: 10.2471/BLT.11.092759
Reliable and accurate public health information is essential for monitoring health and for evaluating and improving the delivery of health-care services and programmes.1–4 As countries report their progress towards achieving the United Nations Millennium Development Goals, the need for high-quality data has never been greater.5,6 Furthermore, funding and support for public health activities, such as immunization programmes, remain contingent on demonstrating coverage using routine statistics.7 However, assuring the quality of health information systems remains a challenge.
Studies of public health information systems in resource-poor countries frequently document problems with data quality, such as incomplete records and untimely reporting.8,9 Yet these systems are often the only data sources available for the continuous, routine monitoring of health programmes.10,11 Efforts have been made to improve the quality and management of public health information systems in developing countries. Two examples are the Health Metrics Network, an international network that seeks to improve the quality of health information from various sources,12 and the Performance of Routine Information System Management (PRISM) framework, which was developed as a method for assessing the strengths and weaknesses of routine health information systems.13,14 Other initiatives, such as the Data Quality Audit, have been used by the GAVI Alliance to improve the monitoring of immunization coverage.7 However, the complex nature of health information systems and the demands placed upon them have complicated efforts to improve the quality of routine data.15
In South Africa, the effect of human immunodeficiency virus (HIV) infection on maternal and child health has raised considerable concern. The reported prevalence of HIV infection among women attending antenatal clinics in the province of KwaZulu-Natal, for example, is 38.7%, the highest in the country.16 This led South Africa to set a target of reducing the rate of mother-to-child HIV transmission to below 5% by 2011.17 Although HIV testing is essential for delivering interventions for the prevention of mother-to-child transmission (PMTCT) and thereby for reducing maternal and infant mortality,18 routinely available health information indicates that the HIV testing rate is highly variable.19,20 The questionable accuracy and reliability of these data compromise efforts to improve the health-care systems that provide PMTCT interventions.20
Routine health data from the primary health-care system in South Africa, which is organized in districts, are collected and stored in the District Health Information System (DHIS).21 Recent studies of this system, however, have reported that the quality of the data, including those used to track PMTCT care, is suboptimal and is hindering efforts to strengthen service delivery.20,22 As a result, in 2008 the KwaZulu-Natal Department of Health, the University of KwaZulu-Natal and the Institute for Health care Improvement launched a large-scale effort, entitled the 20000+ Partnership, to improve the quality of PMTCT services in three health districts in the province. This programme included an intervention to increase the completeness and accuracy of the public health data routinely recorded in the DHIS. The aim of this paper is to report on the effect of that intervention.
Routine data collection for the DHIS in South Africa starts with information being collected in registers at each point where clinical care is provided. Every month the staff at the different primary health-care facilities collate the data and send monthly summaries on paper to a clinic supervisor who, on average, oversees six facilities. The monthly summaries are then converted into electronic format by an information officer based either at the facility or centrally within the district health office. Larger facilities such as community health centres and hospitals have dedicated facility information officers who themselves create electronic files containing the facility data. District information officers oversee the input of all data into the DHIS. In addition, district PMTCT coordinators work regularly with district information officers to ensure that all health-care workers receive training on data reporting requirements, data collection tools and any changes or updates to the definitions of the data elements used as indicators of the performance of PMTCT services.
The study involved data collected before and after the data quality improvement intervention at 58 antenatal clinics and 20 delivery wards in both urban and rural communities in KwaZulu-Natal. These 78 locations were randomly selected from 222 fixed facilities that provide PMTCT services in three of the 11 districts in the province: eThekwini, Umgungundlovu and Ugu districts. Across KwaZulu-Natal province, over 90% of pregnant women attend antenatal care facilities at least once during pregnancy and over 90% of deliveries take place in delivery wards at health-care facilities.23,24 The prevalence of HIV infection among expectant mothers in the three districts ranged from 40.3% to 45.7% in 2008.17
The data quality improvement intervention took place between May and November 2008 and included three main components: (i) training days on data collection, (ii) monthly reviews of data and (iii) data audits at individual facilities.
Three data training days for health-care facility and district information officers and PMTCT programme managers took place in May, August and November 2008, respectively. Each one-day training session dealt with: standardizing data collection procedures; improving data quality; selecting data elements that were useful for monitoring the quality of data on PMTCT services; and providing clear definitions of those data elements. After training, each district information officer liaised on data quality with facilities in his or her district.
Monthly reviews of routine data on PMTCT services were carried out in each district by a PMTCT improvement task team, which comprised the district information officer, the PMTCT programme coordinator and the primary health-care coordinator. During these monthly meetings, the study team provided the task teams with information about the completeness and accuracy of the PMTCT data elements they were using to assess the PMTCT programme. Thereafter, each district task team provided feedback to the facilities on the quality of the data.
Data audits were carried out by study staff at the 78 randomly selected facilities three times: before, during and after the intervention. The original source data in facility registers covering the 3 months preceding the audits were collected and summarized in the same manner in which they were routinely dealt with at the end of each month. The accuracy of the process by which these data were collated at health-care facilities, transmitted to the district offices and entered into the DHIS was assessed by examining the degree of agreement between the data elements for a particular clinic that appeared in the DHIS and the data elements recorded by the study team for the same time period. It is important to note that whether or not the data contained in the registers accurately represented the clinical activities taking place at each facility was not assessed, as resources did not allow for direct clinical observation. After each audit, the study team provided individual facilities with feedback on the quality of its data on PMTCT services.
Ethical approval for the study was obtained from the University of KwaZulu-Natal Biomedical Research Ethics Committee. In addition, consent was obtained from the KwaZulu-Natal Department of Health and the district managers of all three participating districts. The funding agency, the Centers for Disease Control and Prevention in Atlanta, United States of America (USA), also gave approval but did not influence the study design or data reporting.
Data collection and assessment
The study team visited the 78 randomly selected facilities before the intervention between September and November 2007, during the intervention between June and September 2008 and after the intervention in March 2009.
At each visit, the team collected information on the six data elements used for monitoring PMTCT services that were routinely recorded in registers on the number of: (i) pregnant women who were tested for HIV antenatally; (ii) pregnant women who tested positive for HIV; (iii) pregnant women in whom CD4+ lymphocyte quantitation was performed; (iv) pregnant women who received nevirapine in the delivery ward; (v) babies who were born to an HIV-positive woman and who underwent a polymerase chain reaction test for HIV; and (vi) babies who received nevirapine. These six data elements were selected from a total of 24 possible elements because of their clinical importance. The characteristics of each facility were recorded in a simple form that was completed by the facility manager during the site visit.
It was possible to obtain information on the completeness of the data reported between January 2007 and March 2009 to the DHIS for all 222 facilities providing PMTCT services in the study districts. Completeness was quantified by determining whether a value for a given data element from a specific facility was or was not present in the information system. Data completeness was analysed by data element and facility, as well as over time.
Data accuracy was determined by comparing the data recorded in the DHIS with the data obtained from individual facility registers during data audits. An individual data element value was defined as accurate if the value recorded in the information system lay within 10% of the corresponding value “reconstructed” by the study team from the original source data during the data audit. The 10% threshold was based on that used in previous studies.11,20 No assessment of the accuracy of a data element value was made if that value was missing from either the facility register or the DHIS or if the difference between the value reconstructed by the study team and that in the information system was greater than 1000%. Such large differences were regarded as “errors” rather than inaccuracies.
The data analysis was carried out using Excel 2007 (Microsoft, Redmond, USA). The change in data accuracy from before to after the intervention was evaluated using the χ2 test, with an α of 0.05. The degree of agreement between data reported in the DHIS and data obtained from facility registers during the data audits was evaluated using Pearson’s correlation coefficient (r), both before and after the intervention.
Of the 78 facilities randomly selected to participate in the study of the data quality intervention, 37 (47%) were urban sites, 21 (27%) were rural and 20 (26%) were semi-urban. The median number of clients seen daily at each site was 28 and the median number of health-care workers at each site was 6. All facilities had been offering PMTCT services for more than 2 years.
Overall data completeness improved from 26% before the intervention at all 222 facilities to 64% after: of the 1115 data element values that could have been completed in January 2007, 285 were completed, while of the 1010 that could have been completed in March 2009, 643 were completed. In the antenatal clinics, data completeness improved from 25% (i.e. 271 of 1065 relevant data element values were completed) to 63% (i.e. 600 of 960 relevant data element values were completed). In the delivery wards, data completeness improved from 28% (i.e. 14 of 50 relevant data element values were completed) to 86% (i.e. 43 of 50 relevant data element values were completed). Fig. 1 shows the change in completeness for the six data elements individually.
Fig. 1. Completeness of dataa used to monitor the prevention of mother-to-child HIV transmission, before and after a data quality intervention, by data element, South Africa, 2007–2008
Overall, 55% of data element values were excluded from the accuracy analysis before the intervention because of errors or missing data, compared with 37% after. The overall accuracy of the remaining data on all six data elements improved from 37% at the first data audit to 65% at the third data audit (P < 0.0001). In antenatal clinics, data accuracy improved from 41% to 65% (P < 0.0001); in delivery wards, it improved from 32% to 64% (P < 0.0001).
During the first data audit before the intervention, there was poor correlation between data that appeared in the DHIS and data obtained from facility registers (r = 0.54, Fig. 2). The correlation was much better for data obtained after the intervention during the third data audit (r = 0.92, Fig. 2).
Fig. 2. Correlation between data elementa values observed on data audit and those held in the District Health Information System (DHIS), before the data quality intervention, South Africa, 2007 and 2008
A data quality improvement intervention that involved specific training for health-care workers on the importance of public health information, monthly data reviews and feedback, and regular data audits was effective in significantly increasing the completeness and accuracy of the data used to monitor PMTCT services in South Africa.
Training principally involved the health-care facility and district information officers who were responsible for data collection and was intended to help them understand that high-quality data are essential for improving PMTCT programme performance. It was possible to use monthly district programme review meetings, which were often already under way, as an opportunity to reflect on data quality. In the study, data audits were carried out by a small team of four data clerks. Although we did not undertake a formal costing analysis, these activities are likely to be a relatively inexpensive way of improving data quality in resource-poor settings before more systemic or expensive solutions, such as computerized data management, can be implemented.
Although previous reports have made recommendations for improving public health information, few studies have gone further than describing the inadequacy of the information available.11,20,22,25 One systematic review identified 12 studies of activities for improving data quality: all were conducted in well resourced health-care services and only one appeared to be an interventional study.26,27 However, three other studies reported the outcomes of data quality improvement interventions in resource-limited settings, in Kenya, Malawi and Peru. In Peru, phone reminders to epidemiological surveillance teams and clinic visits were used to improve the timeliness and accuracy of data reported in an electronic surveillance system of infectious disease outbreaks used by the navy. Phone reminders but not clinic visits improved timeliness, whereas in some settings data accuracy was improved by visits but not phone reminders.28 In Malawi, the completeness and accuracy of public health information on HIV testing were reported, though not quantified, using a software package provided by a donor-funded HIV programme.29 In Kenya, investigators used on-site assessment and feedback on data completeness and validity to improve the quality of routine health information: the number of HIV tests recorded and the number of HIV positive patients identified both doubled.30 Finally, a recent study found that sequential data quality audits in six countries improved the quality and accuracy of data on immunization,7 which confirms our finding that regular audits can help improve data quality.
Despite the improvement in data quality observed during this study, the absence of a comparison study arm limits our ability to conclude that there was a strong cause-and-effect relationship with the intervention. Part of the improvement may have resulted from the stringent scrutiny that the South African PMTCT programme received from both programme donors and the department of health. However, we know of no specific efforts to improve data quality in the study facilities or districts at the time of the intervention. The well known inaccuracy of the DHIS has led many nongovernmental organizations, academic bodies and other development partners in South Africa to adopt parallel data collection systems to monitor PMTCT programme performance. These systems, while effective in serving the specific needs of these organizations, were not designed to improve the quality of the information system. An additional limitation of our study is that personnel at participating districts often retrospectively inserted missing data and corrected major errors in the DHIS for up to 1 year. These corrections could have made the data that were available at baseline appear to be better than they actually were and could have resulted in an underestimation of the effect of the intervention.
One unexpected complication that arose during the study could have reduced the effect of the data improvement intervention. The PMTCT programme in South Africa is relatively dynamic and the names and definitions of the data elements used for monitoring are frequently changed. Several challenging changes occurred during the study. For example, the data element used in the DHIS to record whether a baby had undergone a polymerase chain reaction test for HIV at 6 weeks was initially titled “HIV 1st test of baby born to HIV-positive woman” but was later changed to “HIV PCR test of baby born to HIV-positive woman at 6 weeks or later”. Such changes were made without the district offices providing definitions to the clinics. This could have caused considerable confusion at individual facilities and compromised the quality of reporting on that particular data element. To lower the chance of a similar occurrence we selected data elements whose descriptions had remained largely unchanged in recent years.
Despite these limitations, the improvement in PMTCT data quality observed in this study is encouraging, for it suggests that similar approaches could improve the quality of the data needed for decision-making and resource allocation in other public health programmes. The rationalization of data collection tools, clear definitions of data elements, continuous feedback on data quality and intermittent but regular data audits are effective ways of improving data quality. However, while this study shows that public health information can be improved, the final result falls short of what we should accept from our health information systems. The quality of public health data could be improved further. In addition, more work is needed to ensure that the data transmitted from bedside to the public health record are complete and accurate and that the original data collected in consultation rooms accurately reflect the interaction between patient and health-care provider.
In conclusion, this study reports a simple, practical approach to improving the quality of public health information, both locally and in a district health information system. Accurate information is vital for health-care planning and for evaluating progress towards specific health targets. Further research is needed to assess the effectiveness of similar interventions on a large scale, to estimate their cost implications and to clarify the connection between improved data quality and better health service delivery.
The authors thank Hlengiwe Gcaba, Thanda Goniwe, Sibongiseni Gumede, Sibongiseni Khambule, Lungisani Mhlungu, Nirvasha Moodley and personnel at eThekwini Municipality, Ugu, eThekwini and Umgungundlovu health districts and the KwaZulu-Natal Provincial Department of Health. In addition to their primary affiliations, KS Mate is also affiliated with the Department of Medicine, Weill Cornell Medical College, New York, USA, PM Barker is affiliated with the Department of Pediatrics, University of North Carolina, Chapel Hill, USA and N Rollins is affiliated with the Department of Paediatrics and Child Health, University of KwaZulu-Natal, Durban, South Africa.
This study was funded by the United Kingdom Department for International Development and the United States Centers for Disease Control and Prevention.
- AbouZahr C, Boerma T. Health information systems: the foundations of public health. Bull World Health Organ 2005; 83: 578-83 pmid: 16184276.
- Nash D, Elul B, Rabkin M, Tun M, Saito S, Becker M, et al., et al. Strategies for more effective monitoring and evaluation systems in HIV programmatic scale-up in resource-limited settings: implications for health systems strengthening. J Acquir Immune Defic Syndr 2009; 52: S58-62 doi: 10.1097/QAI.0b013e3181bbcc45 pmid: 19858942.
- Chan M, Kayatchkine M, Lob-Levyt J, Obaid T, Schweiyer J, Sidibe M, et al., et al. Meeting the demand for results and accountability: a call for action on health data from eight global health agencies. PLoS Med 2010; 7: e1000223- doi: 10.1371/journal.pmed.1000223 pmid: 20126260.
- Aiga H, Kuroiwa C, Takizawa I, Yamagata R. The reality of health information systems: challenges for standardization. Biosci Trends 2008; 2: 5-9 pmid: 20103892.
- Rugg D, Marais H, Carael M, De Laz P, Warner-Smith M. Are we on course for reporting on the Millennium Development Goals in 2015? J Acquir Immune Defic Syndr 2009; 52: S69-76 doi: 10.1097/QAI.0b013e3181baec7c pmid: 19901629.
- Millennium Development Goals [Internet]. How can we track MDG progress? New York: United Nations Development Programme. Available from: http://www.undp.org/mdg/progress.shtml [accessed 15 November 2011].
- Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Health 2009; 14: 2-10 doi: 10.1111/j.1365-3156.2008.02181.x pmid: 19152556.
- Chaulagai CN, Moyo CM, Koot J, Moyo HB, Sambakunsi TC, Khunga FM, et al., et al. Design and implementation of a health management information system in Malawi: issues, innovations and results. Health Policy Plan 2005; 20: 375-84 doi: 10.1093/heapol/czi044 pmid: 16143590.
- Kimaro HC, Twaakyondo HM. Analysing the hindrance to the use of information and technology for improving efficiency of health care delivery system in Tanzania. Tanzan Health Res Bull 2005; 7: 189-97 pmid: 16941947.
- Burger EH, van der Merwe L, Volmink J. Errors in the completion of the death notification form. S Afr Med J 2007; 97: 1077-81 pmid: 18250917.
- Makombe SD, Hochgesang M, Jahn A, Tweya H, Hedt B, Chuka S, et al., et al. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bull World Health Organ 2008; 86: 310-4 doi: 10.2471/BLT.07.044685 pmid: 18438520.
- Health Metrics Network Secretariat. Health Metrics Network framework and standards for country health information systems. 1st ed. Geneva: World Health Organization, Health Metrics Network; 2006.
- Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan 2009; 24: 217-28 doi: 10.1093/heapol/czp010 pmid: 19304786.
- Hotchkiss DR, Agil A, Lippeveld T, Mukooyo E. Evaluation of the Performance of Routine Information System Management (PRISM) framework: evidence from Uganda. BMC Health Serv Res 2010; 10: 188- doi: 10.1186/1472-6963-10-188 pmid: 20598151.
- Strengthening health information systems. Lewes: Health Partners International; 2011. Available from: http://www.healthpartners-int.co.uk/our_expertise/strengthening_health_information_systems.pdf [accessed 15 November 2011].
- 2008 national antenatal sentinel HIV & syphilis prevalence survey. Pretoria: Department of Health of South Africa; 2009. Available from: http://www.info.gov.za/view/DownloadFileAction?id=109007 [accessed 15 November 2011].
- HIV & AIDS and STI strategic plan for South Africa, 2007-2011. Pretoria: Department of Health of South Africa; 2007. Available from: http://www.info.gov.za/otherdocs/2007/aidsplan2007/introduction.pdf [accessed 15 November 2011].
- Rollins N, Little K, Mzolo S, Horwood C, Newell ML. Surveillance of mother-to-child transmission prevention programmes at immunization clinics: the case for universal screening. AIDS 2007; 21: 1341-7 doi: 10.1097/QAD.0b013e32814db7d4 pmid: 17545711.
- Barron P, Day C, Monticelli F, editors. The district health barometer 2006/07. Durban: Health Systems Trust; 2007.
- Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public programme to prevent mother-to-child HIV transmission in South Africa. PLoS ONE 2009; 4: e5483- doi: 10.1371/journal.pone.0005483 pmid: 19434234.
- Shaw V. Health information system reform in South Africa: developing an essential data set. Bull World Health Organ 2005; 83: 632-6 pmid: 16184283.
- Garrib A, Stoops N, McKenzie A, Dlamini L, Govender T, Rohde J, et al., et al. An evaluation of the District Health Information System in rural South Africa. S Afr Med J 2008; 98: 549-52 pmid: 18785397.
- South Africa Demographic and Health Survey 2003. Pretoria: Department of Health of South Africa; 2004; Available from: http://www.measuredhs.com/pubs/pdf/FR206/FR206.pdf [accessed 15 November 2011].
- South Africa: statistics. New York: United Nations Children’s Fund; 2010. Available from: http://www.unicef.org/infobycountry/southafrica_statistics.html [accessed 15 November 2011].
- Doherty T, Chopra M, Nsibande D, Mngoma D. Improving the coverage of the PMTCT programme through a participatory quality improvement intervention in South Africa. BMC Public Health 2009; 9: 406- doi: 10.1186/1471-2458-9-406 pmid: 19891775.
- Brouwer HJ, Bindels PJ, Weert HC. Data quality improvement in general practice. Fam Pract 2006; 23: 529-36 doi: 10.1093/fampra/cml040 pmid: 16868006.
- Del Mar CB, Lowe JB, Adkins P, Arnold E, Baade P. Improving general practitioner clinical records with a quality assurance minimal intervention. Br J Gen Pract 1998; 48: 1307-11 pmid: 9747547.
- Huaman MA, Araujo-Castillo RV, Soto G, Nezra JM, Quispe JA, Fernandez MF, et al., et al. Impact of two interventions on timeliness and data quality of an electronic disease surveillance system in a resource limited setting (Peru): a prospective evaluation. BMC Med Inform Decis Mak 2009; 9: 16- doi: 10.1186/1472-6947-9-16 pmid: 19272165.
- Douglas GP, Killam WP, Hochgesang MS, Deula RA, Limbe W, Davis MK. Improving completeness, accuracy & timeliness of HIV voluntary counseling & testing client data in Malawi using touchscreen computers. AMIA Annu Symp Proc 2005942- pmid: 16779229.
- Otwombe KN, Wanyungu J, Nduku K, Taegtmezer M. Improving national data collection systems from voluntary counselling and testing centres in Kenya. Bull World Health Organ 2007; 85: 315-8 doi: 10.2471/BLT.06.033712 pmid: 17546313.