Bulletin of the World Health Organization

Using health-facility data to assess subnational coverage of maternal and child health indicators, Kenya

Isabella Maina a, Pepela Wanjala a, David Soti b, Hillary Kipruto c, Benson Droti d & Ties Boerma e

a. Health Sector Monitoring and Evaluation Unit, Department of Policy Planning and Health Care finance, Ministry of Health, Nairobi, Kenya.
b. Department of Preventive and Promotive Health, Ministry of Health, Nairobi, Kenya.
c. Kenya Country Office, World Health Organization, Nairobi, Kenya.
d. Regional Office for Africa, World Health Organization, Brazzaville, Congo.
e. Center for Global Public Health, Department of Community Health Sciences, University of Manitoba, 750 Bannatyne Avenue, R3E0W2 Winnipeg, Manitoba, Canada.

Correspondence to Ties Boerma (email: ties.boerma@umanitoba.ca).

(Submitted: 21 March 2017 – Revised version received: 03 July 2017 – Accepted: 04 July 2017 – Published online: 28 August 2017.)

Bulletin of the World Health Organization 2017;95:683-694. doi: http://dx.doi.org/10.2471/BLT.17.194399

Introduction

Countries are increasingly focused on the assessment of performance of health programmes at the subnational level. The sustainable development goals further amplify the importance of local data to assess progress and allocate resources to reduce inequalities within countries.1 Coverage of maternal and child health interventions are among the most commonly used measures to monitor the implementation of health programmes at both national and subnational levels.

During the era of the millennium development goals, monitoring the progress of maternal and child health interventions relied heavily on national household surveys. These are conducted about once every five years and provide data on national-level trends and differentials in maternal and child health indicators.2

Health-facility data are another source of population-based statistics for selected maternal and child health and other indicators. For instance, immunization programmes use such data to obtain coverage estimates at the national and local levels.3 Many countries are using health-facility data to monitor annual progress and sometimes to conduct more advanced analyses.47 Scorecards ‒ for instance in the African Leaders Malaria Alliance initiative8 ‒ are increasingly popular and often based on local facility data.

In general, however, concerns about data quality have hampered the use of health-facility data to obtain population-based statistics. Incomplete and inaccurate reporting of events, and the challenge of estimating the size of the target populations, especially at subnational levels, may lead to implausible high (well over 100%) or low coverage results. Survey-based estimates of maternal and child health intervention coverage are considered reliable if the survey design and implementation are of high quality.912 These are often the preferred source to monitor trends and inequalities.

While such surveys can also provide subnational data at the first administrative level (provinces, regions or counties), they do not meet the demand for local coverage data, both in terms of frequency (annual) and disaggregation down to the second administrative level (districts or subcounties).

Recent progress in the implementation of electronic web-based reporting systems allows for easier and faster reporting and better data quality control and feedback. The system most commonly used (in over 40 countries) is district health information software, version 2 (DHIS 2; Health Information Systems Programme, University of Oslo, Norway).13,14 Wider use of DHIS 2 could result in more accurate reporting on the numerators of the coverage indicators for child vaccinations or antenatal and delivery care. If target populations can be estimated more accurately, facility-based coverage can be used for monitoring trends at subnational levels.

The objective of this study was to develop a systematic approach to obtaining the best possible statistics for maternal and child health coverage indicators from health-facility data. The method focused on assessing and adjusting for incomplete reporting of event data from health facilities and on improving estimates of the target populations. We applied the method to Kenya using data from facilities in the 47 counties and from the Kenya demographic and health surveys.

Methods

Table 1 summarizes the four steps of the method and its application in Kenya. The first step is to obtain data and statistics from different sources. The national bureau of statistics provides the official population projections, by age, sex and subnational unit. The most recent population-based survey provides statistics on the coverage of key interventions at national and subnational levels for a specified time period before the survey. Subnational levels include provinces, regions and counties but usually not districts. At the health facility level, data for key maternal and child health interventions – such as ANC first and fourth visit, place of delivery, Caesarean section, first and third dose of pentavalent vaccination and measles vaccination – are obtained for multiple years (preferably at least three years) to be able to assess consistency over time. In most countries using DHIS 2, these data are derived from paper-based recording and reporting in almost all facilities. The monthly facility reports are then sent to the district or subcounty health office where the data are entered into DHIS 2 and uploaded to the Internet. However, some facilities (mainly hospitals) enter the data directly into DHIS 2.

Step 2 starts with assessing the quality of the numerator of the coverage indicator by analysing completeness of reporting and consistency over time. High levels of reporting (over 80% of health facilities reporting a specific indicator) are essential to be able to compute coverage rates. Internal consistency is checked in terms of trends over time for coverage of each indicator, as well as between first antenatal visit and first pentavalent vaccination, and between first and third pentavalent vaccinations, as recommended by the World Health Organization.15 Outliers, defined as more than two standard deviations from the mean values of the numerators for the multi-year period, are identified and corrected if no satisfactory explanation is found for the outlier value.

For the coverage calculations, we need to adjust for incomplete reporting by facilities. This involves making assumptions about the number of service outputs (pregnancy care, vaccinations, etc.) provided at facilities which did not report compared with those that reported. The adjustment can be expressed as follows:
n(adjusted) = n + n(1/(c)-1)*k

where n  is the number of service outputs, c is the reporting completeness, k is the adjustment factor. If we consider the missing reports an indication that no services were provided during the reporting period, then k = 0, and no adjustment is made for incomplete reporting. However, if facilities provided services but not at the same level as before reporting periods, the apparent incomplete reporting is an indication of a lower level of service provision; k in this case is between 0 and 1. In other cases, it may be assumed that services were provided at the same rate in non-reporting facilities as in reporting facilities, and so k = 1. Important considerations in the selection of a value of k are the extent to which large health facilities and private health facilities are reporting and engaged in the provision of the specific services. This is likely to be different for different services, resulting in different adjustment factors. Subsequently, the selection of the most likely value of k is done through a comparison of facility reports with the survey results, by selecting a value of k that brings the adjusted health facility statistic close to the survey statistic for a particular year with data from both sources.

Step 3 is about finding the best possible denominator or target population size. This is usually obtained from census projections by the country’s national bureau of statistics. Often, problems with the projected subnational denominators lead to unexpectedly high or low coverage rates. An alternative approach is to derive the population size from health facility data on indicators with near-universal coverage (at least 90%), such as the first antenatal visit or the first dose of pentavalent vaccine (normally given at 6 weeks of age). If the health facility reports are of good quality, and almost all children are vaccinated, the first vaccination or first antenatal visit numbers should be very close to the actual target populations. Only a small proportion is added to the reported first pentavalent vaccination or first antenatal visit numbers to account for those who did not receive them (< 10% of people, according to household surveys in many countries).16,17 The estimated young infant target population can then be used to obtain target populations for other maternal and child health coverage indicators (e.g. live births, deliveries, pregnancies and older infants), based on available statistics from recent surveys or other sources.

In step 4, the adjusted numbers and denominators are used to calculate the subnational coverages of immunizations, antenatal care (first and fourth visits) and facility-based deliveries.

In the second half of 2016, we used data from Kenya to apply and refine the method. Health-facility data were analysed across the 47 counties for the fiscal years (1 July to 30 June) 2012/13, 2013/14, 2014/15 and 2015/16 and compared with survey results from the most recent Kenya demographic and health survey in 2014.18 Population projections were obtained from the 2009 Kenya national census.19

All calculations were done using Microsoft Office 365 Excel software version 1705 (Microsoft Corporation, Redmond, United States of America). The spreadsheet with data by county and the adjustment procedure are available from the corresponding author.

Results

Data quality assessment

In Kenya in 2015/16, the national reporting completeness for the vaccination reporting forms was high (93.7%; 69 470/83 179 expected monthly reports; Table 2) and <  80% in only one county. This represented a modest increase in reporting rates since 2012/13 (national rate 89.4%; 60 450/72 384; <  80% in 12 counties). Also, the reporting rates were high throughout the period for antenatal and delivery care forms (from 85.9%; 72 384/84 276, in 2012/13 to 94.9%; 83 179/87 684, monthly reports in 2015/16). There were no outliers at the level of the 47 counties, which indicates good consistency over time. The internal consistency between numbers of first and third doses of pentavalent vaccine was also good, since the first vaccination values were higher in all counties and years, as expected, and the size of the difference between the two doses corresponded well with the survey drop-out rates (Table 2). The numbers of first pentavalent vaccinations and first antenatal visits in counties were similar, as expected on the basis of the Kenya demographic and health survey 2014 results.18 The numbers of first pentavalent vaccinations exceeded first antenatal visits in most counties, suggesting more complete reporting. Therefore, we used first pentavalent vaccination as the key indicator to obtain denominators.

We selected the adjustment factors based on our knowledge of the maternal and child health programmes, the types of facilities where care was provided and comparison with the survey-based statistics. For vaccination coverage the adjustment factor k is likely to be low as vaccine supplies are directly linked to reporting. We used k = 0.25 as some vaccinations may still be given in non-reporting facilities. This value of k also had good agreement with the national coverage rate for first pentavalent vaccination in the demographic and health survey.18 For antenatal and especially delivery care, the non-reporting facilities included a higher proportion of private facilities and most of those provided pregnancy related-services. In the demographic and health survey, one-quarter of all facility-based deliveries took place in private health facilities. To adjust for incomplete reporting, k was set at 0.5 for antenatal care and at 1 for deliveries, bringing the health-facility-based rates close to coverage rates for the three years preceding the survey.

County coverage rates based on the census population projections indicated that there were major denominator issues. Several counties had denominators that were too low (six counties – all in the northern parts of Kenya – consistently had coverage estimates exceeding 120%), while other counties had unlikely low coverage of the first pentavalent vaccination (11 counties were consistently below 80%). Because of these challenges with the accuracy of the census projections at the county level, our confidence in the quality of the facility reports on first pentavalent vaccination and the near-universal national coverage of this vaccination in the household survey data, we used the numbers of children with first pentavalent vaccination as an alternative estimate of the denominator or target population at the county level. To obtain the target population size, we added an estimate of the number of non-immunized infants (3% of the target population, based on the demographic and health survey)18 to the adjusted number of vaccinations, resulting in 1.33 million infants eligible for vaccination in 2015/16. We used this denominator as target population for all vaccinations. For live births, we added 2.0% to the denominator to include neonatal deaths. For deliveries, we reduced the number of live births by 1.5% to allow for multiple births20 and added 2.0% to allow for stillbirths. For pregnancies, we added 3.0% to deliveries to account for fetal loss before stillbirths. Early fetal losses are generally not included as in Kenya these mostly occur before the first antenatal care visit is made (median first visit is made at 5.4 months according to the demographic and health survey).18

Infant vaccination coverage

The four-year trend in vaccination coverage from facility data for all Kenya showed flat or slightly declining coverage for the third dose of pentavalent vaccine and for measles vaccine (Table 3). Third pentavalent vaccination levels were consistent with the survey-based statistics. Measles vaccination coverage was somewhat higher in the facility data than in the survey data (vaccinated by 12 months among children aged 12‒23 months), which may be due to facility reporting of some vaccinations given to children after their first birthday, as the values were very close to the values in the 2014 demographic and health survey, based on children aged 12–23 months. The full vaccination coverages were considerably higher in the facility data than in the survey data, and also implausibly high compared with the coverage of the specific vaccinations. It is likely that over-reporting of full vaccination status occurred in the facility reports.

The differences by county within Kenya were substantial (Fig. 1). In 2015/16, 28 counties had third pentavalent vaccination coverage of 90% or higher, while five counties, all in northern Kenya, had third pentavalent vaccination coverage below 80%. In 25 of the 47 counties the third pentavalent vaccination coverages in 2015/16 were lower than in 2012/13.

Fig. 1. Pentavalent vaccine coverage (receiving three doses in infancy) from health-facility data, by county, Kenya, 2012/13 and 2015/16
Fig. 1. <b>Pentavalent vaccine coverage (receiving three doses in infancy) from health-facility data, by county, Kenya, 2012/13 and 2015/16</b>
Notes: Health-facility data were from routine reporting via DHIS 2 (district health information software, version 2.0) for fiscal years (1 July to 30 June). Adjusted values are shown, aggregated by county. Numerators for facility data were adjusted for incomplete reporting; denominators were derived from facility data on the first dose of pentavalent immunization. Coverage denominators for facility data are in Table 4. Kenya value, shown in grey, is the weighted mean of the 47 counties.

Antenatal and delivery care

Based on the adjusted facility data, first antenatal visit coverage was near-universal and close to the demographic and health survey results (Table 3 and Fig. 2). The consistency between facility and survey data was less satisfactory for the proportion of pregnant women who made four or more antenatal care visits. Household survey data (from two demographic and health surveys,18,22 and also from the Kenya malaria indicator survey 2015)21 showed an increase to 61.5% for the three years preceding the survey (midpoint shown in Fig. 3). The facility reporting data also showed an increase during the period 2012/13 to 2015/16 but at a lower level than the surveys. Since this is unlikely to be due to a problem with the size of the target population, it could be attributed to underreporting of four antenatal visits by health facilities or over-reporting of the number of visits in household surveys.

Fig. 2. Antenatal care visits coverage (first visit and four or more visits) comparing health facility and survey data, Kenya, 2007/08 to 2015/16
Fig. 2. <b>Antenatal care visits coverage (first visit and four or more visits) comparing health facility and survey data, Kenya, 2007/08 to 2015/16</b>
DHIS-2: district health information software, version 2.0.
Notes: Health-facility data were from routine reporting via DHIS 2 (district health information software, version 2.0) for fiscal years (1 July to 30 June) and aggregated for the 47 counties. Survey data were from the Kenya demographic and health survey 2014.18 Coverage denominators for facility data for 2013/14 and 2015/16 are in Table 3. Denominators for survey data for 2011/12 and 2009/10 are in Table 3 and for 2007/08 n = 3101 births.
Fig. 3. Facility-based delivery coverage and caesarean section delivery rate comparing health facility and survey data, Kenya, 2007/08 to 2015/16
Fig. 3. <b>Facility-based delivery coverage and caesarean section delivery rate comparing health facility and survey data, Kenya, 2007/08 to 2015/16</b>
DHIS-2: district health information software, version 2.0.
Notes: Health-facility data were from routine reporting via DHIS 2 (district health information software, version 2.0) for fiscal years (1 July to 30 June) and aggregated for the 47 counties. Survey data were from the Kenya demographic and health survey 2008‒09,22 Kenya demographic and health survey 201418 and Kenya malaria indicator survey 2015.21 Coverage denominators for facility data for 2013/14 and 2015/16 are in Table 3. Denominators for survey data for 2011/12 and 2009/10 are in Table 3 and for 2007/08 n = 3101 births.

The facility data showed an increase in the proportion of women delivering in health facilities from 61.3% (815 959/1 325 124 deliveries) in 2012/13 to 76.9% (1 049 285/1 357 956 deliveries) in 2015/16, up from the survey estimate for 2012‒2014 of 64.0% (6 642/10 378 births) of births in health facilities. The variation in coverage of facility-based delivery by county was considerably greater than that for vaccination coverage, with values over 90% in ten counties and less than 60% in nine counties in 2015/16 (Fig. 4). Health facility delivery coverage was higher in 2015/16 than in 2012/13 in 44 of the 47 counties. The increase was large in almost all counties, and often greater in the lower coverage counties.

Fig. 4. Facility-based delivery coverage from health-facility data, by county, Kenya, 2012/13 and 2015/16
Fig. 4. <b>Facility-based delivery coverage from health-facility data, by county, Kenya, 2012/13 and 2015/16</b>
Notes: Health-facility data were from routine reporting via DHIS 2 (district health information software, version 2.0) for fiscal years (1 July to 30 June). Adjusted values are shown, aggregated by county. Coverage denominators are in Table 5. Kenya, shown in grey value is the weighted mean of the 47 counties.

The number of caesarean sections per 100 deliveries in the population also increased from 7.8 (103 785/1 325 124) in 2012/13 to 10.8 (147 463/1 357 956) in 2015/16 (Table 3), corresponding to the increased proportion of women delivering in health facilities.

Discussion

Health-facility data obtained from routine reporting systems are an important tool for assessing progress at subnational levels. This study presented a systematic approach to analysing routine health-facility data, focusing on data quality assessment and adjustment and obtaining denominators from data for interventions with near-universal coverage. Applying the method to Kenya showed that health-facility data can provide up-to-date information to monitor recent subnational and national coverage trends for key maternal and child health indicators. This study was conducted as part of the midterm review of the implementation of the Kenya health sector strategic and investment plan 2014–2018.23 In this plan the assessment of progress and performance towards the midterm targets in mid-2016 relied heavily on facility data because the last survey with maternal and child health indicators took place in 2014.

Our results provide important information on the maternal and child health component of the implementation of national and subnational health plans. Vaccination coverage rates stagnated or declined modestly, but were still at a high level. In most counties coverages were lower in 2015/16 than four years earlier. The Kenya strategic plan targets and the global goal of reaching and sustaining 90% national full vaccination coverage and 80% in every district or equivalent administrative unit for all vaccines included in the national programme24 were far from being met.

Deliveries in a health facility increased rapidly during the period 2012/13 to 2015/16. While household surveys showed a major increase to 64.0% before the Kenya strategic plan, the facility data indicated a continued national increase to 76.9% in 2015/16, driven by increases in 44 of the 47 counties. In 2013, Kenya introduced a free maternity initiative in all public health facilities, to encourage women to deliver in facilities. Even though it is not possible to wholly attribute the current trends to this initiative, the results obtained from the facility data are encouraging, confirming continued rapid increases in deliveries in health facilities. Further efforts are needed to concentrate in the nine counties with more than 40% of deliveries occurring at home (mostly located in the northern and more sparsely populated areas of Kenya) and in the counties with the largest numbers of home deliveries (located in western Kenya). Furthermore, caesarean section rates were increasing in almost all counties, proportional to the increases in institutional deliveries. Coverage of antenatal care with at least four visits was also increasing but much slower than delivery care coverage and was still only just over 50% in 2015/16 according to the facility data. Household survey data, based on recall by mothers, may overestimate the number of antenatal visits.

Analysis of the health-facility data was possible due to several factors present in Kenya that provide lessons for many other countries now implementing DHIS 2. This also highlights the limitations of this type of analysis. First, the health ministry, both at national and county levels, has a strong commitment to the health facility reporting system. The government made it mandatory that all programmes use the same system for collection of facility-based indicators to ensure that the systems are interoperable. The only exception to date is the disease surveillance system which is not yet fully integrated. The health ministry is also strongly committed to sharing the DHIS 2 data, in line with the Kenya government’s open data initiative.25 The devolution has stimulated the interest in county-level monitoring.

Second, the reporting system has been functioning well. Reporting rates are high and have increased to over 90%. The private sector is included, even though reporting rates are still lower than for the public sector. We adjusted the numbers of reported events for incomplete reporting by making assumptions about the extent to which non-reporting facilities would be different from reporting facilities and using the survey data as an external validity check. This is a somewhat arbitrary process, but the impact of the adjustments is generally relatively modest if reporting rates are high. If reporting completeness is below 80%, adjustment procedures will have a greater impact and facility statistics will become less reliable. Other methods to adjust for missing values include geospatial methods, which have for instance been used in Kenya for estimating outpatient visits rates from facility data.26

Third, the facility data were of good quality, as shown by good consistency over time, consistency across indicators and external comparison with surveys. In Kenya, the districts or counties, supported by the health ministry, usually compute the reporting rates, check for data inconsistencies and do follow-ups to ensure high levels of reporting and accuracy of data. DHIS 2 now includes a standardized module to check for inconsistencies and outliers which makes it easier for staff at county and national health offices to identify problems and follow up with action.15 Previous research also indicated relatively good quality of facility data in Kenya.27

A fourth factor was the availability of an accurate estimate of the target population for the indicator, or the denominator of the coverage estimate. In Kenya, many counties had identified major problems with the denominators provided as part of the official population projections based on the 2009 census. Here, we used county reports on the number of vaccinations with first dose of pentavalent vaccination to obtain denominators for the maternal and child health indicators. This can only be done if the numerators are accurate, with high reporting rates and good quality of data. Supplemental immunization activities, in which children are vaccinated outside of clinical settings,28 are not likely to affect the usefulness of first dose of pentavalent vaccination numbers to obtain a denominator.

Fifth, recent (up to 3‒4 years ago) household survey data are necessary to be able to calibrate the denominators. It has to be kept in mind, however, that surveys are not the absolute gold standard, as the survey results are affected by sampling error (which can be large, especially at subnational levels) and non-sampling error related to recall bias or the quality of the survey implementation.

Lastly, a specific advantage for this study was that Kenya’s unit of analysis – the county – is relatively large (almost all counties have populations exceeding 500 000) which helps to obtain more stable estimates of numerators and denominators. The methods, however, have potential for use in smaller populations, such as subcounties or districts, as target populations are based on the actual volume of health services provided to the same population rather than population projections.

Surveys will continue to be necessary to provide population-based data on a range of maternal and child health coverage indicators and determinants. However, the introduction of national web-based information systems for health-facility data provides an opportunity for more frequent monitoring of progress at the national and subnational levels. This study shows how improvements in the timeliness, completeness and accuracy of a new web-based reporting system can provide a sound basis for subnational and national statistics on key maternal and child health indicators. This approach can be extended to obtain statistics for other indicators, such as stillbirth rates, postnatal care coverage and outpatient attendance.29 The main application of this approach lies at subnational levels where regular monitoring of progress and performance has the greatest potential to improve service delivery and targeting of interventions.


Competing interests:

None declared.

References