Bulletin of the World Health Organization

Validation of community health workers’ assessment of neonatal illness in rural Bangladesh

Gary L Darmstadt a, Abdullah H Baqui a, Yoonjoung Choi a, Sanwarul Bari b, Syed M Rahman b, Ishtiaq Mannan a, ASM Nawshad Uddin Ahmed c, Samir K Saha d, Radwanur Rahman b, Stephanie Chang a, Peter J Winch a, Robert E Black a, Mathuram Santosham a, Shams El Arifeen b & for the Bangladesh Projahnmo-2 (Mirzapur) Study Group

a. Bloomberg School of Public Health, Johns Hopkins University, 615 North Wolfe Street, Baltimore, MD, United States of America.
b. Public Health Sciences Division, International Centre for Diarrhoeal Disease Research, Dhaka, Bangladesh.
c. Department of Pediatrics, Kumudini Women’s Medical College, Mirzapur, Tangail, Bangladesh.
d. Department of Microbiology, Dhaka Shishu Hospital, Dhaka, Bangladesh.
e. See Acknowledgements for participants in the Projahnmo-2 Study Group.

Correspondence to Gary L Darmstadt (e-mail: gdarmsta@jhsph.edu).

(Submitted: 29 December 2007 – Revised version received: 02 May 2008 – Accepted: 07 May 2008 – Published online: 18 November 2008.)

Bulletin of the World Health Organization 2009;87:12-19. doi: 10.2471/BLT.07.050666

Introduction

Four million neonatal deaths occur globally every year, representing 38% of deaths in children under 5 years old.1,2 Almost all of these deaths occur in low and middle-income countries, and more than half occur at home.2,3 Improving neonatal health and survival requires cost-effective interventions at the community level, as well as linkages between the community and the health-care system within the continuum of care for the treatment of severe illness.47 Trained community health workers (CHWs) can promote essential newborn care practices at home by educating parents on how to recognize the signs of illness and seek care, and by identifying danger signs through direct assessments at surveillance visits.8,9 Once illness is identified, CHWs can refer sick infants to a facility10 or manage illness at home if a referral is not complied with or is not feasible.1113 Successful management of neonatal illness requires accurate assessment, supported by an effective clinical guideline. Assessment by CHWs has been validated for older infants and children in facility settings,1421 but few studies have documented CHWs’ accuracy in neonatal assessment at the community level,22 particularly in the first week of life, when 75% of neonatal deaths occur.2

The primary purpose of this study was to evaluate the performance of CHWs in recognizing signs and symptoms of neonatal illness during routine household surveillance in rural Bangladesh. Our main objectives were to estimate the validity of a clinical algorithm as used by CHWs to assess specific signs and symptoms and to classify neonatal illness, with physician assessment and classification used as the gold standard.

Methods

Study population and design

This prospective study was nested within Projahnmo-2, a cluster randomized controlled intervention trial of a maternal–neonatal health-care package in Mirzapur, Bangladesh. The neonatal mortality rate in the study area was 24 per 1000 live births in 2002. The area was served by Kumudini Hospital, a 750-bed, non-profit, private referral-level hospital. A population of about 292 000 resided in 12 rural unions, which were randomly allocated to either an intervention (n = 6) or a control arm (n = 6).

In the intervention arm, we divided each union into 6 CHW areas, each having approximately 4000 people. CHWs identified pregnancies through routine bimonthly household surveillance. Pregnant women were visited at home twice during the antenatal period to promote preparedness for birth and newborn care. The mother and neonate were also scheduled for visits on the day of delivery (i.e. day 0) and on postnatal days 2, 5 and 8. During postnatal visits, CHWs assessed neonates for the presence of illness requiring referral to Kumudini Hospital, following the Mirzapur CHW clinical algorithm, adapted from the Bangladesh Young Infant Integrated Management of Childhood Illness (IMCI) protocol (Table 1, available at: http://www.who.int/bulletin/volumes/87/1/07-050666/en/index.html). CHWs ascertained 16 historical factors and 28 clinical signs, assessed breastfeeding and referred sick neonates to the hospital based on the algorithm. A total of 10 407 live births were reported between January 2004 and December 2006, and 7587 neonates were assessed by CHWs at least once (assessment rate 72.9%).

CHW recruitment and training

CHWs (n = 36) recruited through local advertisements were all female, 20–40 years old, and educated to secondary school certification or higher. To fill vacancies, 12 additional CHWs who met the requirements for initial CHW enrolment were recruited and trained, so that a total of 48 participated in the intervention study described above, hereafter referred to as the parent study. The average age of the 48 CHWs in the parent study was 27 years, 79% were married and the average schooling was 11 years.

We developed a manual for initial and refresher training in six areas: pregnancy surveillance and registration; antenatal counselling on preparedness for birth and newborn care; management of the neonate at birth, including resuscitation; continuing essential newborn care; routine neonatal assessment and illness classification; and management of illness according to the Mirzapur CHW clinical algorithm, including referral to the hospital. CHWs received initial training for 36 days (including 6 days of field practice) through didactic sessions, videos and practice on sick and healthy newborn babies in Kumudini Hospital. Their performance was monitored, evaluated and documented throughout the training, and their assessment of five neonates at the hospital was evaluated before they started field work.

Field supervisors used a standard checklist to monitor CHWs’ neonatal assessment and management during field work, at scheduled and random evaluations. In each union, one field supervisor oversaw six CHWs by meeting with them for about 6 hours each fortnight to review all data collection forms and instructive case histories and to provide refresher training. Of the 36 CHWs recruited initially, 26 remained until the end of the project (attrition rate 27.8%). Of the 12 additional CHWs, 10 remained until the end (attrition rate 16.7%).

Mirzapur CHW clinical algorithm

Illness classification

The algorithm classified neonates with any of seven severe conditions requiring referral-level evaluation: very severe disease (VSD), possible very severe disease (PVSD), perinatal asphyxia, significant jaundice on the first day of life, possible gonococcal eye infection, bloody diarrhoea and diarrhoea with severe dehydration. A neonate was categorized as having VSD or PVSD if the CHW observed one or more of the signs and symptoms listed in Table 1. The algorithm also identified neonates with five minor conditions requiring management at home and a follow-up visit: fast breathing, oral thrush, localized bacterial infection, diarrhoea with dehydration and diarrhoea without dehydration (Table 1).

Feeding assessment

CHWs observed breastfeeding for at least 5 minutes to assess attachment, positioning and sucking. For attachment, CHWs assessed the four signs listed in Table 2 to categorize neonates as “well attached” (all four attachment observation points observed), “not well attached” (one or more of the points not observed) and “not attached at all” (neonate could not take the nipple into the mouth and keep it there to suck). For positioning, CHWs assessed the four signs listed in Table 2 to categorize neonates as “well positioned” (all four positioning observation points observed) or “not well positioned” (one or more points not observed). Finally, CHWs assessed sucking to categorize neonates as “sucking effectively” (slow, deep sucking with occasional pausing; apparently satisfied after feeding), “not sucking effectively” (only rapid and shallow sucks; apparently not satisfied after feeding) and “not sucking at all” (not able to suck breast milk into the mouth and swallow). A neonate was categorized as “having any feeding problem” if one or more of the feeding observation points was not confirmed. The complete inability to feed, attach or suck at all was categorized as a sign of VSD.

Management

For neonates recognized as having VSD, PVSD or any of the five specific severe conditions, CHWs recommended referral-level evaluation at Kumudini Hospital. CHWs facilitated transportation, if necessary, and all care at the hospital was free for referred neonates. If the family refused the referral, the CHW continued to encourage it but managed the neonate in the home according to the clinical algorithm.

CHW clinical algorithm validation study

A study was conducted between November 2005 and December 2006 to assess the validity of the clinical algorithm applied by the CHWs in the intervention arm. Study physicians from Kumudini Hospital randomly selected a CHW on each day and performed a complete history (including assessment of gestational age by maternal report of the first day of the last menstrual period) and physical examination of all the neonates who were enrolled in the parent study and were assessed by that CHW in a 24-hour period, except those being seen in follow-up by the CHW after a hospital visit. If a neonate was enrolled more than once for the validation study, we included only the first assessment in the analysis. A study physician completed the same newborn assessment form as the CHW, but male physicians did not assess breastfeeding due to cultural sensitivity. Neonates were assessed independently by a physician less than 12 hours after the CHW’s assessment either at home (for well babies and referral failures) or at the hospital (for successfully referred neonates).

Data

A total of 4228 live births occurred in the intervention arm between November 2005 and December 2006, and 3038 neonates were assessed by CHWs at least once (assessment rate 71.9%), resulting in 10 957 CHW assessments. The study sample comprised 395 randomly selected neonates from the 3038 neonates. Both a CHW and a physician took a complete history and undertook a physical examination; in 317 (80%) cases, a CHW also undertook a complete breastfeeding assessment. A total of 8 study physicians and 36 CHWs participated in the validation study.

Statistical analysis

The unit of analysis was a CHW–physician assessment pair. We examined the association between the assessment by physicians and CHWs at two levels: assessment of individual signs and symptoms, and classification of neonates according to the presence of seven severe illnesses requiring referral to a hospital and of five minor illnesses requiring follow-up observations. The physicians’ assessment and classification were considered the gold standard for calculating sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Kappa statistics (Κ) were calculated to determine agreement between CHWs and physicians: poor (Κ < 0), slight (Κ = 0.0–0.20), fair (Κ = 0.21–0.40), moderate (Κ = 0.41–0.60), substantial (Κ = 0.61–0.80) and almost perfect (Κ = 0.81–1.00).23

We also examined associations between specific feeding problems observed by CHWs and generic feeding problems reported by mothers (as ascertained by physicians). Physicians asked mothers whether the baby had a feeding problem or not without seeking to define the problem. No gold standard was defined, and Pearson χ² tests were conducted to detect associations between each of the specific feeding problems observed by a CHW and each nonspecific feeding problem reported by the mother.

For χ² tests and Κ, a P-value of 0.05 was considered statistically significant. We used STATA 9.0 statistical software (Stata Corporation, College Station, TX, United States of America) for all analyses.

The study was approved by the Committee on Human Research at the Johns Hopkins Bloomberg School of Public Health, and by the Ethical Review Committee and the Research Review Committee at the International Centre for Diarrhoeal Disease Research, Bangladesh, and was registered at clinicaltrials.gov (No. NCT00198627).

Results

Subjects

Of the 395 study neonates, 15.7% were born before 37 weeks’ gestational age, and 88.1% were born at home (Table 3, available at: http://www.who.int/bulletin/volumes/87/1/07-050666/en/index.html). About 70% and nearly all (97.8%) were assessed during the first 7 and 9 days of life, respectively.

Assessment of illness

Historical problems were reported rarely by mothers, but included yellow colour of body (3.8%), feeding problems (2.5%) and skin problems (1.8%) (Table 4). The most frequent signs were a respiratory rate of 60–69 breaths per minute (7.1%); jaundiced palms and soles after the day of birth (3.5%); and moderate hypothermia (2.5%). The frequency of VSD was 2.8%, and that of PVSD was 6.8%. No neonate presented with 5 of the 16 historical factors, 12 of the 28 clinical signs and 7 of the 12 illness classifications in the clinical algorithm (Table 4).

Sensitivity and PPV varied greatly across individual historical factors and clinical signs; standard errors were large due to low prevalence. Severe fever and severe hypothermia had higher sensitivity (50% and 100%, respectively) and better agreement (Κ = 0.67 and 0.66, respectively) than moderate fever (sensitivity 33%; Κ = 0.39) and moderate hypothermia (sensitivity 20%; Κ = 0.16). Sensitivity for fast breathing was fairly low: 25% for a respiratory rate ≥ 70 breaths per minute, and 7% for a respiratory rate of 60–69 breaths per minute. Κ also indicated fair (0.33) agreement for a respiratory rate ≥ 70 breaths per minute and slight agreement (0.08) for a rate of 60–69 breaths per minute. Specificity was better than 97% and NPVs were higher than 93% across all historical factors and clinical signs.

Sensitivity for VSD classification was 73%, specificity was 98%, PPV was 57% and NPV was 99%. Classification of PVSD had lower validity (sensitivity 33%, specificity 97%, PPV 41% and NPV 95%). Κ was 0.63 for VSD and 0.33 for PVSD.

Assessment of feeding

CHWs observed at least one feeding problem in 20% of neonates (Table 2). About 13%, 16% and 6% had at least one sign of an attachment, positioning or sucking problem, respectively; 1.6% were not attached at all and 0.9% were not able to suck at all. Eight neonates (2.5%) were reported to have a feeding problem by their mothers, as ascertained by physicians. Maternal report of any feeding problem was significantly associated with “not sucking at all” and “not attached” as classified by CHWs, as well as with all individual signs of attachment problems observed on the feeding assessment (Table 2). Only one positioning sign (not facing the breast, nose not opposite nipple) was associated with a reported feeding problem.

Discussion

We assessed the validity of a clinical algorithm used by CHWs to assess clinical signs and symptoms of illness and to classify such illness, particularly VSD, in a community-based sample of neonates largely younger than 7 days. Close links between the community and Kumudini Hospital facilitated physician evaluations of neonates at home.10 Classification of VSD by CHWs – based on identification of one or more of 11 signs and symptoms – showed substantial agreement with that of physicians, as well as high sensitivity and high specificity.

Few studies have evaluated the validity of CHW assessment of clinical signs or symptoms or classification of illness in neonates. Kalter et al. included 234 infants aged 1 week to 2 months in their IMCI referral algorithm validation study (26% of their total sample), but the results were not reported separately for the young infants and did not include neonates in the first week of life.24 Mullany et al. evaluated the validity of village-based workers’ assessment of signs of neonatal umbilical cord infection but used physician evaluation of photographs of umbilical cords rather than direct examinations.22 The validity of VSD classification by CHWs compared to physicians’ assessment in our study was similar to that of pneumonia classification reported for children aged 2 months to 5 years in other IMCI studies in hospital settings: sensitivity of 76% in Uganda,18 sensitivity of 81% and specificity of 89% in the Gambia,14 sensitivity of 88% and specificity of 87% in Ethiopia17 and sensitivity of 97% and specificity of 49% in Kenya.21 In the United Republic of Tanzania, significantly higher interobserver variation was reported, even between physicians, in assessments of oedema and unarousable coma in infants compared to children 1–4 years of age.16

Among individual clinical signs, fast breathing (both respiratory rate ≥ 70 and 60–69 breaths per minute) had low validity, which suggests difficulties in measuring respiratory rate in young neonates or variability over time (or both). The sign “weak, abnormal or absent cry” also had low validity, perhaps due to its subjective nature (as suggested by Mullany et al. as an explanation for the low validity of umbilical cord swelling compared to other more objective signs of cord infection).22 In infants aged 2 months to 5 years, Simoes et al. also found that chest in-drawing had lower sensitivity than tachypnoea, a more objective sign, resulting in most of the false negative classifications recorded for severe pneumonia.17 In another study in the United Republic of Tanzania, assessment of chest in-drawing in children aged 4 months to 6 years showed only fair agreement, even between physicians (Κ = 0.338).16 The prevalence of severe chest in-drawing in our cohort was too low to assess validity.

In addition to clinical signs, our CHWs gathered historical information on neonatal illness. A history of a feeding problem, as reported by the mother to a physician, was significantly associated with the presence of a severe feeding problem (particularly a lack of ability to suck) as assessed by CHWs. The sign “unable to feed, or unable to suck at all, or not attached at all” was included in the VSD algorithm. However, assessing breastfeeding is complex, time-consuming and difficult for male physicians due to cultural sensitivity. Given these limitations, a reported history may substitute for an observed feeding problem in the algorithm, substantially simplifying the assessment. Other historical factors in neonatal illness were of limited use in the algorithm, being either too low in prevalence to assess validity or having high specificity but unacceptably low sensitivity. A larger sample is needed for definitive assessment of their validity.

Our study population had a relatively low burden of severe neonatal illness (VSD prevalence 2.8%) compared, for example, to the neonatal sepsis incidence of 10.5% reported in Maharashtra, India.12 For many of the individual signs and symptoms, the frequencies of true positive, false positive and false negative results were too low to estimate validity. Our study population also had relatively low mortality and prevalence of HIV infection and malaria.25,26 Further validation of CHW performance in assessing neonatal illness in other settings with different rates and types of illness (e.g. high HIV and malaria prevalence) is warranted. The agreement possible between CHW and physician assessments was limited by the average lag time of 3.0 hours, during which volatile signs may have changed. The performance of a group of CHWs may also be affected by attrition, as relatively inexperienced workers enter the pool of CHWs. The implications of this issue for programmes will be addressed in a separate report.

Conclusion

We found that CHWs were able to identify a constellation of key clinical signs and symptoms of severe illness with a high level of validity in the context of routine, population-based household surveillance. The use of CHWs to identify and refer neonates with severe illness in the home using an IMCI algorithm is a promising strategy for improving neonatal health and survival in low-resource developing country settings.10 In certain settings with poor facility-based health care, CHWs can also manage neonates.11,12,27

CHWs’ assessments showed high validity for recognizing neonates with severe illness needing referral-level care. Home-based recognition and management – including referral of neonates with severe illness by CHWs – show promise as a strategy for improving neonatal health and survival in low-resource developing country settings. ■


Acknowledgements

The Projahnmo-2 Study Group includes: ASM Nawshad Uddin Ahmed, Saifuddin Ahmed, Nabeel Ashraf Ali, Abdullah H Baqui, Nazma Begum, Robert E Black, Sanwarul Bari, Atique Iqbal Chowdhury, Gary L Darmstadt, Shams El-Arifeen, AKM Fazlul Haque, Zahid Hasan, Amnesty LeFevre, Ishtiaq Mannan, Anisur Rahman, Radwanur Rahman, Syed Moshfiqur Rahman, Taufiqur Rahman, Samir K Saha, Mathuram Santosham, Habibur Rahman Seraji, Ashrafuddin Siddik, Hugh Waters, Peter J Winch and K Zaman.

Funding: This study was supported by the Wellcome Trust – Burroughs Wellcome Fund Infectious Disease Initiative 2000; the Office of Health, Infectious Diseases and Nutrition, Global Health Bureau, United States Agency for International Development (award HRN-A-00-96-90006-00); and the Saving Newborn Lives program of Save the Children-US through a grant from the Bill & Melinda Gates Foundation.

Competing interests: None declared.

References

Share