Bulletin of the World Health Organization

Pneumonia case-finding in the RESPIRE Guatemala indoor air pollution trial: standardizing methods for resource-poor settings

Nigel Brucea, Martin Weberb, Byron Aranac, Anaite Diazc, Alisa Jennyd, Lisa Thompsond, John McCrackene, Mukesh Dherania, Damaris Juarezc, Sergio Ordonezc, Robert Kleinf, Kirk R Smithd

Introduction

Acute lower respiratory infections (ALRI) are the single most important cause of death of children under 5 years, responsible annually for approximately 20% of the 10 million under-5 deaths globally.1,2 Prevention strategies are required urgently, including control of risk factors. A growing body of evidence links household indoor air pollution from solid fuels with ALRI in developing countries: recent estimates suggest this may be responsible for nearly one million child ALRI deaths.3 However, these figures are based on relatively few observational studies with considerable variation in ALRI case-finding methods, indirect exposure assessment (using proxies such as fuel type) and risk of residual confounding.4 To address these limitations we conducted a community-based randomized controlled trial with improved chimney stoves in rural Guatemala.

Weaknesses in previous ALRI field studies, and the methodological issues common to trials of environmental interventions, highlighted three particular challenges for this study:

1. To ensure that few cases are missed. Frequent home visits by staff trained to recognize signs such as fast breathing can achieve high sensitivity for ALRI.5 It has been suggested that early treatment associated with more frequent visits may reduce cases of severe ALRI, but a recent review found no evidence of association between surveillance interval (less than 2 weeks) and incidence.1,5

2. To ensure high specificity, as ALRI constitutes a minority (~10%) of all acute respiratory infections. Any impact of reduced exposure on ALRI incidence may be missed if ALRI cases are classified mistakenly with larger numbers of acute upper respiratory infections (AURI). To achieve specificity, all cases identified by fieldworkers should undergo physician examination and preferably chest X-ray (CXR).5

3. To take measures to make physicians’ assessments blind and incorporate objective outcome assessments, as it was not possible for subjects or staff visiting homes to be blind to their intervention status.

This report’s objectives are to describe these methods, evaluate the effectiveness of case-finding and identify any evidence of bias by intervention status. Analysis was carried out using Stata version 9.1.6 An annual ethical review was conducted by the Centers for Disease Control and Prevention (CDC) and the institutional review boards of the Universities of California (Berkeley), del Valle de Guatemala and Liverpool (UK).

Methods

Study area and population

Following extensive feasibility studies,7–13 a rural area of San Marcos in western Guatemala was selected. The indigenous population speaks mainly a Mayan language, Mam, and some Spanish. Wood is the main household fuel, burned indoors on open fires. Key features of the study area are presented in Table 1.

Study design and ALRI case-finding

The study design was a randomized controlled trial comparing ALRI incidence in children ⩽ 18 months using the traditional three-stone fire (controls), with intervention homes using a flued wood stove (plancha):7,14 534 homes with either children under 4 months or a pregnant woman were randomized, and planchas constructed in 269. Sample size was determined to detect a 25% change in ALRI incidence of 0.5 episodes per child per year, at 5% significance, 80% power. Surveillance began after 5 weeks when the planchas were ready, from which time 518 children were followed until the age of 18 months, withdrawal or death. ALRI case-finding was carried out at four levels:

Household visits

Weekly home visits ran from December 2002 to December 2004. Each home was visited on the same day every week. If unavailable, one repeat visit was made on the Friday of the same week. From a total of 21 fieldworkers, 16 were recruited from local community leaders, midwives and health promoters; 14 of these were bilingual (Mam-Spanish) and eight (50%) were female. Fieldworkers were allocated equal numbers of plancha and control homes to visit each day.

The fieldworkers’ questionnaire was based closely on IMCI criteria15 and focus groups identified appropriate Mam terms. All fieldworkers underwent one-week IMCI training in symptom and sign recognition (including video for wheeze and stridor), and classification of children as well, sick but suitable for home treatment or requiring referral to study physician. Respiratory rate was measured over one minute using a timer (UNICEF) and repeated when over 60 in children < 2 months (repeat used).

One supervisor carried out repeat home visit assessments in 10% of homes, while a second supervisor directly observed 10% of home visits. Most repeat assessments agreed with the original findings but the child was re-examined when there were disagreements. All forms were reviewed after each day of fieldwork to identify and correct errors.

Not all weekly scheduled visits could be realized, mostly due to internal migration (Table 2). Although there was a slightly lower rate of completed visits to plancha homes, there were similar numbers of realized visits per child, children with no missed visits and dropouts.

A critical indicator of potential bias was how well fieldworkers adhered to the referral algorithm, by intervention status. No child with specific respiratory signs (raised respiratory rate, chest wall indrawing, stridor or wheezing) was classified as well, but approximately 8% (7.9% plancha, 8.8% control) of cases with raised respiratory rates were not referred, with similar findings for other respiratory signs. All non-referrals with fieldworker-assessed fast breathing had respiratory rates in the ranges considered normal for the next youngest age group (Table 3),15 suggesting that non-referral resulted from uncertainty about ages and thresholds for rapid breathing. Most of these referrals (81%) occurred in the first half of the study.

Fieldworker-assessed symptoms and signs were combined to produce four definitions of new ALRI cases (Table 4): 668 cases met the criteria for lower respiratory illness, plus stridor, a rate of 1.12 (95% confidence interval, CI: 1.03–1.20) episodes per child per year. No cases were classified as well, but 8.9% and 9.3% were classified as unwell and suitable for home treatment in plancha and control groups respectively.

Severe fieldworker-assessed cases (severe WHO pneumonia) were defined as new cases (including nine with non-severe ALRI the previous week) of lower respiratory illness, plus stridor, with chest indrawing and/or inability to drink or breastfeed. There were 72 of these cases, a rate of 0.12 (95% CI: 0.09–0.15) episodes per child per year.

It was expected that not all referred sick children would be taken to the study physician. Almost 80% of referrals for possible ALRI attended the physician before the next weekly visit (Table 4), consistently 3–5% higher in the plancha group (NS). About 70% of all children meeting referral criteria attended the physician before the next weekly visit, 5% more in the plancha group (NS).

Approximately two-thirds of all children with ALRI criteria completed consultations on the day of referral if referred or later that week (Table 4), 5–7% higher in the plancha group (16% for cases with wheeze only), 0.05 < P < 0.1 for outcomes including wheeze. Nearly all (96%) of plancha children referred with severe WHO pneumonia attended the physician, compared to 73% of controls (Fisher’s exact P = 0.02). For 295 visits where a child had ALRI signs but did not see the physician that week, 48 (16.3%) had signs of ALRI at the following weekly visit and 7 of these had severe WHO pneumonia; all 7 were in the control group (P = 0.03).

Fieldworkers also referred 1212 episodes of cough or difficulty breathing, but no specific respiratory sign: 842 (69.5%) attended, and 49 (5.8%) were diagnosed with pneumonia.

Clinical assessment by study physicians

Study physicians assessed children in community centres located up to 1 km from their homes. A standardized history and examination was developed from earlier studies.17,18 Training sessions were held every one to two months at San Marcos Hospital in order to maintain consistent interpretation of clinical signs – each physician assessed children independently, then compared findings with other physicians and the resident paediatrician.

Six Guatemalan physicians were employed during the study: one (SO) worked almost throughout and four carried out the majority (94.6%) of consultations. A total of 1991 consultations among 467 study children were completed for illnesses other than minor skin and eye conditions (recorded separately). Five respiratory diagnoses were used: AURI; otitis media; laryngo-tracheo-bronchitis; pneumonia; and wheezing illness. Pneumonia was categorized as none, possible or definite, based on the physician’s clinical judgment, and new cases were defined as those following at least one weekly home visit without ALRI signs. With the exception of wheezing illness, respiratory diagnoses usually were applied exclusively (Table 5).

Physicians recorded 263 new episodes of pneumonia, a rate of 0.44 (95% CI: 0.39–0.49) episodes per child per year: 216 (82%) from fieldworker referrals, 47 (18%) from self-referrals. The incidence of fieldworker-assessed AURI was 3.44 episodes per child per year based on current respiratory symptoms, and 5.96 when “symptoms since last visit but now resolved” are included, thus between 7.8 and 13.5 times the physician-diagnosed pneumonia rate.

Intervention and control groups reported similar signs used in pneumonia diagnosis, apart from crepitations (Table 6). Using physician-diagnosed pneumonia as the reference, a raised respiratory rate had moderate sensitivity, but poor specificity and positive predictive value (PPV). Chest wall indrawing was recorded in 36% of pneumonia cases but few other cases so while sensitivity was low, specificity and PPV were high. Crepitations were recorded in well over 80% of pneumonias, and in almost no other diagnoses, thus having high sensitivity, specificity and PPV. A lower proportion of intervention pneumonia cases had crepitations recorded (P = 0.02).

Fieldworker classifications were compared against physician diagnoses: only 5.8% of referrals with cough or difficulty breathing but no specific lower respiratory signs were diagnosed with pneumonia, compared to 28.1% of non-severe ALRI referrals and 47.3% of severe ALRI referrals (P < 0.001).

Pulse oximetry

Low oxygen saturation is a complication of ALRI associated with increased mortality.19,20 Study physicians used pulse oximetry to measure oxygen saturation (SaO2) in all consultations. A Sims BCI Mini Corr was used initially, left on the foot for two to three minutes until the reading was stable. From November 2003, Nellcor N-20 oximeters were used. These were more reliable with distressed children, taking six readings on the foot at 30-second intervals (mean used).

To obtain typical SaO2 levels at the study altitude (Table 1), 55 randomly selected study children were investigated by using the Nellcor N-20 and recording whether or not the child was well (reported by mother). Well children had mean SaO2 of 93.2% (standard deviation, SD 3.0; 95% CI: 92.7–93.7), median 93.5% and range 82% to 98% (Table 7), consistent with other studies.19 Hypoxaemic pneumonia cases will be defined as SaO2 > 2 SD below the mean for well children19: for this study, 93.2% – (2×3.0) = 87.2% (rounded to 87%).

Table 8 and Fig. 1 illustrate SaO2 values for children with non-respiratory diagnoses, AURI, wheezing illness (without pneumonia), and pneumonia (possible, definite). Distributions for the diagnostic categories differ overall (Kruskal-Wallis P = 0.0001). Similar statistically significant differences are seen for readings from the Sims and Nellcor oximeters. Since SaO2 was measured during the consultation, it is possible that diagnosis was influenced by the reading, although a substantial number of non-serious diagnoses (AURI, non-respiratory) had low values recorded, and vice-versa.

Fig. 1. Distributions of oxygen saturation readings from children examined by the study physicians, by diagnostic category
Fig. 1. Distributions of oxygen saturation readings from children examined by the study physicians, by diagnostic category

Direct antigen test for RSV

A substantial but widely varying proportion of ALRI cases are reported to be viral,21 although mixed viral and bacterial etiology also occurs.21,22 Respiratory syncytial virus (RSV) is the predominant viral agent.21,23 We are aware of only one study reporting on the risk of RSV ALRI associated with IAP finding a protective effect of exposure for severe hospitalized cases (OR = 0.31 for mothers’ cooking at least once daily, 95% CI: 0.14–0.7).24 If reducing IAP does not reduce risk of RSV-related ALRI, failure to distinguish between viral and non-viral cases would mask any impact on bacterial cases. This has added significance because most ALRI deaths result from bacterial infection, probably due to their invasive nature.21,25–27

An enzyme immunoassay test (Becton-Dickinson Directigen RSV) was performed on all children diagnosed with pneumonia (with parental consent), using naso-pharyngeal aspirate following infusion of 1 ml of normal saline. Kits include positive and negative test samples, used at the start and end of each batch. The product manual reports the sensitivity and specificity of this test to be 93–97% and 90–97% respectively.

RSV tests were completed for 236 (89.7%) of the 263 cases of physician-diagnosed pneumonia, but fewer control (87.1%) than intervention (93.6%) cases (0.05 < P < 0.1). Mean SaO2 was 1.2% lower in cases receiving the test (NS).

Referral for chest X-ray

Initially, children diagnosed with pneumonia were referred to San Marcos Hospital for chest X-ray; from April 2004 a private clinic with better quality control was used. Due to apprehension about hospital, and barriers of travel time and cost, initially a substantial proportion of referrals did not attend. During the first six months, the study team worked to build trust and provided transport twice per week. There was less anxiety about attending the private clinic.

All CXRs used the antero-posterior view, and were conducted in 208 (79.1%) pneumonia cases, more intervention (82.3%) than control (76.3%) cases (NS). Median SaO2 was similar in those who did and did not have a CXR. Acceptance varied over the study: lowest during the first year (70%) but rising to 90% during the second. Following training in standardized methods for pneumonia interpretation developed by WHO, two readers (MG, JB) assessed CXRs.28,29 These were reported as: (1) normal; (2) minor changes; (3) major changes, pneumonia or bronchopneumonia, excluding 4 and 5 below; (4) lobar pneumonia or effusion; and (5) overlap between 4 and 5. Good agreement was achieved on five-way classification (Cohen’s kappa = 0.83), and 16 films with disagreement were resolved through consensus reading by two experienced WHO staff. A combination of categories 3, 4 and 5 are taken as radiological evidence of pneumonia. In an attempt to define outcome more strictly (specifically) as primary end-point pneumonia,28 two additional readers reviewed all films (blind). However, agreement was unsatisfactory – all kappa values for pairs with new readers were < 0.4.

Hospital referral

Children requiring hospital admission were referred to San Marcos Hospital, with transport provided. Following discharge or death, hospital clinical information was added to study records. Two additional pneumonia cases admitted by self-referral were obtained from hospital records. CXRs were available, but neither SaO2 nor RSV status.

Deaths

Deaths were investigated by verbal autopsy16 approximately six weeks after death. Interviews were conducted by the field project coordinator (AD) assisted by a bilingual field supervisor, and analysed blind by four WHO readers. During surveillance, 23 deaths occurred (37 per 1000 child years), of which 9 (39%) were due to pneumonia (14.9 per 1000 child years, 95% CI: 7.7–28.6). Verbal autopsies provided 6 additional pneumonia cases, yielding 271 cases in total (0.45 episodes/child/year, 95% CI: 0.40–0.50) and case fatality of 3.3%.

Discussion

The aim of the first stage of case-finding was high sensitivity. Although this cannot be calculated directly, the 47 self-referrals that resulted in a diagnosis of pneumonia provide some indication of false negatives. This must be viewed alongside the large number of cases of respiratory illness without lower respiratory signs referred by fieldworkers, which contributed 49 physician-diagnosed pneumonias. Overall, the weekly visits, referrals (ALRI and AURI) and self-referrals together probably delivered high sensitivity, a conclusion consistent with observed pneumonia rates. The finding that one-third of fieldworker-defined ALRI cases were diagnosed with pneumonia is consistent with the positive predictive value (PPV) expected from the incidence rates of upper and lower respiratory infections in this study and reported sensitivity and specificity of community case-finding.18 It was reassuring to find no evidence of bias by intervention status in the classification of children for referral.

Self-referrals to the physician did not compensate for the differential compliance with fieldworker referral by intervention status. There were more self-referrals for respiratory illness among plancha children, and slightly more of these cases were diagnosed with pneumonia in the intervention group (14.6% versus 13.0%, NS). This, combined with the higher referral compliance for respiratory illness in the plancha group, indicates that intervention cases had a consistently higher likelihood of attending the physician.

Our assessment of physician examinations focused on evidence of bias in the use of clinical signs for diagnosis. The lower proportion of pneumonia cases among intervention children with crepitations was notable. This may be an intervention outcome rather than bias, as diagnosing pneumonia more frequently among plancha children without crepitations seems counterintuitive if blinding of physicians had not been successful.

The investigations (pulse oximetry, RSV test, chest X-ray) were important for increasing specificity and objectivity in outcome assessment. Pulse oximetry was consistent with diagnosis and severity, and with other studies.19 RSV tests were carried out on almost 90% of pneumonias, but control cases had twice as many missed tests as plancha cases. The trend towards higher intervention group compliance was also seen for CXRs.

The nested case-finding allows a range of outcome definitions for analysis of risk estimates, the degree of blindness and objectivity increasing as more specific criteria are introduced (Table 9). Despite incomplete compliance with referral, the rate of physician-diagnosed pneumonia was close to that assumed for power calculation.

Conclusions

Although case-finding met the stated aims with reasonable success, there are several issues for analysis, in particular the differential compliance with referral and investigations. Although not large at any one level, this occurs at each stage of the process and will impact most on outcomes using RSV and CXR information. To address this, analyses will include risk estimates adjusted for missing data. The problem of differential compliance, combined with no apparent evidence of bias by intervention group in fieldworker classification in this study, might suggest that referral for physician assessment is not useful. However, it would seem wise to retain blinded clinical and objective (investigation-based) assessment in any trial where intervention concealment at home is not possible. Adding pulse oximetry to home visit ALRI assessment may be worthy of further investigation. ■

References

Affiliations

  • Division of Public Health, Whelan Building, University of Liverpool, Liverpool L69 3GB, England.
  • Department of Child and Adolescent Health and Development, WHO, Geneva, Switzerland.
  • Center for Health Studies, Universidad del Valle de Guatemala, Guatemala City, Guatemala.
  • Environmental Health Sciences, School of Public Health, University of California, Berkeley, CA, USA.
  • Harvard School of Public Health, Harvard University, Boston, MA, USA.
  • Regional Office for Central America and Panama, Centers for Disease Control and Prevention (CDC-CAP), Guatemala City, Guatemala.
Share
Email Twitter Facebook Google Delicious LinkedIn
More...