Bulletin of the World Health Organization

Disability after encephalitis: development and validation of a new outcome score

Penny Lewthwaite a, Ashia Begum b, Mong How Ooi c, Brian Faragher d, Boon Foo Lai c, Indunil Sandaradura a, Anand Mohan c, Gaurav Mandhan b, Pratibha Meharwade b, S Subhashini b, Gulia Abhishek b, Asma Begum b, Srihari Penkulinti b, M Veera Shankar b, R Ravikumar b, Carolyn Young e, Mary Jane Cardosa f, V Ravi g, See Chang Wong c, Rachel Kneen h & Tom Solomon a

a. Brain Infections Group, University of Liverpool, 8th Floor Duncan Building, Daulby Street, Liverpool L69 3GA, England.
b. Vijayanagar Institute of Medical Sciences, Bellary, India.
c. Sibu Hospital and Institute of Health and Community Medicine, Universiti Malaysia, Sarawak, Malaysia.
d. Liverpool School of Tropical Medicine, Liverpool, England.
e. Neuro-rehabilitation Unit, The Walton Centre NHS Foundation Trust, Liverpool, England.
f. Institute of Health and Community Medicine, Universiti Malaysia, Sarawak, Malaysia.
g. National Institute of Mental Health and Neurological Sciences, Bangalore, India.
h. Royal Liverpool Children's NHS Trust, Liverpool, England.

Correspondence to Penny Lewthwaite (e-mail: pennylewthwaite@doctors.org.uk).

(Submitted: 23 August 2009 – Revised version received: 29 November 2009 – Accepted: 04 December 2009 – Published online: 19 April 2010.)

Bulletin of the World Health Organization 2010;88:584-592. doi: 10.2471/BLT.09.071357

Introduction

Neurological disability is a major problem among children in resource-poor countries but the true burden of disability is unknown because there is no simple and reliable way of measuring it.1 The ability to measure disease burden is especially important for Japanese encephalitis, which is a major cause of death and disability in Asia. The disease is caused by the mosquito-borne flavivirus, Japanese encephalitis virus, and is spreading. Recently, there have been large outbreaks in India and Nepal and it is estimated that there are 20 000 to 175 000 cases globally each year.25 Although vaccines against Japanese encephalitis have been available for many years, they have not been widely used, partly because policy-makers lack information about disease burden.4,6,7 Moreover, the proportion of patients reported to have severe sequelae after infection varies widely, from 19 to 71%.811 A major reason for this uncertainty is the lack of a standard method for assessing the outcome of Japanese encephalitis and other forms of acquired brain injury among children in resource-poor countries.

Even in industrialized countries, tools for assessing disability in children are not as well developed as for adults.12 The gold-standard method requires a large multidisciplinary team and involves multiple lengthy assessments over an extended period of time. Although some tools have recently been redeveloped for use in resource-poor settings, they often still require lengthy assessments by trained personnel.13,14 We set out to develop a simple score for assessing disability in children affected by Japanese encephalitis that can be applied by health-care workers with minimal training. We focused on whether the disability was likely to make a child dependent on others, because this is the key issue in terms of disease burden, as well as the single most important parameter for the children themselves. The score we developed, which has become known as the Liverpool Outcome Score,15 was field-tested at two sites in south-eastern Asia: Bellary in India and Sibu in Malaysia. It is also now being used in Bangladesh, Cambodia, Indonesia,16,17 the Lao People’s Democratic Republic and Viet Nam (S Hills, et al. unpublished data, 2008).

Methods

Setting

The new post-encephalitis disability assessment score was developed, piloted and tested in two different clinical settings representative of locations across Asia where Japanese encephalitis occurs: (i) the Vijayanagar Institute of Medical Sciences, which is a government hospital in Bellary in southern India with basic diagnostic facilities but no paediatric intensive care facilities that serves the city (population: 0.5 million) and district (population: > 2 million) of Bellary;18 and (ii) Sibu Hospital, which is a referral hospital in Sarawak, Malaysia, with full intensive care facilities that serves the town of Sibu (population: 250 000) and the central region of Sarawak (population: 650 000).19

The outcome score

A pilot version of the outcome score based on 20 questions was developed after a literature review and the examination of assessment tools used in developed and resource-poor countries, including the Ten Questions screening questionnaire for childhood disability, the Denver II child development screening test, the Paediatric Evaluation of Disability Inventory (PEDI) and three other assessment tools (Fig. 1).2023 Written informed consent was obtained from the parent or guardian of each child. Approval for the study was granted by the ethics committees of the University of Liverpool in the United Kingdom of Great Britain and Northern Ireland and the Vijayanagar Institute of Medical Sciences in India, and the director of health of the state of Sarawak and the hospital director of Sibu Hospital in Sarawak.

Fig. 1. The process of devising a new outcome score for assessing disability after Japanese encephalitis, India and Malaysia, 2005–2007
Fig. 1. The process of devising a new outcome score for assessing disability after Japanese encephalitis, India and Malaysia, 2005–2007
AES, acute encephalitis syndrome; AMPS, Assessment of Motor and Process Skills; JE, Japanese encephalitis.

Participants

After a pilot study in 2006 involving 51 children in India, the score questionnaire was revised and applied in its current 15-question format in 2006 to a cohort of children who had had Japanese encephalitis in Sibu, Malaysia,11 and to controls. The questionnaire is available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm Subsequently in 2007, after minor modification and clarification of the questions, the score questionnaire was applied to a further cohort of children with suspected Japanese encephalitis (defined according to the World Health Organization surveillance standard definition)24 in Bellary, India. Children were invited to attend a follow-up assessment by post in Bellary and via the radio message system in Sarawak. Japanese encephalitis was confirmed using an enzyme-linked immunosorbent assay on cerebrospinal fluid and serum, and patients who tested negative were classified as having acute encephalitis syndrome of unknown etiology. These patients may also have had Japanese encephalitis but, because of sample timing, we were unable to confirm this. Controls were selected at both sites from the siblings of patients who were assessed using the new score and from children who were well and attending the outpatient department for non-neurological conditions.

Application of the score

The score questionnaire requires the assessor to ask the child’s parent or carer to answer direct questions about the child’s ability to perform various daily activities or functions, such as speaking and feeding, in comparison with other children of the same age in their community. It was decided to compare children with others because expected norms vary enormously across communities and no normative data are available. The child is also observed performing simple motor functions, as described in the questionnaire, available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm

For each question, a set of possible answers scored from 2 to 5 is provided. A child whose response to a particular question is completely normal would score 5 for that question. One having minor sequelae that are reported, for example, as mild behavioural problems would score 4. A child having moderate sequelae that affect function but would not lead to dependence (e.g. difficulty walking) would scores 3. A child whose impairment is so great that it would lead to dependence in that setting (e.g. being unable to walk in rural India) would score 2.

Although impairments do change with time, particularly during childhood, it is difficult to predict the change.11 Consequently, for the purposes of the assessment tool, the child is classified on the basis of the individual evaluation alone. The final outcome score for each child, which ranges from I to V, corresponds to the lowest individual score recorded for any single question in the completed score sheet. For example, children whose impairment is severe enough in one domain to make them dependent will be dependent however well they might score in other domains. A score of I is given if the child has died; children who died were not considered further in this study. A score of II corresponds to a lowest single question score of 2 and indicates severe sequelae. Correspondingly, a score of III indicates moderate sequelae, IV indicates minor sequelae and V indicates full recovery.

Although the assessment tool can identify the specific domains in which each child has difficulty, for the purposes of health economic and epidemiological analyses it is more useful to dichotomize children as either “dependent” or “independent” (i.e. likely to be capable of independent living). Children with a score of II were classed as dependent, while those with a score of III to V were classed as independent. Scores in individual domains could also be examined and a total score ranging from 33–75 could be derived from the sum of all the individual scores, but these parameters were not assessed in this study.

Local doctors were trained to use the new outcome assessment tool by discussing cases and with the aid of a PowerPoint (Microsoft, Redmond, United States of America) teaching tool. In both India and Malaysia medical education and training is mainly conducted in English, hence English versions of the questionnaire forms were used.15 Although less than ideal, this was felt to be a practical approach as more than 20 languages are in use in Sibu and more than 6 languages, in Bellary. It was not felt appropriate to translate the written questionnaire into local languages as the written format of some of the languages used is a more formal format than that used in everyday speech. The new outcome score questionnaire was applied by junior physicians who were not otherwise involved in the study.

To investigate the inter-observer and intra-observer variability in the outcome score each child was assessed twice by each of two independent assessors. For practical reasons, these assessments were performed on the same day. However, assessors were unlikely to remember the classification they had given earlier because so many children were assessed in a single day: in Sibu the median number per day was 7 (range: 4–10); in Bellary, it was 6 (range: 2–9).

The new outcome score was validated by comparing each child’s score with the results of a full clinical consultation carried out on the same day. The consultation comprised an assessment by a physician, including history-taking and developmental and full neurological examinations, and an examination by a specialized occupational therapist using the Assessment of Motor and Process Skills,25 which has been validated internationally and cross-culturally for children aged 3 years and older. For children aged under 3 years, the doctor’s assessment alone was performed. Children were classified on the basis of the clinical assessment as having “severe” sequelae, which were likely to make the child dependent, or “moderate”, “minor” or “no” sequelae. The latter three categories were compatible with independent living. The clinical assessors were blinded to the outcome score and vice versa.

In both India and Malaysia, the presence of Japanese encephalitis virus infection was confirmed using standard local assays for detecting Japanese encephalitis virus-specific immunoglobulin-M antibody, as described previously.26,27

Statistical analysis

To give a measure of item redundancy and the internal consistency of the questionnaire, Cronbach’s α was determined during development of the assessment tool for both the pilot 20-question and the final 15-question scores.28 Inter- and intra-observer agreement for the new outcome score and the comparison between the new score and full clinical assessment were all assessed using the kappa (Κ) statistic and 95% confidence intervals (CIs) were computed using the large-sample modified formula.29 The sensitivity, specificity and positive and negative predictive values of the new assessment score relative to full clinical assessment were determined and their 95% CIs were computed using exact binomial formulae. Predictive validity was calculated as the correlation between the new score and clinical assessment. Data were analysed using SPSS version 15 (SPSS Inc., Chicago, United States of America).

Results

The new outcome score

Cronbach’s α was determined for the data on all children assessed using the pilot questionnaire in India and the results were used to revise the questionnaire and to produce the current 15-question version, shown in the questionnaire, available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm

In Sibu, Malaysia, of the 72 children (78%) invited for a follow-up assessment, 56 attended and were evaluated using the 15-question outcome score. The children were assessed a median of 69 months (range: 6–114) after their acute illness. Their median age was 11 years (range: 5–20; interquartile range, IQR: 8–13) and 18 (32%) were female. Forty-two control children (median age: 8 years; range: 3–18; IQR: 6–10) were also assessed, as were two children who had initially been diagnosed with Japanese encephalitis but who were subsequently classified as having acute encephalitis syndrome of unknown etiology after a review of virology results.

The score questionnaire was then used in Bellary, India, in a cohort of 36 children with prior Japanese encephalitis (median age: 8.5 years; range: 4–15; IQR: 6–11; 19 [53%] female) and 41 with acute encephalitis syndrome of unknown etiology (median age: 7 years; range: 2–17; IQR: 5–10; 22 [55%] female). These children were assessed a median of 15 months (range: 1–38) after acute illness. In addition, 19 healthy control children were also assessed (median age: 7 years; range: 3–13; IQR: 4–11; 10 [53%] female).

Each assessment took approximately 10 minutes for individuals experienced in using the new score questionnaire. In total, 779 assessments were made with the new clinical score in 196 children. If problems were identified, they were discussed with carers and referrals were made to local agencies, where available.

Redundancy in the score questions

In the Malaysian cohort, Cronbach’s α for the 15-question outcome score was 0.927 for all observers and children combined and 0.894 for children who had had Japanese encephalitis. In addition, in the Indian cohort, Cronbach’s α was 0.787 for all observers and children combined and 0.849 for those who had had Japanese encephalitis, 0.708 for controls and 0.585 for those with acute encephalitis syndrome of unknown etiology. No significant improvement in the internal consistency of the questionnaire could be made by excluding any of the 15 items and inter-item correlations were acceptable (data not shown).

Inter- and intra-observer agreement

In the Malaysian cohort, there was very good inter-observer agreement (Κ = 0.714) on the outcome score when children were classified according to the severity of their sequelae; the intra-observer agreement was also very good (Κ = 0.943). Moreover, when children were classified according to the dichotomous outcome of being dependent (i.e. a final outcome score of II) or independent (i.e. a final outcome score of III, IV or V), inter-observer agreement was very good (Κ = 0.942) and intra-observer agreement was perfect (Κ = 1.000). In the Indian cohort, inter-observer agreement was moderate (Κ = 0.584) and intra-observer agreement was good (Κ = 0.799) when the severity of sequelae was examined and inter- and intra-observer agreement were good (Κ = 0.786) and very good (Κ = 0.902), respectively, when the dichotomous outcome was examined. Details of these results are shown in Table 1, Table 2, Table 3 and Table 4.

Validation

Outcomes obtained using the 15-question score and clinical assessment were compared (Table 5). Four scores were used for each child: one from each of the two assessments carried out by each of the two observers. When the outcome compared was the severity of the sequelae, a moderate level of agreement was found between the new score and clinical assessment: Κ = 0.544 for the Malaysian cohort and Κ = 0.467 for the Indian cohort. When the outcome compared was the child being dependent or independent, very good agreement was found, with Κ = 0.906 and Κ = 0.762 for the Malaysian and Indian cohorts, respectively.

The sensitivity and specificity of the new score in identifying children likely to be dependent, as determined by clinical assessment, were 100% (95% CI: 89.1–100) and 98.4% (95% CI: 96.5–99.4), respectively, in Malaysia and 100% (95% CI: 91.2–100) and 93.8% (95% CI: 90.7–96.0), respectively, in India. The positive predictive values were 84.2% (95% CI: 68.7–94.0) and 65.6% (95% CI: 52.3–77.3) for the Malaysian and Indian cohorts, respectively, and the negative predictive values were 100% (95% CI: 98.6–100) and 100% (95% CI: 98.5–100), respectively. Overall only 3.8% of children categorized as independent on clinical assessment were incorrectly classified by the outcome score as dependent.

Discussion

The inability to measure disability using a simple tool has been identified as one of the key reasons for the lack of data on disease burden among children living in poor countries.1 The resulting gaps in knowledge mean that there is often insufficient evidence to drive changes in public health policy.31 Nothing provides a better example of this problem than the failure to control Japanese encephalitis over the past 40 years. Without good data on disease burden, the impetus to implement vaccination programmes has been haphazard. As more vaccines become available and as they become cheaper, countries will have to make important decisions about public health priorities.32,33 In particular, simple reliable ways of measuring disability are needed for diseases such as Japanese encephalitis, whose morbidity rate is much higher than the 8–30% mortality rate.10,11

Our aim was to design and validate a disability assessment tool that can be applied relatively quickly and easily by a range of health-care workers in different settings. None of the currently available scores, such as the Ten Questions, Denver II or PEDI score, meets this need. The Ten Questions was devised as a community screening tool to identify children who should be referred for neurological assessment but is too nonspecific for use as an assessment tool.34 The Denver II tool assesses disability in children and is widely used in Europe and North America.21 However, it is usually applied by paediatricians and requires at least 35 minutes. It is also dependent on the cultural setting, though it has recently been adapted for use in Malawi.14 The PEDI is another well-established and widely-used tool. However, it was designed for use in the developed world.20 Finally, the World Health Organization Disability Assessment Schedule II (WHO DAS II), which is in development, assesses patients’ needs, functioning and outcomes but is designed for an adult population.35

In developing the new outcome score we faced considerable challenges and had to accept many compromises. We had to accept that a scoring system would never match an assessment performed over several months by a multidisciplinary team. However, it would still be better than the disease outcome “discharged alive” so often recorded in hospital notes in many parts of rural Asia. We chose to focus on a single disease, Japanese encephalitis, because it is one of the most important causes of acquired brain injury in Asian children. However, the brain injury resulting from Japanese encephalitis is very similar to that associated with other infectious or noninfectious causes, such as trauma. Most disability assessment tools are generic and, with further validation, the new score can perhaps be used more generally across the spectrum of acquired neurological disability.

One limitation of our study was that the proportion of children that responded to a request to attend a follow-up assessment was limited, especially in rural India, where distances to hospital are great. However, we felt it was important to develop the score in real-life settings where it will be used in practice rather than in the logistically easier, but less relevant, setting of a large teaching hospital. We were concerned that sicker and more disabled children might not be able to attend follow-up assessments, but our visits to rural villages to track down nonattendees indicated that it was those who recovered fully that were less likely to attend. Ideally the new score would have been compared with a full multidisciplinary team assessment performed over several visits, but again this was not practical: even assessment by the clinician and occupational therapist took 60–90 minutes.

One of the challenges was to develop a single scoring system that could be applied in a wide range of age groups, in different settings and in areas where there are no normative data. Our solution was to ask the caregiver to compare the child with other children of the same age in the same community. Although this is a crude measure that is dependent on the caregiver, a parent’s judgement of a child’s level of development and abilities is usually correct. This approach allows for cultural differences across Asia; for example, Indian children feed themselves at a younger age than Malaysian children. Cultural differences and the child’s living conditions could mean that an inability to walk would make the child dependent in one setting, for example in rural India, but not another, for example in urban Malaysia, where wheelchairs are available. We felt this was a pragmatic approach because, when looking at disease burden, the impact of a disability is more important than neurological observations or biological dysfunction.

We did not attempt to classify or quantify disablement in terms of impairment, disability (i.e. in activity) or handicap (i.e. in participation). Rather, we developed an assessment tool that identifies children who, after having Japanese encephalitis, suffer a loss of functional ability compared to their peer group. For practical reasons, junior physicians applied the tool in our study, though other health-care workers have now used it without difficulty (unpublished observations). The tool is, if anything, oversensitive in predicting disability, but only 3.8% of children were incorrectly classified as dependent by the outcome score. We felt this was a reasonable proportion since we wanted to ensure that no dependent child was missed.

Recent data show that children with Japanese encephalitis may improve or deteriorate many months after the initial insult.11 Consequently, further work needs to be done in following up a prospective cohort to determine the correlation between the outcome score at hospital discharge with that 3 months and 3 to 5 years later. This information will enable us to determine the time at which the new outcome score will give the best prediction of long-term outcome. In addition, the test–retest reliability of the score now needs to be examined, as does its sensitivity to change over time and its potential for use in acute brain injury due to other causes.

In summary, we have developed a simple outcome score for detecting disability in children affected by Japanese encephalitis, a common cause of acquired neurodisability in Asia. Although the tool has limitations, its ability to identify children with “likely disability”, as judged by the clinical team, was good, with good to very good inter- and intra-observer agreement. It is now being used in several Asian countries affected by Japanese encephalitis and should be suitable for modification to assess acquired neurodisability due to other causes in children in resource-poor countries.


Acknowledgements

We are grateful to all the patients and their carers for assisting with this work. Thanks are also due to staff and patients at the Vijayanagar Institute of Medical Sciences, Bellary, who assisted with the study, and, in particular, the Medical Director and also Kailash Soni and Thomas Schulz. In Malaysia, we thank the hospital staff and patients at Sibu Hospital. We also thank JM Lewthwaite. We also thank Julie Jacobson and Susan Hills of the PATH Japanese Encephalitis Project for their support and encouragement, Mary Gainsborough for help with the initial development of the score and Janet Daly for help with the manuscript. Mong How Ooi and Rachel Kneen also work with the Brain Infections Group, University of Liverpool, Liverpool, United Kingdom and Tom Solomon is Chair of Neurological Science at the Walton Centre for Neurology and Neurosurgery, Liverpool, United Kingdom.

Funding:

The project was supported by the Bill & Melinda Gates Foundation-funded Japanese encephalitis programme at PATH. Mong How Ooi is funded by the Wellcome Trust in the United Kingdom. Tom Solomon is funded by the United Kingdom Medical Research Council.

Competing interests:

None declared.

References

Share