Verbal autopsy coding: are multiple coders better than one?
Rohina Joshi a, Alan D Lopez b, Stephen MacMahon a, Srinath Reddy c, Rakhi Dandona a, Lalit Dandona a & Bruce Neal a
a. The George Institute for International Health, University of Sydney, Missenden Road, Sydney NSW 2050, Australia.
b. School of Population Health, University of Queensland, Brisbane, Australia.
c. Public Health Foundation of India, New Delhi, India.
Correspondence to Rohina Joshi (e-mail: firstname.lastname@example.org).
(Submitted: 29 April 2008 – Revised version received: 15 May 2008 – Accepted: 11 June 2008 – Published online: 28 November 2008.)
Bulletin of the World Health Organization 2009;87:51-57. doi: 10.2471/BLT.08.051250
Most high-income countries have well-established mortality reporting systems with complete recording of deaths and medical certification for more than 90% of them.1 The information about the pattern of death in the population that is provided by these systems is important for planning health service provision.2,3 However, most low-income countries lack systems to document mortality rates and causes of death. Most deaths occur at home,4 many go unreported and information about the cause of death is typically absent or unreliable.5–8 In India, about 7 million of the 9.5 million deaths that occur annually happen at home and fewer than half have a medically certified cause of death.9
Mortality surveillance systems based on verbal autopsy are a key source of data about causes of death in less developed countries such as India where limited resources are available for the vital registration process.7,9–13 The verbal autopsy method relies on information gathered from a standardized interview with a relative or caretaker; trained coders review the data and apply standardized diagnostic algorithms to arrive at a cause of death.14–17
The usual practice for cause of death assignment has been to either use a panel of expert physicians or to have two or more physician coders independently review the data and arrive at a diagnosis.9,18,19 In the latter case, any discrepancies between the two diagnoses are resolved by consultation or through review of the verbal autopsy questionnaire by a third physician.19 There has been debate about the impact of different methods of cause of death assignment, but in the absence of empirical data no consensus has been reached.14
In this paper we compare verbal autopsy strategies using a single coder versus multiple coders and report on the impact of each method on the reported patterns of death in an Indian community.
This study was conducted as part of a research collaboration (the Andhra Pradesh Rural Health Initiative) between five Australian and Indian institutions (see the acknowledgements section). The data were collected between 1 October 2003 and 30 September 2004, and the main causes of death have been reported previously.20 Ethics approval for the project was received from the ethics committees of the CARE Hospital, Hyderabad, India, and the University of Sydney, Australia. Written informed consent was obtained from each respondent before the collection of any data, and the project was designed and conducted in line with the Declaration of Helsinki and its subsequent amendments. For participants who could not read or write, the participant information sheet and consent form were explained by a multipurpose primary healthcare worker (MPHW), and a thumb impression was taken instead of a signature. An MPHW is a female non-physician health worker who provides basic health care to the community served by a village health centre. Her role is to collect vital statistics, provide primary health care to the villagers and assist the visiting doctor during clinic hours in the village health centre. In this study, the MPHW carried out the mortality surveillance work in addition to regular clinic activities.
This project was conducted in 45 villages in East and West Godavari districts in the state of Andhra Pradesh in southern India. The population (N = 180 162) age and sex structure was defined by a population census conducted in 2002–2003. The age distribution of the population in the villages is characteristic of that found in places where fertility has recently decreased, with relatively low proportions of the population in the very young and very old age groups. A quarter of the population is below 15 years of age, and a tenth is older than 60 years. Most people are engaged in work related to agriculture or aquaculture, and the average household income is 2000.00 Indian rupees (Rs) (US$ 50.78) a month. The literacy level of the adult population 30 years of age and above is 54%.
Identification of deaths
Each MPHW had primary responsibility for identifying deaths in a village. Identification of deaths by the MPHW was facilitated by her daily contact with the villagers and a network of key informants, including the village headman, the panchayat (that is, the village governing body responsible for registration of deaths), priests, cremation staff and other community leaders. The completeness of identification of deaths was checked between 25 April 2005 and 30 May 2005 by the field supervisor or the MPHWs by visiting every house in every village and checking with residents that all deaths between 1 October 2003 and 30 September 2004 had been recorded.
For each death recorded, the MPHW sought to visit the deceased person’s household within a month of the date of death. The family member or other caretaker best able to report on the events preceding the death was identified, and a systematic enquiry into the events leading up to the death was made using a semi-structured verbal autopsy tool in accordance with an established technique. The verbal autopsy tools used in this project were based on validated verbal autopsy tools used in studies in China,10 the United Republic of Tanzania21 and the Registrar General of India’s Sample Registration System,22 with modifications made to suit local terminology.
Separate questionnaires were used for deaths in each of three age groups (0–28 days, 29 days to < 15 years and ≥ 15 years) and all included a series of structured questions and an open narrative section. The open narrative section was completed with the aid of a defined symptom list with specific questions about treatments, medical procedures and associated documentation. The MPHWs were trained in data collection before the study commenced with refresher training provided after 6 months.
Cause of death assignment
Cause of death was assigned using validated materials and processes developed for the Registrar General of India’s Sample Registration System.22 In a system henceforth referred to as the consensus process, every verbal autopsy was assessed independently by two trained physician coders, each of whom assigned an underlying cause of death. Physician coders were unaware of the decisions about cause of death made by their colleagues. Immediate and contributory causes were assigned wherever possible. Causes of death were selected from the International classification of diseases and related health problems (ICD), 10th revision (ICD-10). Assignment of the causes of death by the physicians was facilitated by a series of algorithms developed for the Sample Registration System.23 In the event of disagreement between the two physicians as to the underlying causes of death, a third physician reviewed the evidence and decided on the underlying cause. Resolution of discrepant decisions was done throughout the study and at the earliest available opportunity.
The main outcome for this study was the proportion of deaths with underlying causes corresponding to those corresponding to chapter headings in the ICD-10. For the three chapter headings that included more than 100 deaths, we also conducted analyses for subsets of causes of death within that chapter. As a result, stroke, coronary heart disease, heart failure and other deaths from cardiovascular causes were analysed separately for the chapter on cardiovascular causes; suicide, falls, transport injuries and other external causes of injury-related deaths were analysed separately for the chapter on injury-related causes; and intestinal infections, tuberculosis, HIV infection and other infectious diseases were analysed separately for the chapter on infectious diseases.
We estimated the primary cost of the surveillance system, which included training meetings for MPHWs and physician coders, a salary component for the MPHWs to cover data collection (about 1 day per week), the salary cost for the project coordinator (employed at 0.5 of a full-time equivalent) and the field coordinator (employed at 0.5 of a full-time equivalent), administrative costs and a unit cost for each death reviewed by physician coders (Rs 250.00 or US$ 5.50).
The level of agreement between the two physician coders for the underlying causes of death at the chapter heading level of the ICD-10 was calculated using Cohen’s kappa statistic (Κ). Values above 0.75 were considered to reflect good agreement; those between 0.40 and 0.75, fair to good agreement; and those below 0.40, poor agreement.24
To compare the population pattern of underlying cause of death as assigned by each physician coder with the pattern derived from the consensus process we prepared graphical presentations of the proportions of deaths assigned to each underlying cause and/or ranked tabulations of these data. The proportions of deaths attributed to each underlying cause were calculated by dividing the number of deaths attributed to a particular cause by the total number of deaths for which a verbal autopsy was performed, with the results expressed as a percentage. Unclassifiable deaths (ICD-10 “R” code) were included as a separate category. Analyses were carried out using SPSS, version 12 (SPSS Inc., Chicago, IL, United States of America).
Costs were estimated by multiplying the various components and adding to obtain subtotals for training, data collection, cause of death assignment and project management. A separate projection of the savings that might be achieved with a single coding strategy was made by subtracting the costs associated with one physician coder and one third of the project management cost. The project management costs included photocopying and couriering of the verbal autopsy reports and the salaries of one half-time field coordinator and one half-time senior project manager. Thus, having single coding would decrease the costs of photocopying and courier by half and the salary costs by a third.
Over the 12-month study period, 1354 deaths were documented in the 45 villages. A verbal autopsy was completed for 1329 (98%) of these deaths, and an underlying cause of death was assigned for 1084 of them (82% of all verbal autopsies). The crude death rate was 7.5 per 1000 population (95% confidence interval, CI: 7.1–7.9), and the death rate for males exceeded that for females in all age groups except that of children 4 years of age or under. The top five causes of death were diseases of the circulatory system (32.4%), external causes (13.3%), infections (11.8%), neoplasms (7.3%) and respiratory diseases (5.3%).
Physician coders assigned the same underlying causes of death at the chapter-heading level of the ICD-10 for 1255 (94%) of the 1329 deaths for which a verbal autopsy was performed. These 1329 deaths represented 98% of the total; the remaining 2% could not be located. The kappa statistic for overall inter-coder agreement was 0.93 (95% CI: 0.92–0.94). The following were the kappa statistics for inter-coder agreement for the four main subdivisions of each chapter: cardiovascular diseases (Κ = 0.87; 95% CI: 0.83–0.91) (Fig. 1); external causes of injury (Κ = 0.94; 95% CI: 0.90–0.98) (Fig. 2); infectious diseases (Κ = 0.96; 95% CI: 0.90–0.98) (Fig. 3). For inter-coder agreement on the causes of death assigned in different age groups, the following Κ were obtained: 1.0 for the 11 deaths in children 0–28 days old; 0.65 for the 68 deaths in children 29 days to 14 years old; and 0.93 for the 1250 deaths in individuals 15 years old or above.
Fig. 1. Proportion of deaths assigned each underlying cause by each of two physician coders and by consensus: cardiovascular diseases, Andhra Pradesh, India, 2003–2004
Fig. 2. Proportion of deaths assigned each underlying cause by each of two physician coders and by consensus: external causes of injury, Andhra Pradesh, India, 2003–2004
Fig. 3. Proportion of deaths assigned each underlying cause by each of two physician coders and by consensus: infectious diseases, Andhra Pradesh, India, 2003–2004
Single versus multiple coders
The patterns of the causes of death assigned by each of the physician coders were similar, and the pattern in the causes assigned by both was practically the same as that derived from the consensus process (Table 1). The rank order of the causes of death assigned in each case was identical for the 10 leading causes, which applied to more than 80% of all deaths for which a specific underlying cause was assigned.
Table 1. Inter-physician agreement for the underlying cause of deatha in a study of verbal autopsies in villages in Andhra Pradesh, India, 2003–2004
Once the materials were developed, the main costs involved in establishing and running the surveillance system included training (of interviewers and physician coders), payment to health workers for data collection and to physicians for the coding of verbal autopsies, and project management. Although a formal economic analysis was not performed, we were able to estimate the costs of the surveillance system retrospectively (Table 2).25 Had there been one instead of two coders, the costs for coding would have been reduced by Rs 169 500 (US$ 3730), and there would have been an additional saving on project management fees of about Rs 103 662 (US$ 2281). Overall, we estimate that use of a single coding system would have reduced the total costs for the one-year surveillance system by about 31% from Rs 891 773 (US$ 19 625) to Rs 618 611 (US$ 13 614).
Table 2. Breakdown of estimated annual cost of implementing a mortality surveillance system, Andhra Pradesh, India, 2003–2004
Our findings suggest that verbal autopsy mortality surveillance systems that use trained physician coders gain little from duplicate coding of causes of death. Although there were some discrepancies in the causes of death assigned to individual cases, they had little impact on the overall pattern of reported mortality. This could be either because there were few errors in assigning deaths to the broad causes or because the duplicate coding process failed to identify any errors made. Either way, there seems to be little support for a system of duplicate coding, and guidelines about the design of verbal autopsy mortality surveillance systems should be reviewed accordingly.14,17,26
Our findings represent an important step forward because the impact of single versus multiple coding in mortality surveillance systems based on verbal autopsy has never been studied in detail, despite the fact that inter-observer and intra-observer variations in coding have been recognized for many years.18,19,27,28
The duplicate coding process would guard against random error and also systematic error. However, the high proportion of deaths for which the same cause was assigned by each of the two physician coders suggests that random errors in coding were rare. Most likely, this consistency in cause-of-death assignment reflects the marked difference between the causes of death listed in each of the ICD-10 chapter headings on which the primary analyses were based. The same reason probably also accounts for the strong agreement between the causes assigned by the coders for each of the major subcategories of deaths attributed to cardiovascular diseases, external causes of injury and infections.
An alternative explanation for the strong agreement between the causes assigned by the two coders is that the duplicate coding process may be a poor method for detecting systematic errors in the assignment of causes. For example, the system does not prevent errors that arise from biases in the data collection process: if the cause-of-death decision is made using information that was incorrectly recorded during the verbal autopsy interview, then duplicate coding will have little effect on the reliability of the final results. Similarly, if poor training of coders resulted in a bias towards a particular cause-of-death assignment, the skew would probably be repeated by subsequent coders because the training and supporting materials provided within a verbal autopsy mortality surveillance system are highly standardized. To better understand the effect of these biases on the final results of the mortality surveillance system, validation studies that compare gold standard diagnoses based on reliable medical records with diagnoses derived from the verbal autopsy method would be needed.
Verbal autopsy studies generally use lay interviewers for data collection and physician coders for interpretation and cause of death assignment.12,27,29,30 Coding of verbal autopsies by physicians has several potential drawbacks. First, physician coders may be biased by their prior knowledge of disease patterns in the community, hence two or more physicians drawn from the same population would be expected to have high agreement. Second, expert physician reviewers are an expensive resource, and using them for this type of work diverts them from clinical roles in settings where physicians are often scarce.
Large-scale mortality surveillance systems that use verbal autopsy, such as India’s Sample Registration System,9,31 the INDEPTH Network32 and the United Republic of Tanzania’s Adult Mortality and Morbidity Project,33 use dual coding for cause-of-death assignment.31,34 Our findings suggest that expenses in programmes such as these could be substantially reduced by switching to a system of single coding. Alternatively, the funds currently used for duplicate coding could be reassigned to conduct validation studies that compare the cause-of-death assignments from single-coding verbal autopsy systems with cause-of-death assignments derived from reliable medical records, diagnosis by autopsy, or physician-diagnosed deaths in the community.
Had a validation study been possible within this project, it would have provided important external confirmation of the pattern of death observed; however, the absence of such data has few implications for our conclusions about the value of a single-coding system versus a multiple-coding system. Changes in the assignment of cause of death resulting from a validation process would have had a broadly similar effect on the pattern reported by each coder and little impact on the level of agreement between them.
A final consideration in the interpretation of this study is that the same three physicians coded all the deaths. Verbal autopsy mortality surveillance systems with a greater number of physician coders may stand to gain from a duplicate cause of death assignment process.
In conclusion, the findings of our study indicate that a single trained physician coder may be as effective as two coders and raise important questions about the value of the system of multiple coders that is currently widely accepted. Since the mortality surveillance system studied here follows the same process as other mortality systems based on verbal autopsy methods, our findings are likely to be broadly applicable to other settings. We believe that workers on current and planned verbal autopsy mortality surveillance projects based on duplicate coding systems should review their need for repeat coding and consider whether investment in external validation studies might not be a better use of resources. ■
The Andhra Pradesh Rural Health Initiative is a collaboration between the Byrraju Satyanarayana Raju Foundation (Hyderabad, India), the CARE Foundation (Hyderabad, India), the Centre for Chronic Disease Control (New Delhi, India), The George Institute for International Health (Sydney, Australia) and the School of Population Health, University of Queensland (Brisbane, Australia). We would like to thank all the Multipurpose Primary Healthcare Workers, the physician coders, the project staff and all the respondents who participated in the study.
Funding: Funding support for the India-based component of this project was provided by the Byrraju Foundation and the Wellcome Trust (grant number GR076471MF). The George Institute’s contribution to this project was made possible by an award from the George Foundation. Rohina Joshi is supported by an international post-graduate research scholarship and international post-graduate award from the University of Sydney, and Bruce Neal by a fellowship from the National Heart Foundation of Australia.
Competing interests: None declared.
- Mathers CD, Fat DM, Inoue M, Rao C, Lopez AD. Counting the death and what they died from: an assessment of the global status of cause of death data. Bull World Health Organ 2005; 83: 171-7 pmid: 15798840.
- Kahn K, Tollman SM, Gear JSS. Validation and application of verbal autopsies in a rural area of South Africa. Trop Med Int Health 2000; 5: 824-31 doi: 10.1046/j.1365-3156.2000.00638.x pmid: 11123832.
- Lopez AD. Causes of death: an assessment of global patterns of mortality around 1985. World Health Stat Q 1990; 43: 91-9 pmid: 2375132.
- Gupta I, Sankar D. Medical attention at death: evidence from India. J Health Manag 2004; 6: 73-84 doi: 10.1177/097206340400600105.
- Chapter 7: Health systems: principled integrated care. In: Shaping the future. Geneva: WHO; 2003.
- Quigley MA, Chandramohan D, Setel P, Binka F, Rodrigues LC. Validity of data-derived algorithms for ascertaining causes of adult death in two African sites using verbal autopsy. Trop Med Int Health 2000; 5: 33-9 doi: 10.1046/j.1365-3156.2000.00517.x pmid: 10672203.
- Aguilar AM, Alvarado R, Cordero A, Kelly P, Zamora A, Salgado R. Mortality survey in Bolivia: the final report. Investigating and identifying the cause of death for children under five. Arlington, VA: Basic Support for Institutionalizing Child Survival; 1998.
- Part 1: Ten statistical highlights in global public health. In: World health statistics 2007. Geneva: World Health Organization; 2007. Available from: http://www.who.int/whosis/whostat2007_10highlights.pdf [accessed on 21 November 2008].
- Jha P, Gajalakshmi V, Gupta PC, Kumar R, Mony P, Dhingra N, et al., et al. Prospective study of one million deaths in India: rationale, design and validation results. PLoS Med 2006; 3: e18- doi: 10.1371/journal.pmed.0030018 pmid: 16354108.
- Yang G, Rao C, Ma J, Wang L, Wan X, Dubrovsky G, et al., et al. Validation of verbal autopsy procedures for adult deaths in China. Int J Epidemiol 2005; 35: 741-8 doi: 10.1093/ije/dyi181 pmid: 16144861.
- Yang G, Hu J, Rao KQ, Ma J, Rao C, Lopez AD. Mortality registration and surveillance in China: history, current situation and challenges. Popul Health Metr 2005; 3: 3- doi: 10.1186/1478-7954-3-3 pmid: 15769298.
- Kahn K, Tollman SM, Garenne M, Gear JSS. Who dies from what? Determining the cause of death in South Africa’s rural north east. Trop Med Int Health 1999; 4: 433-41 doi: 10.1046/j.1365-3156.1999.00415.x pmid: 10444319.
- Khoury SA, Massad D, Fardous T. Mortality and causes of death in Jordan 1995-96: assessment by verbal autopsy. Bull World Health Organ 1999; 77: 641-50 pmid: 10516786.
- Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ. Verbal autopsies for adult deaths: issues in their development and validation. Int J Epidemiol 1994; 23: 213-22 doi: 10.1093/ije/23.2.213 pmid: 8082945.
- Measurement of overall and cause specific mortality in infants and children: memorandum from a WHO/UNICEF meeting. Bull World Health Organ 1994; 72: 707-13 pmid: 7955018.
- Snow RW, Armstrong JRM, Forster D. Childhood death in Africa: uses and limitation of verbal autopsies. Lancet 1992; 340: 351-4 doi: 10.1016/0140-6736(92)91414-4 pmid: 1353814.
- Bang AT, Bang RA. SEARCH team. Diagnosis of causes of childhood deaths in developing countries by verbal autopsy: suggested criteria. Bull World Health Organ 1992; 70: 499-507 pmid: 1394784.
- Garenne M, Fauveau V. Potential and limits of verbal autopsies. Bull World Health Organ 2006; 84: 164-5 doi: 10.2471/BLT.05.029124 pmid: 16583068.
- Soleman N, Chandramohan D, Shibuya K. Verbal autopsy: current practices and challenges. Bull World Health Organ 2006; 84: 239-45 doi: 10.2471/BLT.05.027003 pmid: 16583084.
- Joshi R, Cardona M, Iyengar S, Sukumar A, Raju CR, Raju KR, et al., et al. Chronic diseases now a leading cause of death in rural India-mortality data from the Andhra Pradesh Rural Health Initiative. Int J Epidemiol 2006; 35: 1522-9 doi: 10.1093/ije/dyl168 pmid: 16997852.
- Setel PW, Whiting DR, Hemed Y, Chandramohan D, Wolfson LJ, Alberti KGMM, et al., et al. Validity of verbal autopsy procedures for determining cause of death in Tanzania. Trop Med Int Health 2006; 11: 681-96 doi: 10.1111/j.1365-3156.2006.01603.x pmid: 16640621.
- Registrar General of India, Sample Registration System Academic Partners, Centre for Global Health Research. Registrar General of India Prospective Study of One Million Deaths in India: SRS verbal autopsy form. Available from: http://www.cghr.org/project.htm [accessed on 21 November 2008].
- Registrar General of India, Sample Registration System Academic Partners, Centre for Global Health Research. Manual for assigning causes of death from verbal autopsy, 2005.
- Woodward M. Epidemiology: study design and data analysis. Washington, DC: Chapman and Hall/CRC; 2004.
- Reserve Bank of India. Exchange rate of the Indian rupee vis-à-vis the SDR, US dollar, pound sterling, D.M./euro and Japanese yen. Available from: http://rbidocs.rbi.org.in/rdocs/Publications/PDFs/72784.pdf [accessed on 21 November 2008].
- Verbal autopsies for maternal deaths. London: World Health Organization; 1994.
- Todd JE, De Francisco A, O’Dempsey TJ, Greenwood BM. The limitations of verbal autopsy in a malaria-endemic region. Ann Trop Paediatr 1994; 14: 31-6 pmid: 7516132.
- Fauveau V. Assessing probable causes of death without registration or certificates: a new science? Bull World Health Organ 2006; 84: 246-7 pmid: 16583085.
- Pradhan EK, West KP, Katz J, Christian P, Khatry SK, LeClerq SC, et al., et al. Risk of death following pregnancy in rural Nepal. Bull World Health Organ 2002; 80: 887-91 pmid: 12481211.
- Gajalakshmi V, Peto R, Kanaka S, Balasubramanian S. Verbal autopsy of 48,000 adult deaths attributable to medical causes in Chennai (formerly Madras), India. BMC Public Health. 2002;2:7. doi:10.1186/1471-2458-2-7. Available from: http://www.biomedcentral.com/1471-2458/4/47BMC [accessed on 21 November 2008].10.1186/1471-2458-2-7
- Gajalakshmi V, Peto R. Suicide rates in rural Tamil Nadu, South India: Verbal autopsy of 39 000 deaths in 1997-98. Int J Epidemiol 2007; 36: 203-7 doi: 10.1093/ije/dyl308 pmid: 17301103.
- INDEPTH Network. An international network of field sites with continuous demographic evaluation of populations and their health in developing countries. Available from: http://www.indepth-network.org/ [accessed on 21 November 2008].
- Adult Morbidity and Mortality Project. Policy implications of Adult Morbidity and Mortality Project, Tanzania. 1997. Available from: http://research.ncl.ac.uk/ammp/site_files/public_html/ammp_rep/ammpch08z.pdf [accessed on 21 November 2008].
- Fikree FF, Azam SI, Berendes HW. Time to focus child survival programmes on the newborn: assessment of levels and causes of infant mortality in rural Pakistan. Bull World Health Organ 2002; 80: 271-6 pmid: 12075362.