Bulletin of the World Health Organization

Are skilled birth attendants really skilled? A measurement method, some disturbing results and a potential way forward

Steven A Harvey a, Yudy Carla Wong Blandón b, Affette McCaw-Binns c, Ivette Sandino d, Luis Urbina b, César Rodríguez b, Ivonne Gómez b, Patricio Ayabaca e, Sabou Djibrina f, the Nicaraguan maternal and neonatal health quality improvement group

Introduction

Background

Each year obstetric complications kill over 500 000 women worldwide.1,2 Skilled attendance during labour, delivery and in the early postpartum period could prevent many of these deaths, though establishing a causal link between skilled attendance and maternal survival remains problematic.26 Still, the proportion of deliveries assisted by a skilled birth attendant (SBA) has become an indicator for measuring maternal mortality reduction, including the 75% reduction called for by the fifth Millennium Development Goal (MDG-5).7,8

WHO defines an SBA as someone “trained to proficiency in the skills needed to manage normal (uncomplicated) pregnancies, childbirth and the immediate postnatal period, and in the identification, management and referral of complications in women and newborns”.9 WHO uses household survey data to estimate the percentage of women delivered by an SBA in each country.10 (An explanation of the methodology for determining the percentage of women delivered by SBAs is available at: http://www.who.int/reproductive-health/global_monitoring/skilled_attendant.html#methodology. WHO’s most recent estimates of the proportion of women delivered by SBAs are available at: http://www.who.int/reproductive-health/global_monitoring/skilled_attendant_at_birth2006.pdf). Typically, these surveys ask women what type of health professional, if any, assisted at their most recent deliveries.1114 Most surveys report results by cadre: doctor, midwife, nurse, traditional birth attendant, relative and other. Some categorize doctors, midwives and nurses as health personnel to distinguish them from untrained attendants. Since surveys make no claim to ascertain provider skill, using survey data to estimate the proportion of SBA-assisted deliveries assumes that all health personnel qualify as SBAs.15 (In this paper, consistent with the International Confederation of Midwives definition, the term midwife or nurse-midwife means a provider who has graduated from a certified or accredited midwifery training course in the country of practice. These programmes vary in content and duration and may or may not include medical or nursing training beyond midwifery skills. Nurse refers to a provider who has completed nursing training that is not part of a certified or accredited midwifery programme, even when midwifery functions are performed.)15

But are skilled birth attendants really skilled? Do the health personnel enumerated by household surveys fit WHO’s definition? This question was the focus of our two-phase study. In Phase I, we developed and piloted evaluation instruments, then carried out small-scale competency assessments in four countries. In Phase II, shortcomings identified in Phase I were corrected and the revised instruments used to conduct a larger-scale assessment. The results contribute to ongoing competency-improvement efforts.

Methods

We evaluated 166 health providers in Benin, Ecuador, Jamaica and Rwanda during Phase I. In Phase II, we tested 1358 Nicaraguan providers. In each country, our aim was to evaluate the professionals responsible for most deliveries – those most likely to appear in WHO estimates. Like WHO, we excluded traditional birth attendants. Table 1 summarizes maternal health characteristics by country.

Instrument development

We defined competence as possessing skills and knowledge sufficient to comply with predefined clinical standards. In a multi-country setting, this presupposes standards accepted by all providers and sites. Since no such standards exist, we measured competence against WHO’s Integrated Management of Pregnancy and Childbirth (IMPAC) guidelines, an appropriate evidence-based yardstick.16

The three leading direct causes of maternal death are haemorrhage, pre-eclampsia and eclampsia, and sepsis.17,18 We designed instruments to test competence related to prevention, diagnosis and management of these complications, plus management of uncomplicated labour and delivery. We evaluated knowledge with a written test and skills with anatomical models. We ruled out direct clinical observation for three reasons:Complications occur in approximately 15% of pregnancies and cannot be predicted.19,20 It would be prohibitively time-consuming and expensive to observe each participant managing each complication. Ethics would compel intervention if a clinically experienced observer observed sub-standard care. In clinical settings, many factors affect performance. Without essential drugs, equipment or supplies, a highly competent provider might perform poorly. With competent assistants and all necessities at hand, a marginal provider might perform well. Since environment varies by facility, we elected to test competency in non-clinical venues where we could control for environmental factors.The initial knowledge test was modelled on MotherCare, and Maternal and Neonatal Health Program (MNH) materials.21,22 Additional sources included IMPAC guidelines, participating countries’ norms, and the Safe Motherhood Inter-Agency Group’s SBA competencies.5,16,23,24 We adapted two MNH case studies in order to evaluate partograph skills. For initial skills testing, we adapted MNH structured observation checklists for neonatal resuscitation, manual removal of the placenta and bimanual uterine compression.21 We developed an intravenous (IV) insertion skills checklist. Senior clinicians used these checklists to assess each participant’s skill. To standardize observation criteria, we trained clinical observers using the following regimen:The country study coordinator reviewed each checklist line-by-line with observers, clarifying any perceived ambiguities. One observer performed the first procedure on an anatomical model. All others used the first checklist to rate the skill demonstrated in this mock evaluation. Observers then compared checklists step-by-step to resolve differences. The coordinator served as final arbiter. The group repeated steps 2 and 3, with different observers role-playing the evaluee, until 100% agreement was achieved.Typically, observers achieved 100% consensus after four to five practice observations. The principal investigator (Steven Harvey) supervised training in Benin, Ecuador and Nicaragua. Co-investigators supervised training in Jamaica (Affette McCaw-Binns) and Rwanda (Sabou Djibrina).

We piloted the instruments in Ecuador in November 2001, then conducted small-scale evaluations in the Phase I countries from March to June 2002. Phase I participants needed much longer than anticipated to complete the evaluation. Many struggled with complex case-based questions in the knowledge test. For Phase II, we simplified the test language, eliminated case-based questions and adopted a format that was all multiple-choice. We added material from the American College of Nurse-Midwives (ACNM) life-saving skills curriculum and JHPIEGO guidelines for the assessment of skilled providers.25,26 On expert advice, we replaced IV insertion, and mouth-to-mouth and nose resuscitation checklists with one focused on active management of the third stage of labour (AMTSL) and another on immediate newborn care.

In both phases, we relied on multiple rounds of expert review to establish construct and content validity for our instruments. Reviewers included obstetricians/gynaecologists, paediatricians and midwives at the international level and in each study country. To ensure accurate translation, we reviewed each instrument line-by-line with local clinicians before each evaluation. To assess obstetric skills, we used the Gaumard Advanced Childbirth Simulator S500; for neonatal skills we used the Simulaids Sani-Baby CPR mannequin in Phase I and the Gaumard S320 Airway Trainer Newborn in Phase II. Finally, we reduced partograph evaluation to one case.

Study sites and sample characteristics

In Phase I, we selected participants purposively to represent the full range of facilities where women give birth and the cadre(s) primarily responsible for attending facility-based births in each country. We included at least one national-level tertiary care facility, at least two district-level hospitals, and a mix of rural and urban facilities. All participants completed all evaluation activities.

For Phase II, we evaluated health personnel in 20 hospitals, at least 1 in each of Nicaragua’s 17 health areas. This included 3 teaching hospitals, 2 maternal and child hospitals, and 15 district hospitals. We also evaluated personnel from 43 primary health centres, at least 1 from every health area except Carazo, Managua, Masaya and Rivas. Phase II selection criteria required all participating facilities to be qualified basic or comprehensive emergency obstetric care (BEOC or CEOC) facilities and, in the case of primary facilities, located in a health area participating in some obstetric care improvement initiative.19 In the South Atlantic Autonomous Region (RAAS), we enrolled all BEOC facilities. Elsewhere facilities were selected by convenience, principally related to physical accessibility. UNICEF, CARE, the Pan American Health Organization (PAHO), the Quality Assurance Project (QAP) and the Nicaraguan Ministry of Health jointly determined the sampling strategy.

Phase II participants were evaluated by function. Obstetricians and gynaecologists, obstetrics and gynaecology residents, and general practitioners in gynaecology participated only in testing related to labour, delivery and obstetric complications. In Nicaragua, these cadres were not tested on paediatric skills as they work in higher-level facilities and provide no paediatric care. For the same reason, Nicaraguan paediatricians, paediatric residents and paediatric general practitioners participated only in activities related to immediate newborn care and neonatal resuscitation. Hospital emergency personnel and non-specialized medical and nursing personnel from peripheral facilities completed both components (Table 2, available at: http://www.who.int/bulletin/en/volumes/85/10/06-038455/en/index.html).

The study was reviewed for compliance with QAP ethics guidelines, designed to weigh potential risks and benefits, and ensure participant confidentiality, informed consent and host-country ethics review. Ministry of health ethics committees or their equivalent approved the protocol in each country. We obtained verbal consent from all study participants. To protect confidentiality, we observed the following measures:

Results

Table 3 summarizes Phase I results, which are reported in detail elsewhere.27 The text describes Phase II results, presented by provider category in Table 4.

Knowledge test

Grouping all provider categories and all knowledge test versions (obstetric, paediatric, general), participants were correct on 62% of test questions (Table 4). By topic, results ranged from 80% correct for haemorrhage during pregnancy to 16% correct for infection prevention. By cadre, doctors were correct on 72% of questions overall, medical students 67%, professional nurses 57% and auxiliary nurses 51%. These differences were statistically significant (analysis of variance, ANOVA F = 265, P < 0.001), as were individual between-group differences (Bonferroni post hoc P ≤ 0.001 in all cases). Only five nurse-midwives (less than 0.5% of the sample) participated in Phase II as Nicaragua has stopped training this cadre.

Partograph case study

Doctors and medical students performed similarly on the partograph test, answering 65–70% of questions correctly. Professional and auxiliary nurses’ scores were lower: 42% and 33% respectively for written questions, and 19% and 10% for graphing. Differences were statistically significant (ANOVA F = 199, P < 0.001 for combined written and graphing scores). Doctors and medical students had statistically identical scores (Bonferroni post hoc P = 1). Differences between all other groups on all test components were significant (P ≤ 0.001).

Skills stations

AMTSL scores using the childbirth simulator ranged from 53% for doctors to 36% for auxiliary nurses. Doctors and medical students scored virtually identically (ANOVA Bonferroni post hoc P = 1). Nurses scored significantly lower than doctors (Bonferroni P < 0.001) and marginally lower than medical students (P = 0.09). Score differences between professional and auxiliary nurses were not significant (P = 0.49). Doctors correctly performed 53% of the steps for manual removal of the placenta; medical students 45% (t = 2.6, P = 0.009). Bimanual uterine compression scores were lower: 48% for doctors versus 37% for medical students (t = 3.2, P = 0.001). Since Nicaraguan norms prohibit nurses and auxiliary nurses from performing either procedure, they did not participate in these evaluations. Scores were higher and less varied for immediate newborn care, ranging from 76% for doctors to 64% for auxiliary nurses. There was no score difference between doctors and medical students or between professional and auxiliary nurses (Bonferroni P = 1, both cases), but professional and auxiliary nurses scored significantly lower than doctors and medical students (P < 0.05, all cases). Neonatal resuscitation scores followed the same pattern, ranging from 62% for doctors to 45% for auxiliary nurses.

With the exception of infection-prevention knowledge, Phase II scores correlate highly with professional profile: doctors scored highest, followed by medical students, professional nurses and auxiliary nurses. Professional nurses outscored medical students (but not doctors) on infection prevention, though no group exceeded 21%. The average score was 16%.

Discussion

While our results show significant variations in competency between different evaluation components and different cadres, the generally low scores are troubling. Different countries and cadres show different strengths and weaknesses, but several patterns emerge:Many participants scored poorly on basic questions related to infection prevention (hand-washing, proper handling of contaminated instruments, proper disposal of medical waste). In Nicaragua, intramuscular (IM) oxytocin use immediately after birth became a provisional standard in 2003, and AMTSL knowledge is high. In other countries, AMTSL was not routine at the time of the study. This may explain why many providers could not identify its components (IM oxytocin immediately after delivery of the foetus, controlled cord traction, uterine massage) and did not know that it should be practised universally.2830 Many providers did not recognize the diastolic blood pressure level indicative of severe pre-eclampsia or identify the use of magnesium sulfate and rapid termination of the pregnancy as the preferred management strategies for this condition. Ability to correctly use and interpret the partograph was low. Skills scores generally were lower than knowledge scores. For example: the mean AMTSL knowledge score in Nicaragua was 74%; the mean skills score was 46%. Similarly, the mean Phase I knowledge score on management of postpartum haemorrhage was 63%; the skills scores for manual removal of the placenta and bimanual uterine compression – basic evidence-based interventions to control postpartum haemorrhage – were 41% and 22% respectively. This suggests that knowledge of a procedure is no guarantee that it can be performed correctly.Though the pattern was less clear in Phase I, Phase II doctors and medical students generally scored higher than midwives, professional nurses and auxiliary nurses. Table 5 (available at: http://www.who.int/bulletin/en/volumes/85/10/06-038455/en/index.html) presents the pair-wise score differences for each assessment area by provider type. While predictable, this outcome may not be the most desirable for birthing women. In isolated rural settings where technology is limited and the nearest referral facility hours away, a woman is much more likely to be attended by a midwife or nurse than a doctor, so the basic life-saving skills of these cadres may be crucial when complications arise. Since Nicaragua no longer trains midwives and few remain in practice, a birthing woman there is most likely to be attended by a professional or auxiliary nurse in settings where no doctor is available. Even when doctors and technology are more accessible, nursing staff often attend most deliveries and perform routine tasks such as completing the partograph and monitoring for postpartum haemorrhage.

Limitations

Since the samples were non-random, we cannot be certain that they represent all professional health workers who attend births in the study countries. Further, we could not control for many potential confounders: differences in pre-service training, population health status, health system structure, national norms and practices, and inter-observer agreement between countries. The analysis here is bivariate. A multivariate analysis that controlled for health facility type, rural versus urban setting and work experience might yield different results. However, none of these factors seems likely to produce a downward bias in scores. If anything, the inclusion of tertiary care and teaching hospital personnel might inflate scores since these providers perform procedures more frequently and have access to better resources than rural clinicians.

The higher Nicaraguan scores may be attributable to several factors. Phase II instruments were shorter and simpler. In contrast to Phase I, we evaluated Phase II participants only on functions they actually perform. Finally, Nicaraguan participants were all assessed in Spanish, their native language. Beninese and Rwandan participants were assessed in French – a second language for some. But measurement error alone is unlikely to explain our results. In spite of these differences, there were remarkable consistencies between Phase I and Phase II results, suggesting serious cause for concern across a range of settings.

What is the significance of this gap between evidence-based standards and provider competence? One perspective holds that proposed international standards are simple and straightforward; anyone eligible to be designated as an SBA should be competent to implement them. It sends the wrong message if some providers are held to these norms but others are excused. This argument has particular logic in rural areas where highly trained providers often are unavailable. Another perspective holds that there is far from universal agreement on these best practices. Some national norms contradict some IMPAC guidelines or even prohibit their use. In urban areas where provider functions are more specialized, perhaps not everyone needs to be competent at everything. Thus, judging provider competency against standards to which their own health systems do not subscribe casts them in an unfairly negative light. These conflicting perspectives underscore the need to strengthen consensus on which practices maximize opportune identification and management of life-threatening complications and how best to implement them.

A central premise of maternal health programming holds that delivery by a skilled attendant and ready access to a BEOC or CEOC facility are fundamental to reducing maternal death.3,19,31,32 In their recent contribution to the Lancet series on maternal survival, Campbell and Graham reiterate that health centre intrapartum care is the most promising strategy for reducing maternal mortality in time to achieve MDG-5.33 Sufficient numbers of competent birth attendants are essential to this strategy. A health worker shortage is one important barrier, but inadequate competence among existing health workers may be equally important.8 Our findings appear to confirm this conclusion: a woman who delivers at a formal health facility assisted by a so-called “skilled” attendant cannot necessarily assume she will receive competent care.

A potential way forward

In light of our results, participating countries are now taking steps to close the competency gap. QAP, United Nations Population Fund (UNFPA) and the Ecuadorian Ministry of Health have teamed up to develop and implement an eight-session training programme focused on improving complication management. By the end of 2006, this group had trained 81 trainers and 74 clinicians in 5 provinces, plus 12 instructors at Quito’s midwifery school. Provincial trainers are scheduled to train a large number of provincial clinicians in 2007. Quito’s midwifery school is integrating this programme into its pre-service curriculum; other Ecuadorian midwifery and medical schools are considering its adoption. In Nicaragua, a similar effort is under way, coordinated by the Ministry of Health, QAP, UNICEF, CARE and PAHO. By December 2006, this initiative had trained 428 birth attendants in 14 of the country’s 17 health areas. In Niger, a comparable initiative which began in early 2006 had trained 239 providers in 28 of the country’s 37 reference facilities by the year’s end. Correct performance of AMTSL rose from 25% to 97% in participating facilities, affecting an average of 2369 births monthly. Projects in Bangladesh, Benin, Eritrea and Kenya have begun to integrate SBA competency evaluation into maternal health programming. QAP’s competency assessment instruments, along with a manual on how to conduct an assessment, can be downloaded from http://www.qaproject.org/strat/stratsafemotherhood.html/sbacomp.html.

Training, however, is only one part of the equation. No amount of training will lead to more hand-washing if health facilities lack soap and water. Health personnel cannot be expected to identify magnesium sulfate as the drug of choice for pre-eclampsia and eclampsia if it is unavailable – as it was in Benin during this study. Systemic problems require more comprehensive quality-improvement initiatives to address drug availability and distribution, equipment supply and maintenance, ineffective supervision, low morale and other problems that affect health services in many high maternal-mortality settings. However, while work continues to resolve systemic problems, much effort must be directed at raising basic competencies if health personnel are to attain the proficiency and fulfil the functions anticipated by WHO, the International Federation of Gynacology and Obstetrics (FIGO) and the International Confederation of Midwives (ICM) definition. Only then will SBAs be truly skilled and their deliveries become an accurate indicator of progress towards reducing maternal mortality. ■


References

Affiliations

  • Quality Assurance Project, University Research Co., LLC, 7200 Wisconsin Avenue, Suite 600, Bethesda, MD 20814, USA.
  • Quality Assurance Project, University Research Co., LLC, Managua, Nicaragua.
  • Department of Community Health and Psychiatry, University of the West Indies, Mona, Jamaica.
  • United Nations Children’s Fund (UNICEF), Managua, Nicaragua.
  • Quality Assurance Project, University Research Co., LLC, Quito, Ecuador.
  • Quality Assurance Project, University Research Co., LLC, Niamey, Niger.
Share