Bulletin of the World Health Organization

External quality assessment of national public health laboratories in Africa, 2002–2009

John Frean a, Olga Perovic a, Vivian Fensham a, Kerrigan McCarthy a, Anne von Gottberg a, Linda de Gouveia a, Bhavani Poonsamy a, Leigh Dini a, Jenny Rossouw a, Karen Keddy a, Wondimagegnehu Alemu b, Ali Yahaya b, Antoine Pierson c, Virginie Dolmazon c, Sébastien Cognat c & Jean Bosco Ndihokubwayo b

a. National Institute for Communicable Diseases, National Health Laboratory Service, P/Bag X4, Sandringham, Johannesburg, 2131, South Africa.
b. Regional Office for Africa, World Health Organization, Brazzaville, Congo.
c. Global Capacity, Alert and Response, World Health Organization, Lyon, France.

Correspondence to John Frean (e-mail: johnf@nicd.ac.za).

(Submitted: 10 June 2011 – Revised version received: 28 October 2011 – Accepted: 09 November 2011 – Published online: 20 January 2012.)

Bulletin of the World Health Organization 2012;90:191-199A. doi: 10.2471/BLT.11.091876

Introduction

In Africa, communicable diseases constitute a very important public health problem and outbreaks pose serious threats to health.1 These diseases exert enormous influence on the economy and society and should be targeted through aggressive disease prevention, control and surveillance activities. However, successful performance of these activities requires reliable epidemiologic data and generating such data is one of the roles of national public health laboratories.2 This awareness led the Regional Committee for Africa of the World Health Organization (WHO) to recommend strengthening the ability of national public health laboratories to accurately, reliably and promptly confirm epidemics in an effort to improve the public health response and enhance national and global health security. This was the intention behind its adoption of the strategy for integrated disease surveillance and response in 19983 and of Resolution AFR/RC58/R2 in 2008.35 The integrated disease surveillance and response technical guidelines recommend the use of standard laboratory diagnostic methods for priority diseases in the WHO African Region.6

In many African countries, reliable confirmation of suspected infectious diseases is hampered by a lack of standardized diagnostic methods and by a shortage of funds, staff and laboratory supplies for national public health laboratories, despite the critical role played by these laboratories as part of a functional infrastructure for disease surveillance.7 Development and maintenance of high-quality national public health laboratory services require financial and managerial commitment to provide qualified staff, training, equipment, consumables, reagents and physical facilities. Periodic quality assessment of performance is also essential because it can help ensure the reliability of findings and is an important component of laboratory accreditation, towards which national public health laboratories should, ideally, strive.

In July 2002, WHO launched an external quality assessment programme (EQAP) to test the proficiency of microbiological testing for epidemic-prone diseases by laboratories in the African Region.8 The EQAP was extended to laboratories in three African Member States in the Region of the Eastern Mediterranean during 2005. The National Institute for Communicable Diseases (NICD), a division of the National Health Laboratory Service of South Africa, provided technical coordination following an agreement with the co-funders, the WHO Regional Office for Africa and the WHO office in Lyon, France.810 Accordingly, the NICD undertook to provide specimens for laboratory identification of selected agents of bacterial enteric diseases, bacterial meningitis and plague; to advise WHO about the needs of participating laboratories, to correct deficiencies and maintain proficiency; and to further extend the EQAP to include general bacteriological analyses and tuberculosis and malaria microscopy.11 This article reviews findings from microbiology EQAP surveys conducted by WHO and NICD in Africa during 2002–2009.

Methods

Participating laboratories

The WHO Regional Office for Africa invited national public health laboratories (nominated by their ministries of health) and/or the main hospital or research laboratories functioning as such, as well as laboratories in the Paediatric Bacterial Meningitis Surveillance Network,12 to participate in the bacterial enteric diseases and bacterial meningitis components of the EQAP (Table 1, available at: http://www.who.int/bulletin/volumes/90/3/11-091876). Some laboratories were both national public health laboratories and in the Paediatric Bacterial Meningitis Surveillance Network.12 Certain laboratories were also recruited to participate in the plague, tuberculosis and malaria components of the EQAP because of their expertise in these areas.

Coordination and technical guidance

The NICD Technical Implementation Group, together with representatives from WHO’s African Regional Office and the WHO’s Lyon Office, coordinated the EQAP. The EQAP Regional Advisory Group, consisting of the Technical Implementation Group, WHO partners, additional technical consultants and invited experts, met annually to evaluate progress and plan future activities. Technical input provided by the Technical Implementation Group about EQAP specimens included in-house production, quality control and validation of survey materials and evaluation of findings and reports. Eleven international expert laboratories (including some WHO Collaborating Centres) served as external technical advisers or referees.8 These laboratories processed EQAP specimens blindly and provided comments about their quality. Additional technical details of the programme and information about its management are comprehensively described elsewhere.8

Survey contents and distribution

Initial surveys evaluated laboratory proficiency in bacterial enteric diseases, bacterial meningitis and plague components; tuberculosis and malaria components were added in 2005. The range of organisms sent across surveys is shown in Table 2. Current regulations strictly limit or prohibit transfer of Shigella dysenteriae type 1 and Yersinia pestis cultures,13 precluding their use in the EQAP.

Surveys containing bacterial enteric diseases, bacterial meningitis and plague components generally included two challenge specimens per disease component. The NICD used techniques developed by the Clinical Microbiology Proficiency Testing programme at the University of British Columbia8,14 in Vancouver, Canada, to produce clinically relevant simulated preparations (e.g. cerebrospinal fluid smears for Gram staining) and simulated biological fluids (e.g. cerebrospinal fluid, stool and pus) inoculated with appropriate bacteria, including some general pathogens and potential contaminants. Trans-isolate medium,15 which is capable of sustaining small numbers of the common bacterial meningitis pathogens in cerebrospinal fluid during transport, was used in the EQAP and generally performed well. When organism viability in simulated specimens proved to be limited (as in the case of Vibrio cholerae), lyophilized cultures were substituted. Specimens for the plague component included culturable bacteria other than Y. pestis, blood smears containing Y. pestis (inactivated by fixing) for staining and detection of typical bipolar-stained bacilli, simulated plague bubo aspirates for detection of fraction 1 (F1) antigen by means of dipstick-format rapid tests (Institut Pasteur, Antananarivo, Madagascar), and serum for antibody assays. F1 assays were distributed to participating laboratories via the EQAP.

The tuberculosis component included 8 challenge specimens – 4 Ziehl-Neelsen-stained slides and 4 unstained slides – for microscopy. Slides contained smears prepared from pooled clinical sputum specimens in accordance with protocols of the Association of Public Health Laboratories, headquartered in the United States of America.16 Participants were required to stain unstained slides for detection and quantitation of mycobacteria. Laboratories reported on the presence of acid-fast bacilli using the International Union for Tuberculosis and Lung Disease smear-grading system.8,17

For the malaria component, 10 challenges, each consisting of Giemsa-stained thin and thick blood films for identification of parasite species and quantitation of parasite density, respectively, were included. Films were prepared using standard procedures developed for the EQAP.8 Identical challenges were included within and across surveys to test for intra- and interrater reliability.

Survey materials were fully characterized by the NICD before shipment (for bacteria, this included biochemical identification, serotyping or serogrouping and antimicrobial susceptibility profiles).14 Bacterial strains included those routinely supplied by the NICD as controls to South Africa’s national laboratory system, with the intention that EQAP participants would retain them for that purpose. Each survey included written information that described clinically relevant case scenarios for challenges, as well as forms for participants to record activities performed before and after testing. Participants were prompted for their decisions about testing algorithms, interpretation of test results and reporting of findings. A training exercise was sent with the second survey in 2004. It explained the protocols for antimicrobial susceptibility testing of pneumococci of the Clinical Laboratory Standards Institute,18 the Société Française de Microbiologie19 and the British Society for Antimicrobial Agents and Chemotherapy.20

All EQAP documents were prepared in English and French and most were also prepared in Portuguese. They included survey information; report forms; individual results and graded assessment of performance; a commentary on overall performance, including statistical evaluation; assessment criteria, and technical suggestions for improvements.

Three surveys were sent per year. All were packaged according to International Air Transport Association Packing Instruction 6508 and were shipped from Johannesburg, South Africa, to participants and referees simultaneously by express air courier.13 Participating laboratories were requested to return results within 30 days of survey dispatch but some leeway was allowed because of unpredictable delivery times.

Evaluation of responses

Areas of the survey involving critical decision points and tests were assigned grades that accorded with predetermined schemes. The Technical Implementation Group evaluated referee laboratory responses for consensus results before determining evaluation criteria within each grading area.

The grading scheme for the bacterial enteric diseases, bacterial meningitis and plague components was adopted from the Clinical Microbiology Proficiency Testing programme.8,14 This programme uses a discontinuous scale that assesses the response in terms of its clinical influence on a hypothetical patient. The best answer receives a score of 4, answers that are incorrect but have no or little clinical impact receive a 3, answers that are incorrect and could lead to a minor diagnosis or treatment error receive a 1 and answers that are incorrect and could lead to a major diagnosis or treatment error receive a 0. Responses were graded in several technical areas, such as microscopy, culture and identification (including antigen test results), serotyping, choice of antibiotics for antimicrobial susceptibility testing and reporting of susceptibility. Grading areas depended on the particular challenge. For example, microscopy performance (i.e. Gram staining and examination of smears) was usually assessed for the bacterial meningitis and plague components but not for the bacterial enteric diseases component, whereas rapid antigen test performance was only graded for the bacterial meningitis and plague components.

Assessment of microscopy in the tuberculosis component was adapted from the Association of Public Health Laboratories guidelines for external quality assessment.16 In this grading scheme, correct responses and minor quantitation errors receive 10 points, responses with minor misclassification errors receive 5 points and responses with major misclassification errors (i.e. high false positives or high false negatives) receive 0 points.

For the malaria component, species identification was graded using the scheme developed by the Clinical Microbiology Proficiency Testing programme, as described above. Quantitation of malaria parasite density was assessed with regard to consensus counts, calculated as the median of combined counts from participants, referees and NICD. Counts within the target range, defined as 0.5 to 1.5 times the consensus count, were considered acceptable.21

Data analysis

Responses and scores were entered and stored on a customised Access database (Microsoft, Redmond, USA) and analysed on a Microsoft Excel spreadsheet. For bacterial meningitis, bacterial enteric diseases, plague and malaria microscopy, scores of 3 or 4 were acceptable; scores of 1 or 0 were unacceptable. For tuberculosis microscopy, scores of 5 or 10 were acceptable and a score of 0 was unacceptable. Acceptable quantitation of malaria parasite density was defined as a count within the target range. For all periods (i.e. one year or an entire review period), the proportion of acceptable responses in a component and grading area challenge was expressed as a percentage of the number of eligible responses received. Assessment of overall or cumulative performance in a component was performed by grouping all grading area responses together and calculating the proportion with acceptable scores.

Results

As shown in Table 1, 39 laboratories in 30 of 46 Member States in the African Region were enrolled in 2002. In 2005, laboratories from three Member States in the Eastern Mediterranean Region were added; by the end of the year, 73 laboratories had been recruited, including those with the ability to identify agents of tuberculosis and malaria. By the end of 2009, 48 African Member States were involved, with 76 laboratories enrolled in the bacterial enteric diseases and bacterial meningitis components, 70 in the malaria component, 68 in the tuberculosis component and 17 in the plague component. All laboratories evaluated challenge specimens associated with 2 to 4 disease components per survey. The mean response time across all surveys, which reflects transport and bureaucratic delays and specimen processing time, was 25.9 days (standard deviation: 6.2 days; range: 9–54 days). As is generally the case for proficiency testing schemes, the timeline for returning results was not intended to reflect standards for routine clinical sample turnaround times, given the logistical constraints involved. Overall performance in each disease component, combining all grading areas, is summarized in Fig. 1.

Fig. 1. Performance in five disease components among African laboratories that participated in an external quality assessment programme,a 2002–2009
Fig. 1. Performance in five disease components among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2002–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Bacterial enteric diseases

Between 63% and 91% of surveyed laboratories responded. Fig. 2 summarizes the percentages of acceptable responses in four grading areas. Cumulative acceptable performance was 65%. Although 64% of responses involving decisions about antibiotic selection for antimicrobial susceptibility testing were acceptable, 43% involving test reporting were unacceptable. Only a minority of laboratories had the capacity to serotype Salmonella and Shigella species.

Fig. 2. Performance in enteric pathogens component among African laboratories that participated in an external quality assessment programme,a 2002–2009
Fig. 2. Performance in enteric pathogens component among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2002–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Bacterial meningitis

A total of 63–93% of laboratories responded per survey. Cumulative acceptable performance was 52%; performance in five grading areas is presented in Fig. 3. Microscopy findings were reasonable but erratic (62–88% of responses were acceptable). Problems with antimicrobial susceptibility testing of pneumococci were identified early in the programme and subsequent surveys included identified strains of pneumococci as susceptibility testing challenges only. Antibiotic choice for antimicrobial susceptibility testing was poor (54% of responses were acceptable) and reporting of corresponding findings was very poor (only 25% of responses were acceptable). Few laboratories performed minimum inhibitory concentration testing.

Fig. 3. Performance in meningitis pathogens component among African laboratories that participated in an external quality assessment programme,a 2002–2009
Fig. 3. Performance in meningitis pathogens component among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2002–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Plague

Participation per survey ranged from 64% to 90% of laboratories. Cumulative acceptable performance was 66%. Performance in three grading areas is specified in Fig. 4. Microscopy findings for bipolar Gram-stained bacteria were good (77–100% were acceptable), as were the findings for F1 antigen dipstick tests (76% were acceptable). However, use of biochemical tests, including API® strips (Biomérieux, Durham, USA), for identification of Gram-negative organisms generally yielded unacceptable scores and Gram staining of routinely isolated bacteria was erratic.

Fig. 4. Performance in plague component among African laboratories that participated in an external quality assessment programme,a 2005–2009
Fig. 4. Performance in plague component among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2005–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Tuberculosis

Responses were returned by 72–90% of laboratories per survey. The proportion of acceptable responses is shown in Fig. 5. Laboratories generally performed well, with 87% of responses considered acceptable. Although evaluation of prestained slides was good, participant-stained slides resulted in a greater proportion of false-negative results.

Fig. 5. Performance in tuberculosis microscopy component among African laboratories that participated in an external quality assessment programme,a 2005–2009
Fig. 5. Performance in tuberculosis microscopy component among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2005–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Malaria

Between 63% and 83% of surveyed laboratories responded. Results in two grading areas are shown in Fig. 6. Acceptable scores for species identification were given to 82% of responses. Whereas proportions of false positive and false-negative results were low, correct identification of blood parasites other than Plasmodium falciparum (the predominant agent of malaria in Africa) was limited (58% of responses were acceptable). Quantitation of malaria parasites was poor, with only 51% of responses considered acceptable.

Fig. 6. Performance in malaria microscopy component among African laboratories that participated in an external quality assessment programme,a 2005–2009
Fig. 6. Performance in malaria microscopy component among African laboratories that participated in an external quality assessment programme,<sup>a</sup> 2005–2009
a Coordinated by the World Health Organization and the National Institute for Communicable Diseases, a division of the National Health Laboratory Service of South Africa.

Discussion

Several inadequacies have been identified in medical laboratory services in Africa.22 Deficiencies in areas such as consumables, basic equipment, skilled personnel, training programmes, logistical support, national standards and quality assessment have been emphasized. While our programme was limited to the assessment of national public health laboratory proficiency in relation to a preselected range of pathogens of public health importance, we found that many of these problems were present in participating laboratories. Our findings suggest that shortage or lack of appropriate reagents was immediately responsible for poor performance in some programme grading areas. For example, a lack of agglutinating antisera was frequently a primary reason for failure to fully characterize bacterial isolates. Failure to correctly perform antimicrobial susceptibility testing, including inappropriate choice of antibiotics for testing, absence of quality control for both media and antibiotic disks and failure to adhere to internationally recognized guidelines,1820 were also among shortcomings demonstrated by some laboratories. More generally, problems with quality control of media and reagents, including lack of appropriate control organisms, and insufficient training in organism identification and quantitation were evident.

Published external quality assessment activities in Africa have generally been limited in duration, geographic footprint and scope of assessment. Multicountry programmes for quality assessment of automated quantitation of CD4+ T-cells23 and for malaria microscopy in a clinical vaccine trial24 are examples of more extensive efforts. The EQAP is unique in its geographic range (all sub-Saharan and some northern African countries are included), scope of assessed activities and sustainability, demonstrated by 10 years of uninterrupted activity as of 2011. As a direct result of the EQAP, national diagnostic capabilities for the five targeted epidemic-prone diseases include regular proficiency testing and access to training resources in the form of reference material (e.g. characterized bacterial control strains, malaria and tuberculosis control slides and copies of international laboratory standards and guidelines), commentaries, technical advice, bench aids and occasionally scarce reagents, such as rapid plague F1 assays. The multilateral scheme has provided a better understanding of the performance of participating laboratories, identifying their strengths, limitations and continuing challenges. The NICD’s capacity to produce, perform quality control of and distribute samples for the EQAP has been robust. This programme has allowed WHO to identify the capacities and resources needed by countries for sustainable improvement in the competency of participating laboratories.

This international EQAP provides a unique opportunity for African national public health laboratories to continuously assess and improve their performance in microbiological analysis. It is a powerful channel for educating laboratory personnel and for standardizing procedures. Objective data yielded by the programme illustrate the strengths and weaknesses of participating laboratories and constitute an evidence-based tool for guiding and improving laboratory-strengthening activities for WHO in the area. Given the considerable challenges of securing resources to procure and maintain stock of the much needed reagents, transport media and other necessary laboratory supplies, EQAP findings should be used to advocate for mobilization of additional resources at the local level to improve laboratory capacity. This should include budgeting for supplies and communication activities to strengthen the linkage between national public health laboratories and other public health services. The results should also be used to advocate for additional investment in all aspects of health laboratory services. The programme has served as an example for national and regional external quality assessment; several countries in Africa and in WHO’s Region of the Eastern Mediterranean are planning, or have implemented, national or regional external quality assessment schemes that are based on this model.8

Notwithstanding its value as a tool for advocacy, external quality assessment alone does not invariably improve laboratory performance. The lack of a general upward trend in EQAP performance over time across all disease components and grading areas (Fig. 1) corroborates this. Identified deficiencies need to be actively corrected for there to be sustainable improvement. Although several EQAP-associated training visits were provided by NICD in partnership with WHO’s Regional Office for Africa, implementation of corrective actions is primarily the responsibility of laboratory personnel and management and is verified through an accreditation process. Only 340 diagnostic laboratories in Africa are accredited, with 312 (92%) in South Africa.25 The commitment to strengthening African NATIONAL PUBLIC HEALTH LABORATORIEs25 led directly to the development of the WHO-African Region stepwise approach to accreditation.25 External quality assessment is among the 12 areas of laboratory quality evaluated by this approach, making participation in a proficiency testing programme such as the EQAP an indispensable part of its implementation.


Acknowledgements

We thank L Arntzen, C Bopp, R van Deventer, P Hance, H Haritos, R Landsberg, T Mahlanga, L Mayer, P Nicolas, L Rahalison, A Sooka and the staff at referee laboratories, for their highly valued contributions to the programme.

Funding:

This study was funded by the United States Agency for International Development; the Global Alliance for Vaccine Initiative, through the WHO Paediatric Bacterial Meningitis Surveillance Programme; and the Government of the Netherlands.

Competing interests:

None declared.

References

Share