Bulletin of the World Health Organization

Sentinel versus population-based surveillance of pneumococcal conjugate vaccine effectiveness

Lee M Hampton a, Elizabeth R Zell b, Stephanie Schrag b & Adam L Cohen b

a. Epidemic Intelligence Service, Centers for Disease Control and Prevention, 1600 Clifton Road, MS A-24 Atlanta, Georgia, GA 30329, United States of America (USA).
b. Respiratory Disease Branch, Centers for Disease Control and Prevention, Atlanta, USA.

Correspondence to Lee M Hampton (e-mail: lhampton@cdc.gov).

(Submitted: 24 October 2011 – Revised version received: 26 February 2012 – Accepted: 07 March 2012 – Published online: 11 May 2012.)

Bulletin of the World Health Organization 2012;90:568-577. doi: 10.2471/BLT.11.098178

Introduction

Every year Streptococcus pneumoniae, or pneumococcus, causes an estimated 826 000 deaths worldwide in children under 5 years of age.1 In 2000, a seven-valent pneumococcal conjugate vaccine (PCV7) was introduced in the United States of America. The result was a dramatic, sustained reduction in invasive pneumococcal disease (IPD) in young children.2,3 Moreover, a very similar nine-valent pneumococcal conjugate vaccine (PCV9) that contains the seven S. pneumoniae serotypes in PCV7 (i.e. serotypes 4, 6B, 9V, 14, 18C, 19F and 23F) plus serotypes 1 and 5 has been shown to be efficacious in developing countries4,5 and the World Health Organization (WHO) has recommended that all countries introduce pneumococcal conjugate vaccines into their routine childhood immunization programmes.6 As with other new vaccines, measuring the effect of pneumococcal conjugate vaccines helps policy-makers determine whether the benefits of their introduction and sustained use outweigh their costs.7

Trends in the occurrence of a vaccine-preventable disease after a vaccine is introduced can be assessed by either population-based or sentinel surveillance. Population-based surveillance involves identifying all new cases of the disease under surveillance in a defined population. The data obtained can be used to calculate the disease incidence rate since the size of the population under surveillance is known. In contrast, sentinel surveillance involves monitoring a disease at a single facility or a small number of facilities. Generally, incidence rates cannot be derived from sentinel surveillance data since the population covered is rarely known and the data obtained may not be representative of the catchment area. However, sentinel surveillance usually requires fewer resources than population-based surveillance.8 The feasibility of sentinel surveillance has led WHO to recommend that countries adopt the approach for monitoring the effect of newly introduced vaccines, including pneumococcal conjugate vaccines.9 Sentinel surveillance systems have, nonetheless, several limitations and it is not clear whether even well-functioning systems can provide the data required for evaluating the effect of a new vaccine. However, studies have shown that sentinel systems involving hospitals in the United States can provide a reasonably accurate assessment of trends in antibiotic resistance10 and that sentinel surveillance in antenatal clinics in developing countries can be used to assess the prevalence of human immunodeficiency virus infection.11

In this study, we used data from the Active Bacterial Core surveillance system in the United States to determine whether a sentinel surveillance system involving either individual hospitals or groups of hospitals can provide estimates similar to those based on data from a population-based surveillance system when used to quantify the effect of introducing PCV7 on the occurrence of laboratory-confirmed IPD requiring hospitalization among children aged under 5 years.

Methods

The Active Bacterial Core surveillance system in the United States performs active, population-based surveillance for IPD, which is defined as being present when pneumococcus is isolated from normally sterile body fluid or tissue, such as blood, cerebrospinal fluid or pleural fluid, in an individual who is resident in a surveillance area on the date of culture.3,12 We obtained data from this surveillance system on all hospitalized IPD cases identified in children aged under 5 years between 1 January 1998 and 31 December 2006. We adopted 2006 as the endpoint of the study because that year recorded the lowest incidence of IPD to date in the age group.3 Throughout the study period, the Active Bacterial Core surveillance system monitored IPD in all eight counties in the state of Connecticut and in 49 counties overall in the states of California, Georgia, Maryland, Minnesota, New York, Oregon and Tennessee.12 The total population aged under 5 years that was under surveillance averaged 1 140 372 during 1998 and 1999 and was 1 261 188 in 2006 (Table 1). In the study, data were analysed for eight surveillance areas, which corresponded to areas monitored in the eight states. Demographic and clinical information on individual cases of IPD were obtained from medical records. A patient was regarded as having meningitis when S. pneumoniae was isolated from cerebrospinal fluid, irrespective of the clinical diagnosis, or when a clinical diagnosis of meningitis was accompanied by the isolation of S. pneumoniae from blood.

We calculated the observed, or crude, annual incidence of IPD requiring hospitalization in children aged under 5 years for each of the eight surveillance areas using the total number of cases identified in that age group by the Active Bacterial Core surveillance system in a given year and the area’s population of under 5-year-olds as reported in National Center for Health Statistics bridged-race population estimates, which are based on United States census data.13 We calculated the percentage change in the annual incidence of hospitalized IPD cases in children aged under 5 years using the formula: (RR − 1) × 100%, where RR was defined as the incidence in the most recent time period divided by the incidence at baseline. We determined 95% confidence intervals (CIs) for the percentage change using standard interval calculations in SAS version 9.2 (SAS Institute Inc., Cary, USA).

With sentinel surveillance, the effect of the vaccine was quantified by estimating the percentage change between baseline and 2006 in the number of hospitalized IPD cases treated at an individual hospital or group of hospitals participating in the Active Bacterial Core surveillance system. With population-based surveillance, the effect was quantified by estimating the percentage change between the two time periods in the incidence of hospitalized IPD cases in the surveillance area covered by the Active Bacterial Core surveillance system. Subsequently, the effect of the vaccine as assessed by sentinel surveillance was compared with its effect as assessed by population-based surveillance in the area in which sentinel surveillance was carried out. For example, the change in the IPD case count at an individual hospital in Connecticut was compared with the change in the incidence of hospitalized IPD cases for the entire state of Connecticut, as all eight counties in the state were covered by the Active Bacterial Core surveillance system. Since changes in referral patterns within individual surveillance areas could have had a confounding effect on the assessment of the change in hospitalized IPD cases in individual hospitals, all cases that were transferred from one hospital to another were excluded from calculations of the effect of the vaccine made using data from sentinel surveillance.

The effect of PCV7 on the number of children aged under 5 years who were hospitalized with IPD was estimated for three groups, according to the S. pneumoniae serotype causing the disease: (i) IPD due to any serotype, even when it was unknown or not recorded; (ii) IPD caused only by serotypes included in PCV7 (i.e. serotypes 4, 6B, 9V, 14, 18C, 19F and 23F); and (iii) IPD caused only by serotypes not included in PCV7. The period 1998 to 1999 was chosen as the baseline for comparisons and the years 2002, 2004 and 2006 were the end points.

Applying a method similar to that used by Montana et al.,11 we judged that sentinel surveillance was in agreement with population-based surveillance in assessing the effect of the vaccine on hospitalized IPD cases in children aged under 5 years when the magnitude of the change in the number of hospitalized IPD cases determined by sentinel surveillance fell within the 95% CI of the magnitude of the change in the incidence of hospitalized IPD cases in the corresponding surveillance area.

We also examined whether particular characteristics of the individual hospital or group of hospitals involved in sentinel surveillance were associated with a greater likelihood that the sentinel surveillance system’s assessment of the magnitude of the effect of PCV7 would agree with the population-based determination for the corresponding surveillance area. Every possible combination of hospitals in a given surveillance area in which each hospital had at least five hospitalized IPD cases in children aged under 5 years in 1998 and 1999 combined was examined in the analysis of groups of sentinel hospitals. Specifically, we determined whether the number of prevaccine cases in 1998 and 1999 or the proportion of cases in the surveillance area in 1998 and 1999 that was dealt with by a given sentinel system predicted which sentinel systems would produce an assessment of the effect of PCV7 that was in agreement with that obtained in the corresponding surveillance area. We used the Cochran–Armitage trend test to determine whether either of these two characteristics of the sentinel system or the increase in the amount of time since the introduction of PCV7 from 2 to 4 to 6 years was associated with a greater likelihood that a sentinel system would produce an assessment in agreement with that of its surveillance area.

Active Bacterial Core surveillance case reporting and isolate collection were considered to be public health surveillance activities that were exempt from institutional review by the Centers for Disease Control and Prevention in the United States. In addition, the surveillance protocol was evaluated at each participating surveillance site and either it was decided that the protocol was exempt from review or approval was obtained from the appropriate local institutional review board. Neither the Centers for Disease Control and Prevention nor institutional review boards at individual sites required informed consent.

Results

The number of hospitals in each surveillance area that reported at least one hospitalized IPD case in a child aged under 5 years in either 1998 or 1999 varied from 4 to 18 (Table 1) and the average number of cases in 1998 and 1999 combined for each of these hospitals ranged from 1 to 28 (median: 2). The proportion of all hospitalized IPD cases in a given surveillance area in 1998 and 1999 that was treated at a single hospital ranged from 0.01% to 58%. The average baseline population of children aged under 5 years in the surveillance areas during 1998 and 1999 ranged from 34 370 to 288 607 and the change in population from 1998 and 1999 to 2006 ranged from −19% to 34%.

We identified 1667 hospitalized IPD cases among children aged under 5 years between 1998 and 2006: 685 (41%) were cases of bacteraemia of unknown source, 558 (33%) involved invasive pneumonia, 218 (13%) involved meningitis and 206 (12%) involved infections at other locations. The annual number of hospitalized IPD cases caused by any S. pneumoniae serotype in all eight surveillance areas combined declined from an average of 360 in 1998 and 1999 to 148 in 2006, which corresponds to a decrease in incidence of 63% (95% CI: 55–69), from 31.4 to 11.7 cases per 100 000 children aged under 5 years (Table 2). The decrease in hospitalized IPD cases caused by any serotype in individual surveillance areas varied from 37 to 82%. Moreover, the decline in the incidence of hospitalized IPD cases caused by a serotype included in PCV7 in all surveillance areas combined was 99% (95% CI: 97–100), with the decline in individual surveillance areas varying from 96 to 100%. In contrast, the incidence of hospitalized IPD cases caused by a serotype not included in PCV7 increased by 93% (95% CI: 53–144) in all surveillance areas combined, with individual surveillance areas showing an increase ranging from 17 to 342%.

All invasive pneumococcal disease

Of the 98 individual hospitals that reported at least one hospitalized IPD case among children aged under 5 years in either 1998 or 1999, 93 (95%) recorded a decrease in the number of cases by 2006 (data not shown). In particular, all 26 hospitals with an annual average of more than three cases in 1998 and 1999 saw a decrease. However, the magnitude of the decrease observed at individual hospitals agreed with the magnitude of the decrease observed in the corresponding surveillance area for only 21% of hospitals (Table 3). Twelve of the 16 (75%) hospitals with more than five IPD cases a year during 1998 and 1999 and twelve of the 19 (63%) hospitals that treated more than 10% of their surveillance area’s cases in that period recorded a decrease that agreed in magnitude with that in the corresponding surveillance area. In contrast, only 11% of the hospitals with five or fewer IPD cases a year at baseline and 11% that treated 10% or less of their surveillance area’s cases recorded a decrease in agreement with that in the corresponding surveillance area (Table 3).

Fig. 1 illustrates the relationship between the number of children aged under 5 years hospitalized with IPD due to all serotypes in either 1998 or 1999 that was covered by a sentinel surveillance system in Tennessee and the likelihood that the magnitude of the decrease in cases recorded by the sentinel system by 2006 would agree with the magnitude of the decrease in the corresponding surveillance area, which here included five counties in Tennessee. Each sentinel system comprised either a single hospital with at least one hospitalized IPD case in either 1998 or 1999 or a group of hospitals, each of which had at least five cases during 1998 and 1999 combined. Overall, in all surveillance areas, more than 95% of sentinel systems that treated either more than 30 cases per year during 1998 and 1999 or more than 50% of cases in the corresponding surveillance area during that period recorded a decrease in all hospitalized IPD cases by 2006 that agreed with the decrease recorded in the corresponding surveillance area (Table 4).

Fig. 1. Change in number of children aged under 5 years hospitalized with invasive pneumococcal disease (IPD) from 1998–1999 to 2006a recorded by sentinel surveillance systems,b by number of cases in 1998 and 1999, Tennessee, United States of America

PCV7, seven-valent pneumococcal conjugate vaccine.

a PCV7 was introduced in the United States in 2000.

b Each sentinel system comprised either a single hospital with at least one hospitalized IPD case in either 1998 or 1999 or a group of hospitals, in which each hospital had at least five cases during 1998 and 1999 combined.

c A case was a child aged under 5 years who was hospitalized with IPD caused by any Streptococcus pneumoniae serotype.

d The change in the number of IPD cases recorded by the sentinel surveillance system was judged to be in agreement with the change observed in the corresponding surveillance area when the magnitude of the change in number recorded by the surveillance system fell within the 95% confidence interval of the magnitude of the change in the incidence of hospitalized IPD cases for the corresponding surveillance area.

e The whole surveillance area was the area covered by the Active Bacterial Core surveillance system in the United States in which all the sentinel surveillance systems were located; here the area included five counties in Tennessee.

Note: The vertical line indicates the change in the incidence of hospitalized IPD cases recorded in the whole surveillance area.

Consequently, it appears that the likelihood that the magnitude of the decrease in all hospitalized IPD cases recorded by a given sentinel surveillance system agreed with that recorded in the corresponding surveillance area increased as the number of cases at baseline increased, whether determined by the total number of cases treated during 1998 and 1999 or by the percentage of cases in the surveillance area treated during that period.

It was also found that the likelihood of agreement between a sentinel surveillance system and its surveillance area in determining the decrease in IPD cases increased as the length of time from the introduction of PCV7 increased from 2 to 4 to 6 years (P < 0.001 for all comparisons, Cochran–Armitage test for trend; data not shown).

Invasive pneumococcal disease by serotype

When the effect of PCV7 on only hospitalized IPD cases in children aged under 5 years that were caused by S. pneumoniae serotypes included in PCV7 was considered, it was found that 97% of all sentinel surveillance systems produced an estimate of the decrease in cases from 1998 and 1999 to 2006 that agreed with the decrease determined for the corresponding surveillance area: the proportion in agreement rose from 93% of sentinel systems with an annual average of 20 or fewer cases caused by all serotypes at baseline to 100% of sentinel systems with an annual average of more than 20 baseline cases (Table 4). Similarly, the proportion in agreement rose from 91% of sentinel systems that treated 30% or less of their surveillance area’s baseline cases caused by all serotypes to 100% of those that treated more than 30% of baseline cases.

In contrast, when the effect of PCV7 on only hospitalized IPD cases caused by serotypes that were not included in PCV7 was considered, only 70% of sentinel systems produced an estimate of the change in cases from 1998 and 1999 to 2006 that agreed with the change determined for the corresponding surveillance area. Moreover, only sentinel systems that treated an annual average of more than 40 IPD cases caused by all serotypes at baseline or more than 60% of their surveillance area’s baseline cases caused by all serotypes produced estimates that consistently agreed with those for the surveillance area (Table 4).

The likelihood of agreement between a sentinel surveillance system and its corresponding surveillance area in determining the change in hospitalized IPD cases, whether or not caused by serotypes included in PCV7, increased significantly as the length of time from the introduction of PCV7 increased from 2 to 4 to 6 years (P < 0.001 for all comparisons, Cochran–Armitage test for trend; data not shown).

Discussion

Our study of sentinel hospital surveillance of the effect of PCV7 in the United States indicates that sentinel surveillance can play a useful role in monitoring the effect of pneumococcal conjugate vaccines on IPD. We recommend that sentinel surveillance systems for monitoring the effect of these vaccines should: (i) establish the baseline burden of IPD before the introduction of the vaccine; (ii) ensure that the number of cases recorded at baseline is sufficient for monitoring future trends; (iii) determine which S. pneumoniae serotypes are responsible for individual cases; and (iv) measure the effect of the vaccine for at least 2 years after its introduction.

The number of IPD cases that should be recorded by a sentinel surveillance system at baseline depends on the goals of surveillance. In our study, almost all individual hospitals in the Active Bacterial Core surveillance areas correctly showed a decline in the number of children aged under 5 years who were hospitalized with IPD due to any serotype. However, only sentinel systems that included an annual average of more than 30 hospitalized IPD cases caused by any serotype during the 1998 to 1999 baseline period and those that included more than 50% of their surveillance area’s cases caused by any serotype during that period reliably measured the magnitude of the decline between baseline and 2006. That a sentinel system needed to have so many baseline cases to ensure that the magnitude of the effect of PCV7 was reliably assessed suggests that many potential sentinel systems may not be able to provide a dependable measure of a vaccine’s effect. For example, in its efforts to measure the burden of IPD around the world, the PneumoADIP programme14 found that, while many sentinel systems with large populations in their catchment areas identified more than 30 hospitalized, laboratory-confirmed cases a year in children aged under 5 years,7,15,16 others identified substantially fewer.7,1720 However, the alternative approach of establishing sentinel surveillance at hospitals that treat a large proportion of their areas’ IPD cases has the disadvantage that, without a population-based surveillance system, it is difficult to estimate the percentage of cases in the population seen at an individual hospital. Pooling several individual hospitals to create a sentinel system with a large enough total number of baseline cases may be more practical.

We found that sentinel systems performed best when measuring the effect of PCV7 on IPD caused by S. pneumoniae serotypes included in the vaccine: overall, 97% of sentinel systems produced results in agreement with those of their surveillance areas. This was largely because almost no patients were hospitalized with IPD caused by vaccine serotypes in any of the surveillance areas by 2006. The observation that it is important to know which S. pneumoniae serotypes are responsible for the IPD cases is consistent with studies that demonstrated the elimination of invasive Haemophilus influenzae type b in the Gambia21 and substantial reductions in the infection in South Africa,22 despite, respectively, the continued occurrence of and reported increases in invasive non-type-b H. influenzae. In practice, serotyping pneumococcal isolates is even more important than serotyping H. influenzae isolates for assessing the effect of a vaccine since, historically, H. influenzae type b has accounted for almost all invasive H. influenzae disease.23 In contrast, the serotypes in PCV7, PCV10 and PCV13 account for substantially smaller proportions of IPD.24 Moreover, serotyping pneumococcal isolates makes it possible to monitor any increase in IPD caused by serotypes not included in the vaccine, which is a potentially important problem.3,25 In our study, however, the sentinel systems included in the Active Bacterial Core surveillance system were generally less reliable in assessing the magnitude of the change in disease due to non-PCV7 serotypes than the magnitude of the change due to PCV7 serotypes. Although serotyping pneumococcal isolates is difficult in many settings with limited laboratory facilities, the use of techniques based on the polymerase chain reaction may provide a practical solution.2628

Population-based surveillance systems have shown a decline in IPD within 1 to 2 years of vaccine introduction2,29 and sentinel systems may do the same. However, in our study, a sentinel system’s reliability in measuring the vaccine’s effect on IPD, whether or not caused by serotypes included in PCV7, increased with the passage of time.

Although careful selection of sentinel sites is important, it cannot compensate for fundamental problems in the operation of the surveillance system. A high-quality laboratory is essential for consistently identifying possible cases of the disease of interest, for carrying out accurate confirmatory tests and for recording and communicating case information. Potential problems with staff, supplies, support or funding30 mean that surveillance systems monitoring the effect of a vaccine, especially laboratory-based systems performing new tests, should be operating reliably before the target vaccine is introduced. In addition, staff must be alert to the possibility that any change in the occurrence of an invasive bacterial disease may result from a change in the size of the population under surveillance or a change in the operating or referral pattern of the local health-care area rather than a change in the amount of disease in the population.

Even when a sentinel system provides high-quality data on a vaccine’s effect, the findings should not be overgeneralized. For example, the effect of a vaccine measured by a single hospital that treats more than 30 IPD cases per year should not automatically be used to deduce the vaccine’s effect across a country with tens of millions of people.

This study has several limitations that may limit the generalizability of its findings. Other countries may differ from the United States in care-seeking behaviour among the population, clinical practice, laboratory capacity and pneumococcal epidemiology, all of which would affect the ability of sentinel surveillance to reflect accurately population-based surveillance. In addition, staff at the Active Bacterial Core surveillance system actively seeks out cases, so our findings may not apply to systems that collect case reports passively. Also, since the Active Bacterial Core surveillance system’s data are based on laboratory testing, they may not be useful for the surveillance of clinical syndromes, such as meningitis. Finally, although we expect our findings will apply to other pneumococcal conjugate vaccines, our study addressed only the effect of PCV7 and not that of higher-valence pneumococcal conjugate vaccines such as PCV10 or PCV13.

In addition to monitoring the effect of a vaccine on IPD, sentinel surveillance can also help identify the serotypes causing disease and can record the clinical and demographic characteristics of individual cases. However, the recent global push towards using pneumococcal conjugate vaccines suggests that measuring the effect of a vaccine is a key reason for carrying out surveillance. Although laboratory-based sentinel systems can be useful for monitoring the direction of the effect of a vaccine on the incidence of invasive bacterial disease, if they are to measure reliably the magnitude of the effect, they will have to either include a large number of cases at baseline before the vaccine’s introduction or have the ability to identify the serotype and not just the species of the invasive bacteria. While it is more expensive to create sentinel systems that involve several hospitals or that have a laboratory capable of identifying specific serotypes, it is worthwhile finding the additional resources.


Acknowledgements

We thank the following individuals for their assistance: Pam Daily Kirley and Arthur Reingold at the California Emerging Infections Program; Zack Fraser, James L Hadler and Susan Petit at the Connecticut Emerging Infections Program; Kathryn E Arnold, Wendy Baughman, Monica Farley, Amy Holst and Stephanie Thomas at the Georgia Emerging Infections Program; Lee H Harrison, Rosemary A Hollick and Kim Holmes at the Maryland Emerging Infections Program; Richard Danila, Billie Anne Juni, Lindsey Lesher, Catherine Lexau, Ruth Lynfield and Lori Triden at the Minnesota Emerging Infections Program; Nancy M Bennett, Geetha Nattanmai, Glenda Smith, Suzanne Solghan and Nancy Spina at the New York Emerging Infections Program; Karen Stefonek and Ann Thomas at the Oregon Emerging Infections Program; Brenda G Barnes and William Schaffner at the Tennessee Emerging Infections Program; and Felicita David, Melissa Lewis, Matt Moore, Tamara Pilishvili, Karrie Ann Toews, Chris Van Beneden, and Carolyn Wright at the Centers for Disease Control and Prevention.

Funding:

This work was supported by the Emerging Infections Program and the Office of Antimicrobial Resistance of the Centers for Disease Control and Prevention.

Competing interests:

None declared.

References

Share