Bulletin of the World Health Organization

Should this event be notified to the World Health Organization? Reliability of the International Health Regulations notification assessment process

Thomas Haustein a, Helge Hollmeyer b, Max Hardiman b, Stephan Harbarth a & Didier Pittet a

a. Infection Control Programme, University of Geneva Hospitals and Faculty of Medicine, Rue Gabrielle-Perret-Gentil 4, 1211 Geneva 14, Switzerland.
b. International Health Regulations Coordination Department, World Health Organization, Geneva, Switzerland.

Correspondence to Thomas Haustein (e-mail: thomas.haustein@hcuge.ch).

(Submitted: 04 October 2010 – Revised version received: 14 January 2011 – Accepted: 31 January 2011 – Published online: 28 February 2011.)

Bulletin of the World Health Organization 2011;89:296-303. doi: 10.2471/BLT.10.083154


The International Health Regulations (2005) (IHR), which entered into force in June 2007, are a legally binding agreement between 194 States Parties, including states that are not Member States of the World Health Organization (WHO).1 Under the IHR, States Parties are required to notify WHO of “all events which may constitute a public health emergency of international concern”. Whether a given event is notifiable is determined using an algorithm: the decision instrument contained in Annex 2 of the IHR.1

The decision instrument defines an event as notifiable if it satisfies two or more of the following four criteria: (i) the event has a serious public health impact; (ii) the event is unusual or unexpected; (iii) there is a significant risk of international spread; (iv) there is a significant risk of international travel or trade restrictions.

In addition, all cases of smallpox, wild-type polio, novel-subtype human influenza virus infection and severe acute respiratory syndrome (SARS) are “intrinsically” notifiable, without the need to apply the four criteria. Annex 2 of the IHR is designed to heighten the sensitivity of the notification process and thus ensure a timely assessment of and response to critical public health events.2

The IHR stipulate that each State Party designate an office with which WHO can communicate at all times: the National IHR Focal Point (NFP). The NFPs receive guidance and training from WHO.26 They are responsible for contacting WHO about notifiable events and WHO recommends that they coordinate the notification assessment process, though they may not themselves be responsible for actually assessing the public health risk.3

Article 54.3 of the IHR,1 as well as World Health Assembly resolutions 58.37 and 61.2,8 mandate WHO to conduct studies to review and evaluate the functioning of Annex 2 of the IHR. A 2008 WHO technical consultation recommended that an assessment be carried out to evaluate agreement among NFPs when judging certain described events using Annex 2.9 The University of Geneva Hospitals’ Infection Control Programme in Switzerland was commissioned to carry out this assessment.


Survey design

The survey exploring NFPs’ use of Annex 2 of the IHR was based on several fictitious scenarios that described events devised to represent a wide range of public health risks with a varying likelihood of being considered notifiable. The number of scenarios included was limited to 10 and the vignettes (Box 1) were kept succinct to minimize the time and effort required from participants and to maximize the response rate. Box 2 provides summaries of the 10 scenarios.

Box 1. Full description of the public health event in one scenario used to evaluate the use of Annex 2 of the International Health Regulations (2005)

Several hospitals in province X have reported an abnormal increase of admissions due to pneumonia in the last two months.

A significant number of health-care workers are themselves affected by the respiratory disease and three health care workers aged between 20 and 50 years have died from respiratory failure. Influenza was suspected based on the clinical presentation, but tests for this and other common respiratory pathogens have been negative so far. Infection control measures, including the isolation of suspected cases, are being implemented.

Box 2. Short descriptions of the public health eventsa in the 10 scenarios used in the survey to evaluate implementation of Annex 2 of the International Health Regulations (2005)

Scenario 1. Pneumonia of unknown etiology linked to deaths among health-care workers

Scenario 2. Probable large-scale arsenic poisoning among refugees in a border region due to contaminated ground water

Scenario 3. International distribution of an ineffective HIV drug by a pharmaceutical company due to a production error

Scenario 4. Klebsiella pneumoniae resistant to all currently available antimicrobials detected for the first time in this country among a cluster of cases of neonatal sepsis

Scenario 5. Moderate rise in the incidence of dengue fever in an area dependent on international tourism

Scenario 6. Outbreak of salmonellosis linked to chicken meat in a product distributed within a single country

Scenario 7. Imported case of wild-type polio

Scenario 8. Fuel explosion with many casualties; only a single country affected

Scenario 9. Ochratoxin A detected in an imported wheat shipment at the port of entry

Scenario 10. Outbreak of foot-and-mouth disease among livestock

For each scenario, six items about the notifiability of the event had to be answered either “yes”, “no” or “don’t know” (Table 1). These items were: the personal opinion of the individuals who responded on behalf of the NFPs about whether the event was notifiable (item 1); the NFP’s judgement of whether each of the four criteria of the decision instrument in Annex 2 were satisfied (items 2–5); and the NFP’s judgement of whether an actual notification decision under the IHR should be made (item 6). The structure of the survey required participants to go through all four criteria for each scenario, including scenario 7, which described an intrinsically notifiable event (i.e. a case of wild-type polio).

Survey implementation

The survey was made available in the six languages used by the World Health Assembly via a secure web site (available at: https://www.surveymonkey.com/s.aspx?sm=GX8_2f9IUtDrEbMrNQpAFylg_3d_3d) and a printable version was provided on request. All 193 NFPs for which WHO had e-mail addresses were invited to participate by the University of Geneva Hospitals’ Infection Control Programme. Each State Party could submit only one completed survey. Access to the web survey was achieved by individualized links, which ensured that responses could be correctly attributed to a specific State Party. Participants were asked to base their notification assessment on only the information contained in the scenarios. Participation was voluntary and anonymous. The study was exempted from review by the WHO Research Ethics Review Committee.

Gold standard responses

To obtain an independent assessment of responses to survey items 2–6, seven internationally recognized public health experts recommended by the six WHO regions were invited to complete the survey. The names of the experts are listed in the acknowledgements section. Six of these experts had participated in the 2008 technical consultation on the implementation of Annex 2 of the IHR.9 A response was accepted as a “gold standard” response if it was supported by at least five members of the expert panel.


The degree of consensus among NFPs on a particular survey item was expressed as the proportion of participants who chose the most commonly selected response for that item. The degree of concordance between NFPs and the expert panel was expressed as the proportion of participating NFPs who selected the panel’s standard response for that item, where applicable. The degree of consensus or concordance was arbitrarily defined as being high when more than 70% of respondents agreed on a given response. In addition, the degree of concordance between an individual NFP and the expert panel was assessed using a concordance score in which one point was awarded for each notification assessment that matched the “gold standard”.

Differences in continuous variables were assessed using the Wilcoxon rank-sum or signed-rank test or Student’s t test, if values were normally distributed. Differences between paired proportions were tested for statistical significance using McNemar’s test. The association between concordance scores and demographic variables was explored using multivariate linear regression and Stata statistical software, release 11 (StataCorp LP, College Station, United States of America). Economic and population data were extracted from the WHO Statistical Information System (2007 data, available at: http://www.who.int/whosis). The association between gross national income and survey responses was assessed by dividing countries into four groups on the basis of gross national income quartiles.


Response rate

The survey of the NFPs’ use of Annex 2 of the IHR was completed by 142 States Parties between November 2009 and January 2010, giving a response rate of 74%. Three NFPs provided participants for both the expert panel and the survey, and one NFP participated in the expert panel only. Data on population and gross national income per capita were available for 134 of the States Parties who responded (94%) and 41 who did not (79%). There was no significant difference in population or gross national income per capita between those who did and did not respond.


Half of the individuals who responded on behalf of NFPs were medical doctors and one-third defined themselves as epidemiologists. Twenty-six (18%) reported that they had not applied Annex 2 in the last 12 months, 72 (51%) said they had applied it less than once a month, 20 (14%) once every month, and 23 (16%) at least once a week.

Consensus among NFPs

The results of the NFPs’ notification assessments are shown in Table 1. The overall median degree of consensus among NFPs on whether WHO should be notified (item 6) was 78% (interquartile range, IQR: 55–82). The degree of consensus was high (i.e. > 70%) for six of the 10 scenarios: 1, 2, 3, 7, 9 and 10. For each of the corresponding events, most participating NFPs (median: 80%; range: 76–91) felt that the notification requirement had been met. For the remaining four scenarios, the NFPs’ judgements were much more divided (median degree of consensus: 55%, range 54–60). Overall, NFPs judged a median of 7 events (IQR: 6–9) notifiable under the IHR.

When the four criteria of the decision instrument were assessed individually, the degree of consensus was found to be high (i.e. > 70% of NFPs provided the same response for a criterion) on 24 of 36 occasions (67%). Application of the four criteria was not required for scenario 7 (i.e. a case of wild-type polio). On 10 occasions (28%), the degree of consensus among NFPs was minimal (i.e. < 60% for the most frequently chosen response). Five of these occasions, in scenarios 1, 2, 3, 5 and 9, concerned the third criterion (i.e. a significant risk of international spread).

Concordance between NFPs and the expert panel

The expert panel considered that events in five scenarios met the requirements for notification under the IHR: scenarios 1, 2, 3, 7 and 9 (see Table 1). Most of the NFPs (median: 82%; range: 76–91) concurred with the panel on the notifiability of these events. The median sensitivity of the NFPs in identifying an event considered notifiable by the expert panel was 4 out of 5 (IQR: 4/5 to 5/5). For the four events not deemed notifiable by the expert panel (i.e. in scenarios 4, 5, 6 and 8; Table 1), the degree of concordance between NFPs and the panel was considerably lower (median: 51%; range: 42–60). Correspondingly, the median specificity of the NFPs in identifying an event not considered notifiable by the expert panel was 2 out of 4 events (IQR: 1/4 to 3/4), which was significantly lower than the sensitivity (P < 0.001). Finally, for scenario 10 (i.e. a foot-and-mouth disease outbreak), there was a high degree of consensus among NFPs (i.e. 78% for notification), but disagreement among expert panel members.

The expert panel gave a clear opinion on whether or not an individual assessment criterion was fulfilled on 27 of 36 occasions (75%): on 11 occasions, individual criteria were considered fulfilled by the expert panel, while on 16 occasions they were considered “not met”. On the remaining nine occasions (25%), there was no clear majority among panel members (Table 1). For the 11 occasions on which criteria were considered fulfilled by the expert panel, most NFPs concurred (median: 81%; range 59–97); on nine of those 11 occasions, more than 70% of NFPs agreed. For the 16 occasions on which criteria were not considered fulfilled by the expert panel, the median proportion of NFPs concordant with the panel was 74%, but responses were highly variable (range: 14–96%). Overall, more than 70% of NFPs agreed with the expert panel on their judgment using the decision instrument criteria in 18 of 36 occasions (50%).

Application of Annex 2 of the IHR

According to Annex 2 of the IHR, events that “meet any two of the four criteria (I–IV) in the algorithm” are notifiable to WHO. The seven members of the expert panel adhered to this rule in their notification assessments wherever it was applicable (i.e. in all scenarios except scenario 7).

In contrast, 64% of NFPs disregarded this rule at least once and this affected 15% of all individual NFP notification assessments. For all scenarios except scenario 7, two or more criteria were considered fulfilled by 71% of the NFPs, but only 66% deemed the events to be notifiable under the IHR, while only 62% considered that the events should be notified to WHO solely on the basis of the respondents’ personal judgment (P < 0.001 for all comparisons by McNemar’s test). If all events for which two or more criteria were considered fulfilled were regarded as notifiable, the specificity of the NFPs’ assessments would have been significantly lower (median: 1 out of 4; IQR: 1/4 to 2/4; P < 0.001), but the sensitivity would not have been significantly higher.

Concordance score

For the nine scenarios (i.e. scenarios 1–9) on which the expert panel expressed a clear opinion on notifiability under the IHR, the mean concordance score achieved by individual NFPs was 6.19 (standard deviation: 1.33). Univariate analysis showed a significant association between the concordance score and both gross national income (P = 0.04) and self-reported frequency of use of Annex 2 (P = 0.015). The corresponding levels of significance of these two variables found on multivariate analysis were P = 0.03 and P = 0.005, respectively. However, the two variables alone explained only 15% of the observed variation in concordance score, which suggests that other unmeasured factors may have had a more important effect.


The survey was completed by NFPs from nearly three-quarters of the States Parties to the IHR (2005). This suggests that there is considerable interest among NFPs in Annex 2 of the IHR and bears witness to the excellent support provided by WHO regional and country offices in implementing the survey. Although the degree of consensus between NFPs in assessing the 10 scenarios was variable, comparison with the “gold standard” responses provided by the expert panel showed that the sensitivity of the notification assessment process when used by NFPs to identify events considered notifiable by the expert panel was quite high. In contrast, the specificity was only moderate.

The scope of the IHR is intentionally broad and non-specific, which is a major strength that makes the IHR future-proof against new and unforeseeable threats.10 At the same time, the lack of specificity of the decision instrument in Annex 2 leaves considerable room for users’ perceptions, experience and knowledge to have an influence. Consequently, the disagreement observed between NFPs in assessing some events should not be surprising.

The level of agreement tended to be high when the implications of an event were “obvious” and could be judged using common sense. Examples of survey items that were not subject to dispute are the seriousness of drinking water being heavily contaminated by arsenic11 (scenario 2), the seriousness and unusualness of an infection that causes symptoms similar to SARS and results in deaths among health-care workers12 (scenario 1), and the absence of the risk of international spread or of international travel and trade restrictions due to accidental blast and burn injuries (scenario 8).

The level of agreement was lower when the applicability of the criteria was debatable in the absence of more detailed guidance. Thus, the NFPs disagreed on whether the moderate increase in dengue cases described in scenario 5 or the foot-and-mouth disease outbreak detailed in scenario 10 should be considered as posing a serious public health threat or not (i.e. the first criterion). The seriousness of the dengue scenario depends in part on how much the incidence increases.13 Consequently, a complete consensus on this criterion would only be achieved by defining epidemiological thresholds for seriousness. Foot-and-mouth disease was considered a notifiable event by many NFPs and even by some members of the expert panel, even though human cases of this animal disease are a rarity.14 Annex 2 may not be clear enough about how it should be applied to animal diseases. There was also a lack of consensus among NFPs and expert panel members on whether the local emergence of pan-resistant Klebsiella pneumoniae infection (scenario 4) was unusual or unexpected (i.e. the second criterion). The phenomenon of increasing antimicrobial resistance could, somewhat cynically, be called expected,15 but this particular scenario is unusual at present.16 Interestingly, the decision on whether the event in scenario 4 was notifiable hinged on the applicability of the second criterion, as most participants considered it as serious.

Agreement was also limited on the use of the third criterion: a significant risk of international spread. Scenario 1 (i.e. pneumonia with deaths among health-care workers) could represent the beginning of an outbreak of a viral infection that causes symptoms similar to SARS, with a clear potential for global spread.12 However, the examples given in the second part of Annex 2 put more weight on the circumstances of the event than on the known or expected properties of the disease agent (e.g. transmissibility) and suggest that only an imminent risk of spread is significant for the purposes of the IHR.1 The variability in the responses given when applying the fourth criterion in scenarios 1, 3 (i.e. HIV drug failure) and 9 (i.e. ochratoxin A in wheat) may in part be explained by the additional level of complexity introduced by the need to assess the risk of international travel or trade restrictions, which may be heavily influenced by political considerations.

These examples illustrate that users of Annex 2 have a considerable amount of freedom in judging events. In the present study, for example, the rule that the fulfilment of two criteria should lead to notification was not strictly followed. We found indications that the appropriate use of Annex 2 might increase the likelihood of certain events being notified to WHO: 66% of events were deemed notifiable under the IHR, while 62% were considered notifiable on the basis of personal judgment alone.

In accordance with the recommendations of the WHO technical consultation,9 the principal aim of this study was to assess the reliability of the responses given by public health professionals involved in the IHR notification process at the national level when applying Annex 2 of the IHR. In addition, we explored the validity of the assessments by reference to “gold standard” responses. Overall, the findings suggest that, in the hands of professionals, Annex 2 is a sensitive instrument for identifying events that are notifiable to WHO under the IHR. In contrast, a recent WHO audit revealed that, in practice, NFPs “are not yet a major source of early information to WHO on events”.6 Our findings indicate that this is unlikely to result from a malfunctioning of Annex 2 but instead may be due to barriers within countries. These barriers may include inadequate surveillance infrastructure or a poor flow of information within countries, perhaps resulting from limited resources or the administrative structure.17 In addition, political and economic considerations may also play a role.18,19 These considerations had already been recognized as important obstacles to reporting under the previous version of the IHR.19


First, to some extent, this survey is a theoretical exercise since the results tell us only what respondents think should be notified under the IHR, not what would be notified in practice; for example, notification may depend on how much relevant information is available to NFPs or on political considerations. The sensitivity of the notification assessment process observed in this survey could have been increased by a social desirability bias, as participants may have felt that it was more acceptable to make an unnecessary notification than not to notify an event. Second, the condensation of real-life events into short fictitious scenarios necessarily involves a high degree of simplification. For example, real events develop over time and have to be re-assessed at intervals. It would be difficult for a global survey to reflect this. Moreover, the notification assessment process in the present study may also have been simplified because fewer individuals may have been involved than under real-life conditions. Third, the number of scenarios had to be limited to 10 to make the survey manageable for participants. Although the aim was that the scenarios should cover a large variety of public health risks, the selection may not be representative of the risks encountered in the real world. Fourth, NFPs may not necessarily play the key role in the risk assessment of an event occurring within the territory of a given IHR State Party. Assessment may involve decision-makers based outside the NFP,2,9 such as other public health experts within the country or at WHO, individuals from governmental agencies not normally involved in health matters, or politicians. Moreover, depending on the nature of the event, fundamentally different stakeholders may be involved in the notification assessment process. Ideally, all these individuals and organizations should have been involved in responding to this survey, but the practical barriers to their systematic inclusion were deemed too high. Therefore, the survey was addressed to the NFPs, the only national bodies that could be clearly identified and contacted by WHO. Fifth, due to the anonymous nature of the survey we do not know whether the answers we received from NFPs represent the views of individual risk assessors within the NFP, of the entire NFP team or of a group of collaborators that included national experts outside the NFP. Sixth, although a response rate of 74% was achieved, our findings may have been affected by a non-response bias. Finally, the “gold standard” proposed in this study cannot claim to provide responses that are universally correct or applicable to all settings.


The low specificity of the notification assessment process in Annex 2 when used by NFPs to identify an event not considered notifiable by the expert panel is not a major concern as long as the volume of notifications is low. However, WHO might want to consider whether guidance on the use of Annex 2 of the IHR should be expanded. For example, additional scenarios could be described and more specific criteria for common public health events and a clearer definition of terms such as “a significant risk of international spread” could be given. In addition, the variability observed in the use of the Annex 2 decision instrument by NFPs in this study and the low self-reported frequency of its application in the past suggest that the proficiency of the NFPs in using Annex 2 could be further improved.


We would like to thank all participating NFPs, all staff at WHO headquarters and at regional and country offices who assisted us in the development and implementation of the study, and Angèle Gayet-Ageron for statistical advice. We are grateful to Preben Aavitsland, Abdel Hakim AlKohlani, Eduardo Hage Carmo, Ambrose Talisuna, Theresa Tam, Enrique A Tayag and Kumnuan Ungchusak for serving on the expert panel.


The study was funded by WHO.

Competing interests:

Helge Hollmeyer and Max Hardiman are employed by WHO at the International Health Regulations Coordination Department, WHO Headquarters, Geneva, Switzerland.