Validation of the shake test for detecting freeze damage to adsorbed vaccines
Ümit Kartoglu a, Nejat Kenan Özgüler b, Lara J Wolfson c & Wiesław Kurzatkowski d
a. Department of Immunization, Vaccines and Biologicals, World Health Organization, 20 Avenue Appia, 27 Geneva 1211 Switzerland.
b. Emergency Department, Karaman Hayat Medical Centre, Karaman, Turkey.
c. Global Influenza Programme, World Health Organization, Geneva, Switzerland.
d. National Institute of Hygiene, Warsaw, Poland.
Correspondence to Ümit Kartoglu (e-mail: firstname.lastname@example.org).
(Submitted: 14 July 2009 – Revised version received: 23 November 2009 – Accepted: 01 December 2009 – Published online: 09 February 2010.)
Bulletin of the World Health Organization 2010;88:624-631. doi: 10.2471/BLT.08.056879
Good temperature control during the storage and transport of vaccines is critical to ensure their potency and safety. Liquid formulations of aluminium-based vaccines against diphtheria, pertussis, tetanus, hepatitis B and Haemophilus influenzae type b, alone or in combination (adsorbed vaccines), should not be frozen.1 However, practices that expose vaccines to sub-zero temperatures are widespread in both developed and developing countries at all health system levels.2–11 The most recent systematic literature review of vaccine freezing practices showed that accidental freezing occurs across all parts of the cold chain.12 Between 14% and 35% of refrigerators or transport shipments were found to have exposed vaccines to freezing temperatures, while in studies that examined all segments of the distribution chain, between 75% and 100% of the vaccine shipments were exposed. More rigorous study designs were associated with higher levels of exposure to freezing.
When a vaccine is damaged by freezing, the potency lost can never be restored – the damage is permanent.13–16 Freeze-damaged vaccines have lower immunogenicity and are more likely to cause local reactions, such as sterile abscesses.1,17
The shake test is designed to determine whether adsorbed vaccines have been affected by freezing. After freezing, the lattice (made up of bonds between the adsorbent and the antigen) in a vaccine is broken. Separated adsorbent tends to form larger, heavier granules that gradually settle at the bottom of the vial when this is shaken. When freezing and thawing cycles are repeated, the granules appear to increase in size and weight. In a typical demonstration of the shake test, two identical vials of a vaccine (i.e. from the same batch and the same manufacturer) that is suspected of having been exposed to freezing temperatures are selected; one of the two vials is purposely frozen and then thawed as the negative control, while the second vial serves as the vial to be “tested” against this negative control. The two vials are held together in one hand and shaken; they are then placed side by side on a flat surface. Provided the test vial has not been frozen, sedimentation is slower in the test vial than in the control vial that has been frozen and thawed.1,18 If the test vial has been frozen, the test and control vials will have similar sedimentation rates. Fig. 1 illustrates how the appearance of frozen (i.e. frozen and thawed, and therefore freeze-damaged) and non-frozen test vials compares to that of their frozen control vial, 1 minute and 28 seconds after shaking.
Fig. 1. Visual difference in sedimentation rates after shake test for detecting freeze damage to adsorbed vaccines
During the late 1980s, a shake test protocol was developed based on empiric observations in the field. However, its description was available nowhere except on a poster in the archives of the World Health Organization (WHO). Although the shake test is widely practiced in the field by staff at all levels of the health system, it has never been validated as a reference test by comparison to a “gold standard”. The test is also used as a decision tool in accepting international shipments of freeze-sensitive vaccines if a temperature monitoring device indicates that freezing has occurred (i.e. a “freeze alarm” is activated if the temperature is continuously at or below −0.5 °C for more than 1 hour), and in determining whether vaccines exposed to temperatures below zero could be safely used. There are anecdotal reports of concerns from health-care workers (HCWs) about the usefulness of the shake test.
The most appropriate gold standard for the shake test (apart from testing the vaccine in humans, which is impractical) is visual observation under a phase contrast microscope. Phase contrast microscopy is a confirmed method of identifying freeze damage in vaccines. Damaged vaccines contain large conglomerates (massed precipitates with amorphous, crystalline, solid and needle-like structures), whereas vaccines maintained within the optimal temperature range (2 to 8 °C) show a fine-grain structure under phase contrast microscopy.19 These findings have been confirmed by scanning electron microscopy and X-ray analysis, which have shown that aluminium is the main element in the conglomerates (data not shown).
This study was designed to establish the shake test's sensitivity (proportion of vials identified as freeze-damaged by the shake test among vials identified as freeze-damaged by phase contrast microscopy) and specificity (proportion of vials identified as non-frozen by the shake test among vials identified as non-frozen by phase contrast microscopy), using phase contrast microscopy as the gold standard. Positive predictive value – i.e. the probability that a vial identified as freeze-damaged by the shake test is truly freeze damaged – was also calculated.
At the time of the study, 14 manufacturers had 8 freeze-sensitive products that were prequalified by WHO (details of prequalification are available at: http://www.who.int/immunization_standards/vaccine_quality/pq_system/en/index.html).
This study was designed to determine whether the sensitivity or specificity of the shake test varied depending on the specific product or manufacturer. To ensure that the study was statistically sound, it included all products produced by only one manufacturer (e.g. liquid Haemophilus influenzae type b vaccine produced by Merck, Whitehouse Station, USA); all manufacturers having only one product (e.g. hepatitis B vaccine produced by CIGB [Centro de Ingeniería Genética y Biotecnología], Havana, Cuba); a maximum of three randomly selected combinations of product and manufacturer (product selection per manufacturer was limited to a maximum of three; thus, all products from manufacturers with two or three products were included); a minimum of 30 vials of each vaccine type (reduced to 20 vials if more than one manufacturer produced the same product type); and 10-dose presentations (i.e. the most common type), unless the vaccine was only available in smaller doses (WHO experience in the field suggested that if the shake test worked with the selected 10-dose vial, it would also work with vials of any other size; also, vials with different doses have identical substances and mix, so that the behaviour of the shake test should be the same). Sample size calculations were based on an expected 95% specificity and 95% sensitivity of the shake test when compared with phase-contrast microscopy. The overall desired sample size was calculated to be 480 vials. The desired sample size by vaccine type was 30–140 vials, and by manufacturer, 30–60 vials. Table 1 illustrates the sampling framework for the study.
Table 1. Sample design for validating vial shake test for detecting freeze damage to adsorbed vaccines
The sample sizes selected were appropriate to calculate specificity and sensitivity with a high degree of precision. A sample size of 480 could be considered unnecessarily high, but 30 samples of each vaccine type were needed to demonstrate the absence of a statistically significant difference between presentations.
Receiving and storing samples
Each vaccine manufacturer was asked to send either 25 or 35 vials of selected vaccines to WHO in Geneva, Switzerland. The five extra vials (practice vials) were needed for teaching the shake test, carrying out the interobserver variation test and validating the test protocol with phase contrast microscopy.
All vaccines were sent in insulated shipping containers with cool water packs (to maintain a temperature of 2 °C to 8 °C). Temperatures during shipment were monitored with the WHO prequalified 10-day electronic shipping indicator, Q-tag2plus® (Berlinger & Co. AG, Ganterschwil, Switzerland). All samples included in the study were received in good condition as indicated by the Q-tag2plus® and were then stored in a WHO 5 °C storage facility.
Preparation of samples
Samples were divided into three groups for preparing frozen and non-frozen vaccines. The temperature treatment plan for samples is shown in Table 2. First, the original label was removed from each vial and replaced with a study label having a 7-digit numerical code. The codes were assigned by the first co-investigator and were known only to that person until the study was completed. Samples to be exposed to negative temperatures were taken to the Thermometry and Ionizing Radiation Section of the Federal Office of Metrology in Berne, Switzerland. They were placed in one of two chambers in which temperature ranges were recorded as −25.1 °C to −24.7 °C, and −2.3 °C to −1.7 °C. After 24 hours, vaccines were removed from the temperature chambers and the physical status of each vial was examined. Vaccines were then transported to the National Institute of Hygiene in Warsaw, Poland, in insulated containers with cool water packs and one Q-tag2plus® per carton. Vaccines arrived at the study site within 23 hours of pickup and were immediately placed in a storage facility at 5 °C at the National Institute of Hygiene.
Table 2. Temperature treatment of vaccine samples used in study for the validation of the shake test for detecting freeze damage to adsorbed vaccines
The study was organized to be fully blind in each of the three phases – shake test by HCWs, phase contrast microscopy by study centre staff and statistical analysis by the second co-investigator. These individuals had no information on the findings of the others and were unaware of the meaning of the coding in the data sheets.
Shake test by HCWs – first blinding
Five HCWs with no previous experience of vaccines or the shake test were recruited. The principal investigator taught the HCWs how to conduct the shake test following the standard “Shake test learning guide” (Appendix A, available at: http://www.who.int/vaccines-documents/DocsPDF06/847.pdf, pages 59–62). HCWs were given the extra five vials from each type of vaccine to practice with for half a day on their own. As a first step, interobserver variation was checked on 10 frozen and 10 non-frozen samples. During the interobserver variation tests, all HCWs performed “pass” and “fail” tests correctly, and all five were recruited.
Study vials were distributed to the HCWs with a freeze-damaged control vial for each vaccine type used. HCWs recorded the results of each test using an established code for “pass” and “fail” and the time taken to reach a decision. If a test vial contents sedimented at a similar or a faster rate than the contents of the frozen control vial, this was recorded as a “fail”; if the vial contents sedimented at a slower rate than the contents of the frozen control vial, this was recorded as a “pass”.
Phase contrast microscopy – second blinding
Phase contrast microscopy was validated using the extra five vials from each type of vaccine. Once the procedure had been validated, all study samples were examined under phase contrast microscopy. Each vial was vigorously shaken, the aluminium crimping and the rubber stopper were removed from each vial, 10 µl of the vaccine in each vial were dropped onto a slide using a Biohit 0.5–10 µl automatic pipette (Biohyt Oyj, Helsinki, Finland), and a coverslip was placed over the sample. All samples were examined for structural formations under 200 × magnification and were photographed under 50 × magnification. The tests were conducted using a Docuval phase contrast microscope and Docuval and Planachromat 20/0.40 and 160/017 camera equipment (Carl Zeiss, Jena, Germany), and 24 × 36 mm Kodak 200 ASA negative film (Kodak, Hemel Hempstead, England). All photographs were digitized and particle size was measured. Results were coded numerically for “frozen” and “non-frozen” vaccines.
Statistical analysis – third blinding
Results from the shake test and phase contrast microscopy were tabulated separately by the second co-investigator on a 2 × 2 table for each product, by vaccine type, vaccine manufacturer, aluminium content and expiry date as well as together (phase contrast microscopy versus shake test). Sensitivity, specificity and positive predictive values were calculated.
The study was conducted with 480 vials of 8 types of freeze-sensitive WHO prequalified vaccines from 10 manufacturers. During the unpacking of vaccines in Warsaw, 5 vials were broken and were excluded from the study. This reduced the sample to 475 vials. All vaccines exposed to −2 °C for 24 hours (117 vials, excluding one vial broken on arrival in Warsaw) were in a liquid state, as were vaccines kept at 5 °C. A total of 319 vials were not frozen and 156 were frozen.
Phase-contrast microscopy confirmed the known status of the vials: 319 were identified as non-frozen and 156 as frozen. Fig. 2 shows the appearance of non-frozen samples (fine-grain structure) and freeze-damaged samples (large conglomerates of massed precipitates with variable structures) under the phase contrast microscope.
Fig. 2. Phase contrast microscopy findings using study vials of various vaccines kept at different temperatures
Particles in samples identified as non-frozen measured between 1 μm (for vaccines for diphtheria–tetanus–pertussis; Haemophilus influenzae type b; hepatitis B; diphtheria–tetanus–pertussis–hepatitis B) and 20 μm (for vaccines against diphtheria and tetanus, alone or in combination). Aggregates in samples identified as frozen measured up to 700 μm (diphtheria–tetanus–pertussis) and 350 μm on average. The shape of aggregates in frozen samples varied; it included amorphous, solid, crystalline and needle-like structures with sharp edges.
Table 3 shows how the shake test results compared with those of phase contrast microscopy. There was complete concordance between the shake test results as interpreted by the HCWs and phase contrast microscopy readings, with no false positive or false negative readings. Sensitivity, specificity and positive predictive value were calculated as 100% each. Thus, the shake test correctly identified that a vaccine had been affected by freezing 100% of the time (95% confidence interval, CI: 0.97–1.00) and it also correctly identified that a vaccine has not been frozen 100% of the time (95% CI: 0.99–1.00).
Table 3. Concordance between the results obtained with phase contrast microscopy and the shake test for detecting freeze damage to adsorbed vaccines
Since the specificity, sensitivity and positive predictive value of the shake test were all calculated as being 100%, no further statistical analyses were conducted by manufacturer or product. The results fully support the original hypothesis that the sensitivity and specificity of the shake test do not vary by product, type of vaccine, vaccine manufacturer, aluminium content or expiry date.
An additional finding of the study was the time taken by HCWs to reach a conclusion. Although the time required is influenced by the experience of the HCW, results suggest that the test takes longer for smaller vials. The shortest decision time was 44 seconds with a 10-dose tetanus toxoid vaccine, and the longest was 20 minutes with a monodose Haemophilus influenzae type b vaccine. Apart from these extreme values, all other products were analysed within 1 to 5 minutes.
Since the −2 °C exposure did not produce any partial freezing (incomplete crystallization), the study team designed additional tests to complement the findings by generating slushy frozen vaccines. A total of 30 vials from 3 different manufacturers containing 7 different types of vaccines were exposed to −10 °C, with 15 minutes checks to record their freezing status. Eighteen of these vials were removed when they had reached a slushy but not fully solid frozen state, and 12 were allowed to freeze fully and reach a solid state. All samples were tested by the same HCWs in Warsaw using 7 different control vials and were examined by phase contrast microscopy, together with 7 matching vaccine vials kept at 5 °C. All vials that were slushy frozen and vaccines kept at 5 °C produced a “pass” test, and a fine-grain structure was observed in phase contrast microscopy. All 12 vials that were fully frozen produced a “fail” test, and large conglomerates were observed in phase contrast microscopy (Table 4 and Fig. 3). Sensitivity and specificity of the shake test for slushy vaccines were both calculated as 100% (sensitivity 95% CI: 0.86–1.00; specificity 95% CI: 0.93–1.00).
Table 4. Concordance between the results obtained with phase contrast microscopy and the shake test for detecting freeze damage to adsorbed vaccines (test with some vaccines partially frozen)
Fig. 3. Phase contrast microscopy findings using study vials of hepatitis B vaccine kept at different temperatures
This study was conducted to establish the sensitivity and specificity of the shake test by comparison against the actual freezing status of freeze-sensitive vaccines, using phase contrast microscopy as a gold standard. The concordance in establishing the status of a vaccine as frozen or non-frozen was 100% between the phase contrast microscopy and the shake test performed by HCWs. These findings indicate that the shake test has 100% sensitivity, specificity and positive predictive value.
Under the phase contrast microscope, frozen vaccines showed large conglomerates of large precipitates with variable structures. This confirms that freezing breaks the lattice between the adsorbent and the antigen, leading the aluminium to form granules that grow in size. Heavy granules sediment at a faster rate than lighter granules; this is the basis of the shake test.
None of the vaccines that were exposed to −2 °C for 24 hours were found to be frozen; all were in a liquid state. This confirms that actual freezing depends on various factors, including low temperature, duration of exposure to low temperature and agitation during the exposure. Under the phase contrast microscope, vaccines exposed to −2 °C looked identical to those kept at optimum temperatures – all showed fine-grain structure. In a temperature-monitoring study conducted in Thailand, investigators also found “pass” shake test results with vaccines exposed to negative temperatures documented by freeze indicators.20 This finding again confirms that exposure to negative temperatures and actual freezing are two different concepts. Since freeze indicators are the only practical tools available for checking or documenting whether vaccines have been exposed to negative temperatures, the authors strongly recommend the continued use of freeze indicators during in-country vaccine distribution of freeze-sensitive vaccines.
The shake test can also be used on slushy, partially frozen vaccines, and our results indicated that the lattice structure is broken only when solid freezing occurs.
We identified two publications claiming that the shake test is either impractical or not valid for identifying freeze-damaged vaccines. In a paper from Canada, based on a study of 80 vials of diphtheria–tetanus–pertussis and diphtheria–tetanus–pertussis–poliomyelitis vaccines, none of the frozen vials produced a positive shake test.15 The authors indicated that accelerated sedimentation was evident in all frozen vials but found that it took too long – up to 45 minutes – to produce a definitive result. Despite this finding, it is not clear how the authors concluded that the test is “impractical”. An article from India was described as a “validation” study.21 However, the methods used did not correspond to a validation study design. Shake test results were not compared against any gold standard, no control vials were prepared and no standard shake test protocol was followed. Each participant was given one previously frozen and one never frozen vial, and was asked to report results after 15 minutes – an insufficient and arbitrary time-limit. Thus, this publication cannot be considered as a valid study of the shake test.
Two other publications support the shake test. One study evaluated Diphtheria–tetanus–pertussis and DT vaccines from six manufacturers, and concluded that the shake test is useful for absorbed vaccines.13 The other study did not examine the shake test directly, but documented a high rate of sedimentation in aluminium adjuvant vaccines kept at −18 °C compared to non-frozen vaccine samples; the sedimentation rate in frozen samples was 100% in 15–20 minutes compared to a maximum of 34% sedimentation in 24 hours for non-frozen samples.19
Our findings confirmed the importance of using a standard learning guide for training, coupled with a demonstration and coaching for HCWs with no prior knowledge and experience with the test. This approach resulted in HCWs being able to perform and read test results with 100% accuracy. The key to deciding whether a shake test has passed or failed is the patience of the observer. All HCWs were told that they should continue to observe until they were completely confident about the difference or similarity between the control and test vials. Accurately observing the sedimentation requires greater attention with smaller vials because the amount of liquid (and thus the height of the liquid in the vial) is significantly less in monodose and other small vials when compared to multidose vials.
These findings confirm the value of the shake test in deciding whether aluminium-based freeze-sensitive vaccines have been affected by freezing. The specificity and sensitivity of 100% found in this study will bring confidence to staff handling vaccines at the country level.
Note: The following videos about the shake test are available: Shake and Tell – video article (duration 00:22:17), available at: http://vimeo.com/8381355 – and Step by Step Shake Test – educational/instructional video (duration: 00:10:07), available at: http://vimeo.com/8389435
We thank the DHL–UNICEF Partnership for co-funding the validation study. We also thank WHO prequalified vaccine manufacturers BB-NCIPD Ltd, Bulgaria; Berna Biotech Korea Corp., Republic of Korea; CIGB, Cuba; GSK Biologicals, Belgium; Merck & Co., Inc., USA; Panacea Biotec Ltd, India; PT BioFarma, Indonesia; Serum Institute of India Ltd, India; Shantha Biotechnics Ltd, India; and Sanofi Pasteur, France, for participating in the study; Berlinger & Co. AG for donation of Q-tag2plus® devices; and Fride Martin, Steve Wiersma and Philippe Duclos for their assistance during development of the study protocol. We also thank Andrew Garnett for his assistance during the development of this article. Authors acknowledge Nigel Cryer (former DHL, United Kingdom) for establishing the link between the WHO and the DHL–UNICEF Partnership.
This study was co-sponsored by the DHL–UNICEF Partnership.
- Milstien J, Kartoglu U, Zaffran M. Temperature sensitivity of vaccines. Geneva: World Health Organization; 2006 (unpublished document WHO/IVB/06.10)
- Burgess MA, McIntyre PB. Vaccines and the cold chain: is it too hot...or too cold? Med J Aust 1999; 171: 82- pmid: 10474582.
- Milhomme P. Cold chain study: danger of freezing vaccines. Can Commun Dis Rep 1993; 19: 33-8 pmid: 8467304.
- Lugosi L, Battersby A. Transport and storage of vaccines in Hungary: the first cold chain monitor study in Europe. Bull World Health Organ 1990; 68: 431-9 pmid: 2208556.
- Hanjeet K, Lye MS, Sinniah M, Schnur A. Evaluation of cold chain monitoring in Kelantan, Malaysia. Bull World Health Organ 1996; 74: 391-7 pmid: 8823961.
- Jeremijenko A, Kelly H, Sibthorpe B, Attewell R. Improving vaccine storage in general practice refrigerators. BMJ 1996; 312: 1651-2 pmid: 8664720.
- Thakker Y, Woods S. Storage of vaccines in the community: weak link in the cold chain? BMJ 1992; 304: 756-8 doi: 10.1136/bmj.304.6829.756 pmid: 1571683.
- Wawryk A, Mavromatis C, Gold M. Electronic monitoring of vaccine cold chain in a metropolitan area. BMJ 1997; 315: 518- pmid: 9329306.
- Bishai DM, Bhatt S, Miller LT, Hayden GF. Vaccine storage practices in pediatric offices. Pediatrics 1992; 89: 193-6 pmid: 1734382.
- Bell KN, Hogue CJ, Manning C, Kendal AP. Risk factors for improper vaccine storage and handling in private provider offices. Pediatrics 2001; 107: e100- doi: 10.1542/peds.107.6.e100 pmid: 11389298.
- Wirkas T, Toikilik S, Miller N, Morgan C, Clements CJ. A vaccine cold chain freezing study in PNG highlights technology needs for hot climate countries. Vaccine 2007; 25: 691-7 doi: 10.1016/j.vaccine.2006.08.028 pmid: 16968657.
- Matthias DM, Robertson J, Garrison MM, Newland S, Nelson C. Freezing temperatures in the vaccine cold chain: a systematic literature review. Vaccine 2007; 25: 3980-6 doi: 10.1016/j.vaccine.2007.02.052 pmid: 17382434.
- World Health Organization. The effects of freezing on the appearance, potency, and toxicity of adsorbed and unadsorbed DTP vaccines. Wkly Epidemiol Rec 1980; 55: 385-92.
- Klotz SA, Normand R, Silberman R. Hepatitis B vaccine in healthy hospital employees. Infect Control 1986; 7: 365-9 pmid: 3637168.
- Dimayuga R, Scheifele D, Bell A. Effects of freezing on DTP and DTP-IPV vaccines, adsorbed. Can Commun Dis Rep 1995; 21: 101-3 pmid: 7647743.
- Dietz V, Galazka A, van Loon F, Cochi S. Factors affecting the immunogenicity and potency of tetanus toxoid: implications for the elimination of neonatal and non-neonatal tetanus as public health problems. Bull World Health Organ 1997; 75: 81-93 pmid: 9141753.
- Mansoor O, Pillans PI. Vaccine adverse events reported in New Zealand 1990-95. N Z Med J 1997; 110: 270-2 pmid: 9269289.
- World Health Organization. Guideline for establishing or improving primary and intermediate vaccine stores. Geneva: WHO; 2002 (WHO/V&B/02.34).
- Aleksandrowicz J, Drozdz M, Fiejka M, Kurzatkowski W. Evaluation of the physico-chemical state of aluminium hydroxide in biopreparations stored in various conditions Exp Med Microbiol 1990; 42: 163-70.
- Techathawat S, Varinsathien P, Rasdjarmrearnsook A, Tharmaphornpilas P. Exposure to heat and freezing in the vaccine cold chain in Thailand. Vaccine 2007; 25: 1328-33 doi: 10.1016/j.vaccine.2006.09.092 pmid: 17157419.
- Grover A, Amarjeet S, Aggarwal AK, Thakur JS. Does Indian version shake test correctly identify frozen damage vaccines? Bull Postgrad Inst Med Educ Res Chandigarh 2004; 38: 125-30.