Question
How many test results run by different nucleic acid amplification tests (NAAT) and what combinations of specimens comprise the best infected-patient gold standard for evaluating the performance of a new NAAT for the detection of C. trachomatis?

Design
Multiple different infected-patient gold standards for estimating swab and urine specimen sensitivity and specificity for one NAAT method were created by varying the number and combination of swab and urine comparator results obtained with two different NAATs.  Data from a large study of the performance of a new NAAT for the detection of C. trachomatis in men and women were used to construct receiver-operator-like curves that assessed the different gold standard definitions using a single test with a predetermined definition of positive.  

Participants
The results of NAAT performed on endocervical and urine specimens from 1412 women and urethral and urine specimens from 1045 men enrolled at seven clinical sites in the United States, including sexually transmitted disease and family planning clinics, were used for the analyses.  Specimens were excluded if the patient had urinated within 1 hr before providing the specimen, or had taken antibiotics within the previous 21 days, or if collection, storage, or transport requirements were not met.    

Description of Tests and Diagnostic Standard
Endocervical swabs and urine specimens from men and women were tested for C. trachomatis using 3 NAAT, including transcription mediated amplification (TMA) (APTIMA Combo 2, GenProbe Inc., San Diego, CA), ligase chain reaction (LCR) (LCx, Abbott Laboratories Inc., Abbott Park, IL), and polymerase chain reaction (PCR) (Amplicor, Roche Diagnostic Systems, Indianapolis, IN).   Urethral swabs from men were tested using TMA and LCR.  

The results of swab and urine specimens tested by two different assays were used equally in a rotating fashion to establish definitions of true positive results to assess the performance of the swab and urine specimens tested by the third assay.  The sensitivity and specificity for each assay were calculated using a decreasing number of results from the comparator assays to define the infected patient standard (see table for definitions of the standards).  Curves were constructed by plotting sensitivity on the y axis against 1-specificity on the x axis.  For women, swab and urine specimens tested by two different assays were used to assess the performance of the swab and urine specimens tested by a third assay, generating 6 sets of plots (performance of swab or urine tested by PCR, LCR, or TMA) with four curves (one for each number of comparator assay results used) in each set.  For men, only one plot of the urine PCR performance could be generated because the urethral swabs were tested by only 2 of the 3 NAAT.  Direct comparisons of TMA and LCR were done with the PCR results and TMA was compared to the PCR assay with the LCR results.

Definitions of true positive results for determination of performance of a third assay using up to 2 NAAT assays and 2 specimen types per assay. 

Main Outcome Measures
The effect on the performance characteristics of the NAAT under evaluation was determined when the available comparator NAAT results used to define the infected-patient standard were reduced from four tests to three tests to two tests.  

Main Results
Comparisons of the curves revealed several facts.  1) Curves with 3 comparator results closely approximated the curves with 4.  2) Requiring all comparator results to be positive (definitions a, f, j, l) maximized the sensitivity but lowered specificity.  3) Requiring 3 of 4 results, or 2 of 3 results to be positive (definitions b, g, h) resulted in higher specificity without lowering sensitivity significantly.  4) Requiring only one comparator result to be positive (definitions e, i, k, m) maximized specificity but lowered sensitivity.  5) The highest combined estimates of sensitivity and specificity were obtained from definitions b, c, and h.  6) The highest combined estimates of swab sensitivity and specificity were derived by using 2 swab and one urine specimen, while the highest combined estimates of urine performance were provided by 2 urine and one swab specimen as comparators.  

In the comparisons of TMA to LCR using PCR swab and urine results and to PCR using LCR swab and urine results as the standards (definition m), TMA was more sensitive than both PCR and LCR, but had lower specificity.   

Authors’ Conclusions
Using 3 comparator results and defining the infected patient as any two positives of the 3 possible results provided accurate estimates of sensitivity and specificity and is less costly than a four-comparator standard.   For evaluating new C. trachomatis diagnostic tests in women, we recommend comparing one urine and two swab results tested by 2 different FDA-cleared NAAT and one swab and two urine results tested by 2 different FDA-cleared NAAT for the evaluation swab specimens and urine specimens, respectively, with any 2 out of 3 positive comparator results as the positive standard.  For evaluation of new tests in men, given the difficulty in obtaining more than 2 urethral swabs specimens, comparing one swab and two urine results tested by 2 different FDA-cleared NAAT would be optimal for estimating the performance of urine specimens and adequate for swab specimens.  

Source of funding:  None given

For correspondence:  David H. Martin, LSU Health Sciences Center, Department of Medicine, Section of Infectious Diseases, 1542 Tulane Ave., New Orleans LA 70112.  E-mail address:   dhmartin@lsuhsc.edu.