< > Home : FAQ : Professional : Chlamydiales : Immunobiology : Infections : Diagnosis & Treatment : Links : Contact Us

The traditional reference test for determining whether chlamydial infection is present  is the growth of chlamydiae in cell culture. Cell culture is usually considered 100% specific, since the presence of a chlamydial inclusion is an unambiguous demonstration of infection [always assuming the cell culture hasn't been contaminated]. However, cell culture is an inadequate "gold standard" as it is substantially less than 100% sensitive, depending on the comparators. This is due to  the necessity of maintaining the viability of chlamydiae in patient samples prior to inoculation, plus the effects of many potential variables that can affect the culture process. Discrepant analysis is an attempt to identify the true positive cases of infection that cell culture misses. Apparent "false positives" [culture negative, test positive] are subjected to a battery of further tests. If any of these are positive, the original positive result is considered a true positive and, conversely, the original cell culture negative is considered a false negative. Discrepant analysis has been used for many years [Schachter et al., 1988] to evaluate laboratory tests for chlamydial infection. Its advocates argue that  the discrepant analysis-based estimates of specificity and sensitivity are typically less biased than those based on culture. However, its use has been strongly criticised as being inherently unscientific, leading to results biased in favor of new tests [Hadgu, 1996; 2000].   

To make his case, Hadgu, 1996 analyses a paper by Schachter et al., 1994 on the use of LCR, [a nucleic acid amplification based method], for the diagnosis of cervical C. trachomatis infection. In the latter paper, 2132 women were tested by LCR and cell culture. 139 women were LCR and culture positive and 84 women were culture negative but LCR positive. Sensitivity before discrepant analysis was 139/152 = 91.4% and specificity was 1896/1980 = 95.8%. The 84 false positives were tested by direct flourescence for chlamydial antigen, when a further 48 were considered 'positive'. The remaining 36 were tested with an LCR directed against a different gene target, giving rise to 28 further 'positives'. Of the remaining 8 discrepants, 6 were positive when tested by the second LCR at lower dilution [presumably diluting out inhibitors]. Thus 82 or the 84 samples originally false positive were declared true positives [Schachter et al., 1994]. Hadgu 1996 points out that almost all studies using discrepant analysis eventually reclassify over 95% of false positives as true positives. He considered [Hadgu, 1997; 2000] that the use of discrepant analysis violates a fundamental principle of diagnostic testing: that the new test should not be used to determine the true disease status. Bias occurs because only those samples considered false positive are subjected to a battery of further testing, leading to an overestimate of the performance of the new test. In the case of the paper he reviewed, the 1,896 LCR negative cell culture negative specimens were not subject to testing by direct fluorescence or by the second LCR. Sampling variability makes it inevitable that repeat nucleic acid amplification tests on initially negative samples with low amounts of target nucleic acid will in practise lead to significantly increased numbers of positive specimens, a phenomenon which can be investigated by probit analysis [Smieja et al., 2001]. There were also concerns that the use of a second LCR for confirmation was fundamentally flawed because both tests, although directed against different gene targets, were susceptible to being contaminated with laboratory-derived chlamydial DNA [Hadgu, 1996]. Statistical modelling should be used to estimate test performance where a "gold standard" reference test is considered to be inadequate [Hadgu, 1996b]. This view was broadly supported by Danielsson et al., 1996 who pointed out that in their experience, high quality cell culture gave similar yields of infected cases to a battery of two or more DNA amplification-based tests. McGee & Reynolds, 1996, were concerned that even a small drop in the true sensitivity and specificity of a diagnostic test could have a substantial effect on its utility for screening for infection in low prevalence populations [see: diagnosis, positive and negative prediction]. Hilden, 1997 considered that the use of discrepant analysis introduced  bias in favour of the new test and was against "clinicometric logic". However, the sensitivity and specificity of a diagnostic test are not some universal constant, but are dependent on operator skill and the chosen cut off level.  Infection cannot be absolutely classified as being absent or present, as the early stages are often difficult to detect. He considered it unlikely that overestimation arising from the use of discrepant analysis would greatly affect anti chlamydial therapy in individuals, because of the complexities and unknowns that already existed. However clinical outcomes might certainly be affected, particularly if exaggerated statistics lead to poor tests ousting better ones from the market.

In a series of letters, Schachter et al., 1996; 1997; 1998 considered that criticism of the use of discrepant analysis for evaluating the performance of chlamydial diagnostic tests was "much ado about nothing" [Schachter et al., 1996]. The choices facing the investigator of a new chlamydial diagnostic test were: 1. use an imperfect gold standard (culture); 2. use discrepant analysis as applied; or 3. use a 'tie-breaker' test on both culture negative / amplification positive specimens and with culture negative amplification negative specimens. They pointed out that the variable sensitivity of cell culture is well known. From their experience of evaluating chlamydial diagnostic tests, they considered that the use of a 'tie breaker' applied to test negative as well as positive specimens was likely to result in a huge and expensive work load for little gain.

Green et al., 1998, compared the bias in estimates based discrepant-analysis with that in estimates based on the culture classification. They concluded that the bias in estimates of amplification-based test specificity based on discrepant analysis was small and generally less than that in estimates based on culture. However, the accuracy of discrepant analysis-based estimates of sensitivity depended crucially on whether culture specificity is equal to, or slightly less than 100%. It was also affected by competing biases that are not fully taken into account by discrepant analysis [Green et al., 1998; Green et al., 2001].  

As a subject's true disease status is seldom known with certainty, it is necessary to compare the performance of new diagnostic tests with those of a currently accepted but imperfect 'gold standard'. The effect of choice of gold standard is clearly seen in the paper of Darwin et al., 2002 [see hyperlinked abstract]. Errors made by the gold standard mean that the sensitivity and specificity calculated for the new test are biased. Hawkins et al., 2001 provide a theoretical discussion of the problems and give formulae designed to show how to evaluate the impact of different strategies for discrepant analysis so as to settle on a design that gives the required precision of estimates [Hawkins et al., 2001].

Relative measures of test accuracy are less prone to bias in incomplete study designs. One study attempted to estimate the relative sensitivity and relative false-positive rate (RFP) and associated biases for PCR and LCR versus cell culture among 1,138 asymptomatic men. The potential bias in relative sensitivity and relative false positive estimates were <5 and <20%, respectively, based on the most likely, probably conservative parameter settings [Cheng et al., 2001].

[Comment: Specimen transport for chlamydial culture, and the culture process itself, are subject to considerable variability, not all of which can be adequately and consistently controlled. The specificity of culture is potentially excellent as there is little ambiguity about the presence of chlamydial conclusions. However in an active chlamydial diagnostic facility, cultures as well as amplification-based tests can be cross-contaminated with other patient specimens. Overall, there can be little doubt that culture is an inadequate gold standard by which to judge other diagnostic tests. Equally there can be little doubt that subjecting only new test-positive tests to a battery of further tests is unscientific and potentially a source of bias, although that bias is probably fairly small. Inappropriate use of discrepant analysis is one reason why many of the early evaluations of amplification based tests against culture gave higher specificity and sensitivity figures than are generally accepted today.]

[MEW] April 2002

NEXT: Benefits of sample pooling

References

Cheng, H., Macaluso, M., Vermund, S. H. & Hook, E. W. 3rd. (2001). Relative accuracy of nucleic acid amplification tests and culture in detecting Chlamydia in asymptomatic men. Journal of Clinical Microbiology 39, 3927 - 3937. 

Danielsson, D., Falk, T. & Persson, K. (1996). Discrepant analysis and screening for Chlamydia trachomatis. Lancet. 348, 1308. [Letter].

Darwin, L. H., Cullen, A. P., Arthur, P. M., Long, C. D., Smith, K. R., Girdner, J. L., Hook III, E. W., Quinn, T. C. & Lorincz, A. T.  (2002). Comparison of Digene Hybrid Capture 2 and Conventional Culture for Detection of Chlamydia trachomatis and Neisseria gonorrhoeae in Cervical Specimens. Journal of Clinical Microbiology 40, 641 - 644.

Hadgu, A. (2000). Discrepant analysis: a biased and an unscientific method for estimating test sensitivity and specificity. Journal of Clinical Epidemiology 52, 1231 - 1237.

Hadgu, A. (1997). Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis. Statistics in Medicine 16, 1391 - 1399.

Hadgu, A. (1996). The discrepancy in discrepant analysis. Lancet 348, 592 - 593.

Hadgu, A. (1996b). Discrepant analysis and screening for Chlamydia trachomatis. Lancet. 348, 1309. [Letter].

Hawkins, D. M., Garrett, J. A., Stephenson, B. (2001). Some issues in resolution of diagnostic tests using an imperfect gold standard. Stat Med 20, 1987 - 2001.

Hilden, J. (1997). Discrepant analysis - or behaviour? Lancet 350, 902. [Comment].

Green, T. A., Black, C. M. & Johnson, R. E. (1998). Evaluation of bias in diagnostic-test sensitivity and specificity estimates computed by discrepant analysis. Journal of Clinical Microbiology 36, 375 - 381. Full article.

Green, T. A., Black, C. A. & Johnson, R. E. (2001). In defense of discrepant analysis. Journal of Clinical Epidemiology 54, 210 - 215.

Johnson, R. E., Green, T. A., Schachter, J., Jones, R. B., Hook, E. W. 3rd., Black, C. M., Martin, D. H., St Louis, M. E. & Stamm, W. E. (2000). Evaluation of nucleic acid amplification tests as reference tests for Chlamydia trachomatis infections in asymptomatic men. Journal of Clinical Microbiology 38, 4382 - 4386. Full article   

McGee, D. L. & Reynolds, G. H. (1996). Discrepant analysis and screening for Chlamydia trachomatis. Lancet. 348, 1307 - 1308. [Letter].

Schachter, J., Moncada, J., Dawson, C. R., Sheppard, J., Courtright, P., Said, M. E., Zaki, S., Hafez, S. F. & Lorincz, A. (1988). Nonculture methods for diagnosing chlamydial infection in patients with trachoma: a clue to the pathogenesis of the disease? Journal of Infectious Diseases 158, 1347 - 1352.

Schachter, J., Stamm, W. E. & Quinn, T. C. (1998). Discrepant analysis and screening for Chlamydia trachomatis. Lancet. 351, 217 - 218. [Letter].

Schachter, J., Stamm, W. E. & Quinn, T. C. (1996). Discrepant analysis and screening for Chlamydia trachomatis. Lancet. 348, 1308 - 1309. [Letter].

Schachter, J., Stamm, W. E., Quinn, T. C., Andrews, W. W., Burczak, J. D. & Lee, H. H. (1994). Ligase chain reaction to detect Chlamydia trachomatis infection of the cervix. Journal of Clinical Microbiology 32, 2540 - 2543.

Smieja, M., Mahony, J. B., Goldsmith, C. H., Chong, S., Petrich, A. & Chernesky, M. (2001). Replicate PCR Testing and Probit Analysis for Detection and Quantitation of Chlamydia pneumoniae in Clinical Specimens. Journal of Clinical Microbiology 39, 1796 - 1801.