Spectrum Bias: Patient Selection May Affect Performance of Calcium Volume in Detecting Carotid Stenosis

We read the article recently published on-line in the American Journal of Neuroradiology by Marquering et al¹ with great interest, and we appreciate the authors' efforts to test the hypothesis suggested in one of the articles published by our group in 2005,² by reproducing the same methods, albeit on a different subject population. In that particular regard, we would like to note several prominent differences from ours in their study, most notably the method of selecting their patient population. The preliminary work by McKinney et al² had been intended to lay the basic foundation for future studies with the possibility of using calcium (Ca) volume detection as a screening test for carotid atherosclerotic disease. Given the inherent features of a screening test, such a test should be an adequate one to detect the disease at certain cutoff points, not solely in the diseased or the high-risk population, but in the general population (ie, including mostly nondiseased as well as diseased subjects).³ Hence, we think that the current study deviates from this belief by incorporating high-risk patients as the selected subject population.

Another notable point of difference is that the current study used Ca thresholds of 0.03 and 0.09 mL, whereas the previous study by McKinney et al² found the best combination of sensitivity and specificity at the 0.06-mL threshold, as reflected in the abstract. Instead of using the 0.06-mL threshold, the authors chose differently but note in their article “The chosen Ca volume thresholds used in our test were the largest (0.09 mL) and smallest (0.03 mL) that performed well in the study of McKinney et al.” This is a correct statement, but one that does not clearly reflect the stated results of the earlier study.

Third, in the study by McKinney et al,² several factors had been listed as limitations, which are also in effect for the article by Marquering et al,¹ because the methods were quite similar, with the exception of the patient population. In a very recent study,⁴ coincidentally published nearly at the same time as the article by Marquering et al, we made 3 important improvements in the study protocol: 1) We replaced CTA with catheter angiography as the diagnostic tool to assess the luminal patency, leaving the use of the gold standard NASCET criteria instead of NASCET-like surrogate criteria to assess the stenosis; 2) we replaced CTA with nonenhanced CT to overcome overlapping of Ca and contrast densities as was seen in the previous study, which could have resulted in overestimation of the Ca burden in the previous study; and 3) we changed the subject population to a more random one that simulates a sampling of the general population. Our patient population in this newer study consisted of patients imaged for reasons other than stroke or cerebrovascular disease, which was thought more reflective of the disease-free general population. The results of this second study were quite similar to those published in 2005, pertaining to the correlation of Ca volume with luminal stenosis; we note that in this more recent study, the Ca volume threshold of 0.06 mL was again found to have the best performance.⁴

Suggesting a definitive hypothesis at this point to explain the discrepant results between the 2 studies published by our group and the study by Marquering et al¹ is actually beyond the scope of this communication. However, we briefly state that such discordant findings could well be explained in the context of “spectrum bias,” implying that the performance of a diagnostic test may vary between different clinical settings due to changes in the patient case mix, therefore affecting the reproducibility of study results.⁵ Because “spectrum bias” is not a true bias in the statistical sense, some authors instead suggest using the term “spectrum effects.”⁶ As a side note, Marquering et al actually showed some poor correlation on the nonaffected side, which, in a way, could again be explained by spectrum bias (ie, the nonaffected side might be regarded as a different subgroup). Other factors might also play a role in the interpretation of their results as outlined in the first paragraph.

Ca volume detection for the carotid and intracranial circulation is a relatively newer topic for research with many unknowns at the moment. However, the facts we learned from the cardiology literature should be enlightening. Ca volume in the form of a Ca score has been used for many years to predict atherosclerotic disease of the coronary arteries.^6–8 Besides a well-known correlation with the luminal stenosis, using the additional diagnostic value of Ca volume has been suggested because the luminal diameter might not always reflect the severity of the atherosclerotic disease alone, due to positive remodeling.⁹ Unfortunately, but understandably, to our knowledge, there is no published work in the literature to test the Ca score against luminal stenosis in patients with acute myocardial syndrome. These patients would most likely undergo catheter angiography with possible intervention in the first place without any less accurate test under development because conventionally, there has been no need to perform a “screening test” on the diseased population.

Hence, we certainly value the results presented in the article by Marquering at al¹ and appreciate their important contribution to the literature and understanding of this topic. In our opinion, all 3 studies cited here are essentially preliminary studies.^1,2,4 Ongoing research concerning Ca volume detection in the carotid and cerebral vasculature could focus on the direct relation of the Ca volume with disease presence as a marker, regardless of the degree of luminal stenosis, because there is a growing body of evidence suggesting that there is a correlation between the Ca volume and ischemic white matter disease burden, and also ultimately with clinical disease states such as dementias.^10–13 Future effort could focus on easy, reliable, and reproducible methods of Ca volume detection in the carotid and cerebral vasculature.

• © 2012 by American Journal of Neuroradiology