Interobserver Reliability of Baseline Noncontrast CT Alberta Stroke Program Early CT Score for Intra-Arterial Stroke Treatment Selection

Discussion

Among highly experienced readers, there was substantial interobserver agreement for grading baseline ASPECTS on acute noncontrast head CT in the setting of anterior circulation proximal artery occlusion. Most differences in ASPECTS were within 1–2 points in this study. However, only moderate interrater agreement was achieved for assessing dichotomized ASPECTS > 7 versus ≤7. In approximately 25% of our cohort, treatment decisions for IAT based on dichotomized ASPECTS would have been affected by the interrater reliability.

Our findings with respect to total ASPECTS scoring support a previous study that also demonstrated substantial agreement (weighted κ = 0.69) across the entire ASPECTS scale.¹⁸ In that study, the authors found a mean interrater difference of zero, with a standard deviation of 1.1 points. Similarly, we found a mean difference of 0.2, with 76.8% of scores being within 1 point.

Previous studies have shown that baseline ASPECTS >7 identifies patients who are more likely to benefit from endovascular treatment.^8⇓–10 However, based on our findings, the utility of this approach may be adversely affected by its limited interrater reliability. The literature is conflicting with respect to this question. The original ASPECTS article by Barber et al⁷ reported substantial to near-perfect agreement for ASPECTS dichotomized at 7, with κ values ranging from 0.71 to 0.89. However, another study by Mak et al¹¹ demonstrated a κ score of 0.34 (fair), with a strikingly low 42% rate of observed agreement. When adjusting for prevalence and bias, agreement improved to moderate (PABAK = 0.44). Our results confirm only a moderate level of reliability (κ = 0.53, PABAK = 0.54) for dichotomized ASPECTS in the setting of proximal occlusions.

It is unclear exactly why the interrater agreement for dichotomized ASPECTS was so different between the study by Barber et al⁷ and ours. However, this difference is probably related to variable selection bias and consequently differing patient cohorts. Timing of imaging is 1 factor that can influence lesion detection. In their study, patients underwent baseline NCCT within 3 hours of stroke onset and were treated with intravenous alteplase, while in the present study, at least half of the patients were imaged beyond 3 hours (median 188.5 minutes, IQR 83–278 minutes) and were treated with IAT. However, the increased time to imaging in our study would be expected to yield more conspicuous lesions and hence a better agreement rather than a worse one. Another possible explanation for the varying agreement is a difference in ASPECTS distribution. Although the median baseline ASPECTS was 8 in both studies, a broader distribution (ie, larger variance) in 1 study would result in a smaller proportion of scores around the threshold of 7 and therefore fewer potential discrepancies. Unfortunately, Barber et al⁷ did not report the proportions of individual scores in their study. In our cohort, almost half (48.4% [75/155]) of the consensus scores were 7 or 8 and thus prone to interrater disagreement.

This last issue underscores the problem with using a binary classification scheme. Clearly, dichotomization will penalize scores that are discrepant by only 1 point if they are around the threshold value. Therefore, despite our excellent agreement for total ASPECTS, we achieved only moderate agreement for dichotomized ASPECTS because of the clustering of scores around 7. This limitation may particularly affect favorable IAT candidates (ie, those who present with minimal tissue injury). If a small infarct is present in the setting of a proximal occlusion, it typically involves the caudate nucleus, lentiform, and/or insula, which are end-artery territories. As such, the baseline ASPECTS is often 7 or 8 in these patients. Also in such patients, the moderate level of agreement that we observed for the lentiform nucleus may further contribute to the relatively low rate of agreement for dichotomized ASPECTS.

Our findings with respect to individual ASPECTS regions are novel, as no study, to our knowledge, has reported interobserver agreement by region. Notably, there was only fair to moderate agreement in the internal capsule and the cortical regions. However, this was mostly related to the low prevalence of ischemic involvement in our study. When accounting for prevalence and bias, these areas demonstrated substantial or near-perfect agreement, suggesting that there was strong consensus between the readers as to what was considered abnormal. It is somewhat surprising that the agreement for total ASPECTS was high despite the low level of agreement in these multiple regions. This suggests that overall agreement is determined largely by agreement within the more prevalent ASPECTS regions (eg, basal ganglia, insula).

This study is limited primarily by our highly selected patient population, all of whom underwent IAT. This explains why our ASPECTS distribution was skewed toward smaller infarcts, which are thought to be more likely to benefit from treatment. However, it is in this specific patient subset where interobserver agreement is most important. Moreover, for the analysis of dichotomized ASPECTS, and of the individual ASPECTS regions, we were able to mitigate this issue by adjusting for prevalence. Another limitation is that we did not assess intraobserver reliability. However, we believe that interobserver agreement is more relevant with respect to the clinical utility of ASPECTS. In order for NCCT ASPECTS to be widely adopted for selecting patients for IAT, it must have good reliability between readers at different medical centers. A further limitation is that our findings cannot be extended to general clinical practice, where stroke-imaging studies are often interpreted by physicians who are not expert at ASPECTS grading. Previous studies have shown that the accuracy of ASPECTS evaluation is dependent on the level of rater experience.^7,19,20 However, we wanted to assess the reliability of ASPECTS under the most ideal conditions (ie, using readers with significant experience with stroke imaging and ASPECTS grading). In support of this last point, a recent study by Wardlaw et al²¹ found that the reliability of grading acute ischemia on NCCT (versus an expert neuroradiologist) was improved if the reader was a neuroradiologist and if the scan was acquired later in the 6-hour window. In our study, both readers were neuroradiologists and at least half of the scans were acquired after 3 hours. Future studies are needed to verify our findings in an unselected population of patients with AIS with proximal occlusions.