Effect of Online Training on the Reliability of Assessing Sacroiliac Joint Radiographs in Axial Spondyloarthritis: A Randomized, Controlled Study

Anna E.F. Hadsbjerg; Mikkel Østergaard; Joel Paschke; Raphael Micheroli; Susanne J. Pedersen; Adrian Ciurea; Michael J. Nissen; Kristyna Bubova; Stephanie Wichuk; Manouk de Hooge; Simon Krabbe; Ashish J. Mathew; Monika Gregová; Marie Wetterslev; Karel Gorican; Karlo Pintaric; Ziga Snoj; Burkhard Möller; Alexander Bernatschek; Maurice Donzallaz; Robert G. Lambert; Walter P. Maksymowych

doi:10.3899/jrheum.2024-0075

Article Figures & Data

Figures

Tables
Additional Files

Download figure
Open in new tab
Download powerpoint
Figure 1.
Examples of the scoring interface in the RETIC module, with 1 case scored in agreement with the RETIC developers (on the image named Expert 1 and Expert 2) regarding mNY positivity (left image) and 1 case in disagreement (right image). mNY: modified New York; RETIC: real-time iterative calibration; SIJ: sacroiliac joint.
Download figure
Open in new tab
Download powerpoint
Figure 2.
Flowchart describing the scoring exercise. Two participants randomized to group A did not complete the scoring exercise; both never started the exercise. RETIC: real-time iterative calibration; SIJ: sacroiliac joint.
Download figure
Open in new tab
Download powerpoint
Figure 3.
Mean reliability (Cohen and ICC) between readers, stratified by level of prior experience (inexperienced, level 0-2 vs experienced, level 3-4), and RETIC developer (RGL) in the 3 exercises. ICC: intraclass correlation coefficient; RETIC: real-time iterative calibration.

View popup

Table 1.

mNY grading: agreement with expert reader (RETIC developer).

	RETIC Developer^{^a}	Group A and B Combined			Group A^{^b} All, n = 8	Group B^{^c} All, n = 11
	RETIC Developer^{^a}	All, n = 19	Inexperienced, n = 4	Experienced, n = 15	Group A^{^b} All, n = 8	Group B^{^c} All, n = 11
Percentage exact agreement (mNY positivity), mean, median (min-max)
Exercise I	96	83, 84 (56-96)	72, 74 (56-84)	86, 88 (72-96)	81, 84 (56-88)	85, 84 (68-96)
Exercise II	96	85, 86 (72-100)	80, 80 (76-84)	86, 88 (72-96)	88, 88 (76-100)	82, 80 (72-96)
Exercise III	96	94, 96 (72-100)	87, 86 (76-100)	93, 92 (84-100)	92, 96 (72-100)	93, 96 (72-96)
(mNY positivity), mean, median (min-max)
Exercise I	0.92	0.66, 0.68 (0.09-0.92)	0.43, 0.48 (0.09-0.68)	0.72, 0.76 (0.43-0.92)	0.61, 0.68 (0.09-0.61)	0.70, 0.68 (0.35-0.92)
Exercise II	0.92	0.73, 0.75 (0.51-1.00)	0.61, 0.61 (0.53-0.68)	0.74, 0.76 (0.51-1.00)	0.76, 0.75 (0.51-1.00)	0.68, 0.68 (0.53-0.92)
Exercise III	0.92	0.80, 0.92 (0.45-1.00)	0.70, 0.72 (0.45-0.92)	0.89, 0.92 (0.60-1.00)	0.84, 0.92 (0.45-1.00)	0.86, 0.92 (0.51-1.00)
ICC (mNY grades 0-8), mean, median (min-max)
Exercise I	0.90	0.76, 0.84 (0.28-0.92)	0.56, 0.57 (0.28-0.83)	0.83, 0.87 (0.56-0.92)	0.71, 0.82 (0.28-0.88)	0.81, 0.85 (0.56-0.92)
Exercise II	0.94	0.79, 0.83 (0.54-0.95)	0.70, 0.70 (0.54-0.88)	0.82, 0.83 (0.64-0.95)	0.85, 0.87 (0.72-0.95)	0.75, 0.81 (0.54-0.85)
Exercise III	0.93	0.85, 0.90 (0.43-0.93)	0.72, 0.76 (0.43-0.91)	0.88, 0.90 (0.78-0.93)	0.82, 0.87 (0.43-0.92)	0.87, 0.90 (0.64-0.93)
Sensitivity/specificity, mean, median (min-max)
Exercise I	0.92/1.00	0.87, 0.92 (0.69-1.00)/ 0.79, 0.92 (0.17-1.00)	0.88, 0.91 (0.77-0.92)/ 0.55, 0.55 (0.17-0.92)	0.87, 0.85 (0.69-1.00)/ 0.86, 0.92 (0.50-1.00)	0.85, 0.85 (0.69-0.92)/ 0.76, 0.88 (0.17-1.00)	0.89, 0.92 (0.77-1.00)/ 0.81, 0.92 (0.42-1.00)
Exercise II	0.93/0.93	0.91, 0.91 (0.73-1.00)/ 0.81, 0.79 (0.57-1.00)	0.89, 0.91 (0.73-1.00)/ 0.73, 0.72 (0.57-0.93)	0.92, 1.00 (0.73-1.00)/ 0.83, 0.79 (0.57-1.00)	0.83, 0.82 (0.73-1.00)/ 0.92, 0.93 (0.79-1.00)	0.98, 1.00 (0.91-1.00)/ 0.73, 0.79 (0.57-0.93)
Exercise III	0.91/1.00	0.94, 1.00 (0.73-1.00)/ 0.93, 0.96 (0.82-1.00)	0.87, 0.87 (0.73-1.00)/ 0.84, 0.86 (0.64-1.00)	0.96, 1.00 (0.82-1.00)/ 0.94, 0.93 (0.79-1.00)	0.93, 0.96 (0.82-1.00)/ 0.91, 0.97 (0.64-1.00)	0.94, 1.00 (0.73-1.00)/ 0.92, 0.93 (0.79-1.00)

Sensitivity and specificity were calculated with the RETIC developer radiologist (RGL) score as the gold standard reference.
↵^a Numbers represent the agreement in percentage exact agreement, , and ICC between the 2 RETIC developers (RGL and WPM). Sensitivity and specificity are for WPM’s scores with RGL as the gold standard reference.
↵^b RETIC module done between exercises I and II.
↵^c RETIC module done between exercises I and III. ICC: intraclass correlation coefficient; max: maximum; min: minimum; mNY: modified New York; RETIC: real-time iterative calibration.

View popup

Table 2.

Reliability of the different lesion scores: agreement with expert reader (all participants, n = 18).

	Sclerosis	Erosion	JSW	JSN	Ankylosis
(based on binary sum score 0-1) mean, median (min-max)
Exercise I	0.22, 0.18 (−0.20 to 0.80)	0.36, 0.32 (−0.26 to 0.84)	0.35, 0.36 (−0.07 to 0.66)	0.26, 0.28 (−0.28 to 0.52)	0.65, 0.69 (0.21-1.00)
Exercise II	0.57, 0.59 (0.14-0.91)	0.51, 0.52 (0.23-0.84)	0.22, 0.27 (−0.03 to 0.43)	0.18, 0.18 (−0.16 to 0.55)	0.47, 0.46 (0.00-0.72)
Exercise III	0.51, 0.47 (0.26-0.84)	0.56, 0.60 (0.07-0.84)	0.50, 0.52 (0.14-0.82)	0.41, 0.41 (0.00-0.82)	0.38, 0.39 (0.00-0.72)
ICC, mean, median (min-max)
Exercise I	0.44, 0.43 (0.08-0.78)	0.55, 0.60 (0.11-0.76)	0.41, 0.44 (0.06-0.66)	0.30, 0.30 (−0.20 to 0.60)	0.71, 0.75 (0.30-0.94)
Exercise II	0.62, 0.63 (0.14-0.86)	0.58, 0.61 (0.30-0.87)	0.39, 0.43 (0.02-0.64)	0.26, 0.29 (−0.11 to 0.70)	0.59, 0.63 (0.00-0.80)
Exercise III	0.63, 0.65 (0.28-0.87)	0.64, 0.66 (0.43-0.80)	0.55, 0.58 (0.11-0.92)	0.37, 0.35 (−0.03 to 0.81)	0.58, 0.62 (0.00-0.83)

ICC: intraclass correlation coefficient; JSN: joint space narrowing; JSW: joint space widening; max: maximum; min: minimum.

View popup

Table 3.

Subgroup analyses of the mNY gradings.

	Patients With No/Mild Radiographic Damage^{^a}				Patients With Moderate Radiographic Damage^{^b}
	RETIC Developer^{^c}	Group A and B Combined, n = 19	Group A^{^d}, n = 8	Group B^{^e}, n = 11	RETIC Developer^{^c}	Group A and B Combined, n = 19	Group A^{^d}, n = 8	Group B^{^e}, n = 11
Percent exact agreement (mNY positivity), mean, median (min-max)
Exercise I	93	75, 80 (33-93)	73, 80 (33-87)	77, 80 (47-93)	92	81, 81 (69-100)	78, 77 (69-85)	83, 85 (77-100)
Exercise II	94	82, 81 (69-100)	87, 88 (75-100)	78, 75 (78-94)	94	82, 81 (63-100)	81, 81 (63-100)	82, 81 (68-94)
Exercise III	100	92, 93 (67-100)	92, 97 (67-100)	90, 93 (73-100)	93	94, 93 (79-100)	94, 97 (79-100)	92, 93 (79-100)
(mNY positivity), mean, median (min-max)
Exercise I	0.76	0.35, 0.29 (0.00-0.82)	0.29, 0.33 (0.00-0.44)	0.40, 0.29 (0.05-0.82)	0.83	0.57, 0.55 (0.16-1.00)	0.50, 0.52 (0.16-0.68)	0.62, 0.58 (0.42-1.00)
Exercise II	0.82	0.54, 0.46 (0.29-1.00)	0.58, 0.52 (0.29-1.00)	0.51, 0.46 (0.35-0.82)	0.87	0.60, 0.60 (0.24-1.00)	0.63, 0.64 (0.24-1.00)	0.61, 0.60 (0.33-0.87)
Exercise III	1.00	0.57, 0.63 (−0.10 to 1.00)	0.72, 0.82 (0.19-1.00)	0.47, 0.63 (−0.10 to 1.00)	0.85	0.85, 0.85 (0.55-1.00)	0.87, 0.93 (0.55-1.00)	0.84, 0.84 (0.55-1.00)
ICC (mNY grades 0-8), mean, median (min-max)
Exercise I	0.72	0.35, 0.38 (0.01-0.76)	0.35, 0.36 (0.01-0.59)	0.35, 0.39 (0.05-0.76)	0.89	0.73, 0.77 (0.22-0.90)	0.65, 0.74 (0.22-0.90)	0.79, 0.77 (0.72-0.89)
Exercise II	0.87	0.57, 0.58 (0.31-0.86)	0.67, 0.65 (0.38-0.86)	0.50, 0.49 (0.31-0.65)	0.90	0.68, 0.67 (0.35-0.91)	0.69, 0.67 (0.47-0.91)	0.66, 0.70 (0.35-0.87)
Exercise III	0.79	0.47, 0.50 (−0.21 to 0.75)	0.46, 0.48 (0.10-0.74)	0.47, 0.50 (−0.21 to 0.75)	0.85	0.82, 0.84 0.54-0.97)	0.82, 0.80 (0.69-0.97)	0.82, 0.86 (0.54-0.93)

No. of cases in exercise I, II, and III are 15, 16, and 15, respectively, for the subgroup with no/mild radiographic damage; and 13, 16, and 14, respectively, for the subgroup with moderate radiographic damage.
↵^a Grade 3 bilaterally or at least unilateral grade 4 according to the RETIC developer (RGL) were excluded.
↵^b Grade 0 and unilateral and bilateral grade 4 according to the RETIC developer (RGL) were excluded.
↵^c Numbers represent the agreement in percentage exact agreement, , and ICC between the 2 RETIC developers.
↵^d RETIC module done between exercises I and II.
↵^e RETIC module done between exercises I and III. ICC: intraclass correlation coefficient; max: maximum; min: minimum; mNY: modified New York; RETIC: real-time iterative calibration.