Abstract
Objective. To develop and validate a knowledge transfer (KT) module aimed at enhancing feasibility and reliability of semiquantitative assessment of bone marrow lesions (BML) and synovitis-effusion using the Hip Inflammation Magnetic Resonance Imaging Scoring System (HIMRISS).
Methods. Three radiologists naive to the HIMRISS method reviewed the manuscript describing the method and then scored MRI scans from 16 patients with hip OA obtained at baseline and 8 weeks after intraarticular injection of corticosteroid. Readers then reviewed a KT module comprising an instructional presentation and 8 reference DICOM (digital imaging and communications in medicine) cases scored by 3 readers with expertise in the HIMRISS method, and then used electronic overlay software to score scans from 23 patients with OA. The same format was followed with a second group of 3 readers naive to HIMRISS using a KT module revised to incorporate the overlay with a Web-based DICOM viewer to enhance feasibility. Interobserver reliability was assessed with the intraclass correlation coefficient (ICC).
Results. In both exercises, reliability for baseline scores was excellent for femoral BML, very good for acetabular BML, and good for synovitis-effusion (overall ICC = 0.91, 0.89, 0.62, respectively) even without prior calibration using the KT module. However, reliability for detecting change was substantially worse than for expert readers, especially for acetabular BML and synovitis-effusion (overall ICC = 0.59 vs 0.19, and 0.42 vs 0.25, respectively). Reliability improved for detection of change in these lesions, especially after reader calibration with the revised KT module.
Conclusion. Development and validation of a systematic method for KT may enhance external validation of certain imaging instruments.
Hip osteoarthritis (OA) is a major international health problem, with 200,000 joint replacements annually in the United States alone. Advances in magnetic resonance imaging (MRI) include the identification of important features of disease such as bone marrow lesions (BML) and synovitis. These have provided new insights into the factors that might predict the development of cartilage degeneration. In particular, development and validation of scoring systems that allow semiquantitative assessment of BML and synovitis on MRI have allowed prognostic assessment in longitudinal studies and shown that such lesions are associated with subsequent development of cartilage degeneration in OA of the knee1,2,3. The degree of BML and synovitis has also been shown to correlate with pain severity4,5. Establishment of surrogate status of MRI for structural damage in OA may substantially enhance the feasibility of new clinical trials of potential disease-modifying agents.
While MRI is used extensively clinically to assess hip joints, and similar abnormalities (e.g., BML) are frequently observed in clinical practice, methods for assessment of BML are limited to semiquantitative assessment of different sectors of the femur and acetabulum. In the original description of the Hip Inflammation MRI Scoring System (HIMRISS), BML are assessed on coronal scans using a transparent overlay comprising a circle divided into octants6. The MRI scan is then sized so that the circle fits the largest diameter of the femoral head. The presence/absence of BML in each of the octants is scored dichotomously in 5 consecutive coronal slices. The overlay is also used to score BML in the acetabulum, which is assessed in 3 sectors. In the Hip OA MRI Scoring System (HOAMS), lesions are assessed on both coronal (5 sectors for the femoral head, 3 sectors for the acetabulum) and sagittal (4 sectors for femoral head, 3 sectors for acetabulum) scans7. Synovitis is scored in 4 locations in HOAMS and 2 locations in HIMRISS according to a grading scheme that reflects thickness of synovitis-effusion.
Although preliminary assessment of these instruments by the respective development teams has shown high reliability for status and change scores6, the complicated subdivision of the hip into many sectors may compromise feasibility and reliability in the hands of inexperienced readers. In particular, assessment of lesions on consecutive slices is difficult using a fixed overlay on a transparency because of shifts in anatomical boundaries during review of consecutive slices. The MRI in hip OA special interest group has focused on the development and validation according to the OMERACT Filter of a knowledge transfer (KT) module aimed at enhancing feasibility and reliability of semiquantitative assessment of BML and synovitis-effusion using the HIMRISS method. This was especially targeted at the creation of electronic Web-based overlays that dispense with the need for transparencies, allow interactive resizing of the overlay, and allow assessment of consecutive slices without the need to reposition the overlay. Because most imaging-based scoring instruments in rheumatology use sectoral approaches to quantitation, we anticipate that this process could also help guide the development and external validation of KT modules for other imaging instruments.
MATERIALS AND METHODS
Development of the HIMRISS KT Module
The HIMRISS KT module comprises the following 3 sections.
1. Instructional PowerPoint presentation (85 slides; available from www.carearthritis.com/himriss)
This presentation describes the detailed methodology for scoring BML and synovitis using the HIMRISS method and incorporates revisions from exercises conducted in preparation for OMERACT 116. The presentation is divided into sections focusing on slice selection, correct positioning of the electronic overlay, detection of BML, and definition of abnormal signal on fluid sensitive sequences; as well as pitfalls in scoring of BML; detection, grading, and pitfalls in the scoring of synovitis; and recording of scores in the customized Excel spreadsheet. Images with appropriate scores for BML and synovitis are included.
2. Reference DICOM (digital imaging and communications in medicine) cases
Three musculoskeletal radiologists (readers 1, 2, 3) experienced in the HIMRISS method independently scored right and left hip MRI scans from 8 patients with hip OA obtained at baseline and 8 weeks after intraarticular (IA) injection with corticosteroid. Discrepancies in scores for each data cell (total of 2080 data cells) were reviewed and final scores determined by consensus. These cases are made available as a training set to readers unfamiliar with the HIMRISS method. An interactive Excel spreadsheet was also developed that automatically highlights scoring discrepancies compared to the expert readers.
3. Web-based overlay for HIMRISS
A Web-based overlay integrated with a Web-based DICOM viewer was designed to be used with DICOM MR images of the hip for direct Web-based evaluation according to the HIMRISS method (available from www.carearthritis.com/himriss). Using a mouse and drag and drop, the overlay can be positioned over the femoral head on the central slice and resized to ensure alignment with the subchondral bone. In addition, pretreatment and posttreatment scans can be displayed and scored simultaneously.
Patients and Reading Exercises
Reading exercises
The study was conducted in 2 sets of reading exercises (Figure 1). In reading exercise one, 3 musculoskeletal radiologists with no prior knowledge of the HIMRISS method (readers 4, 5, 6) were asked to review the manuscript describing the HIMRISS method and then to score hip MRI scans from 16 patients with hip OA without being provided any further information regarding the HIMRISS method. Scans were obtained at baseline and 8 weeks after an IA injection of corticosteroid and were scored blinded to timepoint. Readers were then asked to review the instructional presentation describing the HIMRISS method in detail, to score the 8 reference DICOM cases, and review discrepancies with expert reader scores. Readers were then provided with instructions on the use of software for an electronic overlay (http://vitrite.en.softonic.com/ for Windows and http://www.colinthomas.com/overlay/ for Macintosh) that can be used with any DICOM viewing software. After training with the KT module, readers were asked to conduct a second reading of scans from 23 patients with OA. Scans were obtained at baseline and 8 weeks after IA injection of corticosteroid and were scored blinded to timepoint. This reading included 16 paired scans from the precalibration reading that were recoded and randomly incorporated into the reading list according to a nested imaging study design. The purpose of this nested imaging study design was to evaluate improvement in reliability. This would ensure that variability in difficulty level of the cases would not confound an expected improvement in the reliability of readers. The 3 readers experienced in the HIMRISS method conducted exactly the same scoring exercise to allow comparisons with naive reader scores.
Study design for development and validation of a knowledge transfer module for the Hip MRI Scoring System (HIMRISS). MRI: magnetic resonance imaging; DICOM: digital imaging and communications in medicine; PPT: PowerPoint.
Feasibility was reported as a substantial problem by the 3 readers in exercise 1 because of the extended time needed to use the overlay software to apply and position the electronic overlay over the femoral head on the DICOM MR image of the hip. The KT module was therefore modified for exercise 2 by dispensing entirely with the overlay software (HIMRISS KTv2). Instead, we created a Web-based DICOM viewing interface that incorporated an electronic overlay so that both the DICOM images and the overlay could be viewed simultaneously on the same Web page as shown at www.carearthritis.com/himriss/. In exercise 2, the effect of this innovation on feasibility and reliability of the HIMRISS method was tested using 3 naive readers (readers 7, 8, 9) different from those in exercise 1. These readers first assessed the same 16 cases with paired MRI scans after review of the HIMRISS manuscript alone, followed by the assessment of all 23 cases with paired MRI scans after exposure to HIMRISS KTv2.
Patients
All patients were aged > 50 years and diagnosed with OA by their physician, who referred them to the radiology department where they either received imaging-guided corticosteroid injection into one or both hips as a component of routine care (n = 15) or were recruited into a placebo-controlled trial of IA injection of corticosteroid administered under imaging guidance into the hip (n = 8) as reported8. MRI was conducted at baseline and at the 8-week primary endpoint. All patients were ambulatory at the time of imaging and had chronic hip pain without a history of trauma within the last 6 months. Patients were excluded if they had a history of rheumatoid arthritis, other inflammatory rheumatic conditions, or recent trauma. Patients had a mean (SD) age of 59.6 years (13.8), 12 were males (52.2%), and mean (SD) Kellgren-Lawrence grade was 2.5 (1.1). The local institutional review board (University of Alberta Health Research Ethics Board) approved these study designs, and all patients provided written informed consent.
MRI Acquisition
MRI scans included coronal short-tau inversion recovery (STIR; TR 3550, TE 51, TI 145 ms) and T1-weighted spin echo (TR 475, TE 16 ms). Slice thickness/gap were 4/0.4 mm, field of view 40 cm, matrix size 512 × 256 (STIR) or 512 × 307 (T1).
Statistics
We assessed status and change scores in HIMRISS BML and synovitis-effusion descriptively according to mean and SD. Interreader reliability among expert and naive readers was evaluated using analysis of variance to provide the intraclass correlation coefficient (ICC) for status and 8-week change scores according to reader pairs as well as at the group level. A 2-way mixed effects model with observer as a fixed factor was used. ICC > 0.5, > 0.6, > 0.8, and > 0.9 were regarded to represent moderate, good, very good, and excellent reliability, respectively. Status and 8-week change scores were compared before and after calibration with the HIMRISS KT module v1 and v2 in exercises 1 and 2, respectively. A substantial change in ICC from precalibration to postcalibration was prespecified as ≥ 0.10.
RESULTS
At a group level there was relatively little change in expert reader scores for BML and synovitis-effusion (Table 1) although cumulative probability plots demonstrated that about half the patients had change in femoral BML score, which was distributed equally between increased and decreased scores (Figure 2). There were somewhat more patients with increased scores for BML and synovitis-effusion at the 8-week timepoint after IA administration of corticosteroid. Mean reader time per scan was 15 min and 12.3 min before and after calibration with the first version of the KT module, respectively. Mean reader time per scan was 15.8 min and 11 min before and after calibration with the HIMRISS KTv2 module, respectively.
Cumulative probability plots demonstrating change in expert reader scores for BML and synovitis-effusion in patients with hip OA between baseline and 8 weeks after receiving an injection with intraarticular steroid. BML: bone marrow lesion.
HIMRISS scores at baseline and 8 weeks for BML and synovitis-effusion recorded by 3 readers experienced in the HIMRISS method in 23 patients with osteoarthritis receiving intraarticular injection with corticosteroid.
For the 16 cases assessed by the 3 readers experienced in the HIMRISS method, interobserver reliability for assessment of femoral and acetabular BML at baseline was very good to excellent for all reader pairs, while assessment of synovitis-effusion was variable and less reliable [overall ICC (95% CI) = 0.63 (0.24–0.82); Table 2]. Interobserver reliability for assessment of 8-week change in femoral BML was very good [overall ICC (95%) = 0.82 (0.70–0.90)], moderate for acetabular BML [overall ICC (95% CI) = 0.57 (0.37–0.74)], and poor for synovitis-effusion [overall ICC (95% CI) = 0.45 (0.23–0.65)]. Reliability was similar for all 23 cases although ICC was higher for change in femoral BML [overall ICC (95% CI) = 0.90 (0.84–0.94); Table 2].
Interobserver reliability [ICC (95% CI)] for 3 readers experienced in the HIMRISS method (readers 1, 2, 3) scoring BML and synovitis-effusion according to the HIMRISS method on STIR MRI scans from patients with hip OA obtained at baseline and 8 weeks after receiving intraarticular steroid.
In exercise 1, which included the first group of 3 readers naive to the HIMRISS method, interobserver reliability for baseline scores was noted to be excellent for femoral BML [overall ICC (95% CI) = 0.91 (0.85–0.95)], very good for acetabular BML [overall ICC (95% CI) = 0.89 (0.82–0.94)], and good for synovitis-effusion [overall ICC (95%) = 0.62 (0.41–0.79); Table 3]. Paired reader scores were comparable to expert reader scores. However, reliability for detecting change was substantially worse [overall ICC (95%) for change in femoral BML = 0.62 (0.48–0.76), acetabular BML = 0.19 (0.08–0.38), synovitis-effusion = 0.25 (0.12–0.43)]. After calibration with the first version of the KT module, reliability for status scores did not change, but there was substantial enhancement in reliability for detecting change in acetabular BML [overall ICC (95% CI) = 0.29 [0.18–0.44)] and synovitis-effusion [overall ICC (95% CI) = 0.35 (0.23–0.49)]. Nevertheless, reliability was less than that recorded for expert readers. Feedback from naive readers reported that feasibility needed to be improved for application of the overlay and that the overlay software was cumbersome, difficult to apply in a time-efficient manner, and would work far more efficiently if a way could be found to integrate the overlay into a DICOM viewing software program.
Interobserver reliability [ICC (95% CI)] for 3 readers (readers 4, 5, and 6) naive to the HIMRISS method scoring BML and synovitis-effusion before and after calibration with the HIMRISS KT module v1 on STIR MRI scans from patients with hip OA obtained at baseline and 8 weeks after receiving intraarticular steroid.
Exercise 2 was conducted after the design of a Web-based DICOM viewer that integrated the electronic overlay into the software program (HIMRISS KT v2) and all scans were viewed and assessed online. The exercise included a second group of 3 readers naive to the HIMRISS method. In this group, interobserver reliability for baseline scores was noted to be very good for femoral BML [overall ICC (95% CI) = 0.86 (0.78–0.92)], excellent for acetabular BML [overall ICC (95% CI) = 0.91 (0.86–0.95)] and good for synovitis-effusion [overall ICC (95%) = 0.70 (0.50–0.83); Table 4]. Paired reader scores were comparable to expert reader scores. However, as was the case for exercise 1 readers, reliability was substantially less for detection of change scores [overall ICC (95%) for change in femoral BML = 0.60 (0.46–0.74), acetabular BML = 0.13 (0.02–0.29), synovitis-effusion = 0.38 (0.22–0.56)]. After calibration with the HIMRISS KTv2 module, reliability for status scores did not change, but there was substantial enhancement in reliability for detecting change in acetabular BML [overall ICC (95%) = 0.43 (0.31–0.58)] and synovitis-effusion [overall ICC (95%) = 0.45 (0.31–0.60)], which approached the reliability observed with expert readers, especially for detection of change in synovitis-effusion. In addition, these readers expressed considerable satisfaction with the feasibility of this innovation.
Interobserver reliability [ICC (95% CI)] for 3 readers (readers 7, 8, and 9) naive to the HIMRISS method scoring BML and synovitis-effusion before and after exposure to the HIMRISS KT module v2 on STIR MRI scans from patients with hip OA obtained at baseline and 8 weeks after receiving intraarticular steroid.
DISCUSSION
We have demonstrated in 2 reading exercises and 6 readers naive to the HIMRISS method that BML on MRI can be reliably detected in both femoral head and acetabulum, even in the absence of any calibration beyond the review of the published manuscript describing the HIMRISS method. Detection of change in BML, however, is more difficult, especially for acetabular BML, and can be enhanced by the use of our KT module, which describes the scoring methodology, provides reference DICOM cases with examples of the spectrum of abnormalities in hip OA observed on MRI, and provides an electronic overlay with a DICOM viewer in a customized HTML Web browser environment to facilitate scoring based on defined sectors in an anatomically complex joint. Reliable detection of synovitis-effusion with semiquantitative methods is challenging even for expert readers, and may also be enhanced with our KT module. Feasibility is a relatively overlooked aspect of development of image-based scoring instruments, and should be a primary consideration for external validation of these instruments in future.
We previously demonstrated that BML in the femoral head and change in these lesions after IA administration with corticosteroid can be reliably detected using either the HOAMS or the HIMRISS methodologies for scoring BML6. This is not surprising in view of the morphological symmetry of the femoral head and consistent fatty marrow signal in healthy subjects, although scoring can be challenging in advanced OA when the head may become deformed. In contrast, reliable detection of change in acetabular BML is more challenging owing to the complicated anatomy of this region, the unpredictable variation in distribution of erythropoietic marrow in the pelvis, and the limited change in BML noted after administration of IA corticosteroid. Reliable detection of synovitis-effusion has limitations, as noted6, and reflects both the limited change noted at 8 weeks in response to IA administration of corticosteroid, as well as challenges in scoring according to a limited grading scheme.
Several considerations that could improve the reliable detection of synovitis-effusion were raised during these reading exercises. First, clearly defining the extent to which the femoral neck is visible to measure synovitis-effusion perpendicular to it; second, raising the cutoff from 2 mm to 3 mm before synovitis-effusion is considered abnormal; third, potentially dispensing with the 3-part grading scheme in favor of a simpler, dichotomous scoring framework (abnormal synovitis-effusion yes/no), although this may in turn limit discrimination; and fourth, automated quantitation as described for knee OA should be evaluated in the hip.
We noted that the effects of training with the KT module varied according to the reader. A major consideration for all readers was feasibility, especially the effect on the time required to review the DICOM reference cases. Most readers did not review all the DICOM cases in the KT module and it was suggested that the identification of discrepancies be expedited in real time through an iterative process during the reading exercise. While this was partially addressed through our development of an Excel spreadsheet that identified discrepancies by means of color-coding of discrepant cells, we anticipate this can be further expedited during a process of direct scoring of lesions on a Web interface that presents the MRI scan together with the overlay. The integration of the electronic overlay with a DICOM viewer in a customized HTML Web browser environment in the revised KT module was considered a substantial advance in feasibility by the second group of readers. Consequently, a major objective for further development and validation of the KT module in future OMERACT exercises will be to enable readers to score lesions online directly on the DICOM image, thereby dispensing entirely with spreadsheets or any other type of scoring sheet. This approach lends itself to the HIMRISS method, which scores lesions in different sectors on a dichotomous basis (present/absent).
We have developed and conducted preliminary validation of a KT module aimed at enhancing reliability of detection of BML and synovitis-effusion on STIR MRI in the hip joints of patients with OA using the HIMRISS method. We demonstrated in 2 reading exercises with 6 readers naive to the HIMRISS method that this KT module particularly enhances detection of lesions in the most anatomically complex region of the joint, the acetabulum. Our developmental process culminated in the innovation of an electronic overlay integrated with a DICOM viewer in a customized HTML Web browser environment. We anticipate that this will lead to further innovation enhancing the external validity of semiquantitative scoring methods based on a framework similar to the HIMRISS method, that is, direct scoring of lesions in different sectors of a joint through a Web interface that presents the DICOM image together with the overlay.
Acknowledgment
We thank the Alberta Osteoarthritis team for providing funds to support the acquisition of images.
In this issue
