Abstract
Objective. To determine whether an educational intervention could result in changes in physicians’ practice behavior.
Methods. Twenty rheumatologists performed a prospective chart audit of 50 consecutive patients with rheumatoid arthritis (RA) and again after 6 months. Ten were randomized to the educational intervention: monthly Web-based conferences on the value of systematic assessments in RA, recent evidence-based information, practice efficiency, and other topics; this group also read articles on targeting care in RA. The others were randomized to no intervention.
Results. One thousand serial RA charts were audited at baseline and 1000 at 6 months, with no between-group differences in patient characteristics: mean disease duration of 10 years; 77% women; 74% rheumatoid factor– positive; mean Disease Activity Score (DAS) 3.7; and 68% taking methotrexate, 14% taking steroids, and 27% taking biologics. At 6 months the intervention group collected more global assessments (patient global 53% preintervention vs 66% postintervention, and MD global 51% vs 60%; p < 0.05) and Health Assessment Questionnaires (37% vs 42%; p > 0.05; p = nonsignificant), whereas controls had no change in outcomes collected. For the intervention group there was a 32% increase in calculable composite scores [such as DAS, Simplified Disease Activity Index (SDAI), Clinical Disease Activity Index; p < 0.05] but no change in the controls. There was more targeting to a low disease state. For those with SDAI between 3.3 and 11, the percentage of patients receiving a change in therapy was 66% in the intervention group and 36% in controls (p < 0.05). When DAS was between 2.4 and 3.6, 57% of the intervention group and 38% of controls made changes to treatment (p < 0.05).
Conclusion. Small-group learning with feedback from practice audits is an inexpensive way to improve outcomes in RA.
- EDUCATION
- KNOWLEDGE TRANSLATION EXCHANGE
- RHEUMATOID ARTHRITIS
- CHART AUDIT
- SMALL GROUP LEARNING
- COMPARATIVE FEEDBACK
Multiple studies have demonstrated that treating to a target in rheumatoid arthritis (RA) leads to more patients reaching that target and thus is likely to translate into better care1,2,3,4,5,6,7,8,9. However, to change clinical practice behavior is a challenge. Practice guidelines, for instance, are followed only about half the time10,11. The reason for this may be a knowledge gap, or more likely a gap between knowledge and behavioral change. We devised a randomized trial to determine whether comparative feedback from chart audits and targeted small-group learning (especially targeted toward breaking down barriers to good care) would alter behavior, compared with solely performing chart audits without feedback.
We combined an educational small-group learning program with a chart audit that compared an individual’s practice to other rheumatologists to see whether this would change behavior in the assessment and management of RA. A review by Jamtvedt, et al has shown that audit and feedback intervention approaches can be effective in improving patient care and adherence to guidelines, especially if original adherence to guidelines is minimal and intensive feedback is provided, but the effects may be small to moderate12. Another review concluded that reminders, patient-mediated interventions, outreach visits, opinion leaders, and multifaceted activities could change physician behavior more effectively than traditional continuing medical education (CME) events, but of the interventions, chart audit with feedback and educational materials were less effective13. However, the educational materials in the studies quoted were usually not small-group learning based on a needs assessment. Opinion leaders providing education (academic detailing) was found to be more effective than a chart audit and feedback when studying a trial of vaginal delivery after cesarean birth14. Part of our current study was to provide evidence-based small-group learning, with opinion leaders directing the discussion and preparing the materials.
The objectives were (1) to document current practice patterns, including frequency of measurements performed and proportion of patients with RA treated to a low disease state; (2) to assess the effect of an educational intervention with chart audit comparative feedback on physician behavior/practice patterns; and (3) to pilot a needs-based intervention strategy that could be subsequently implemented in other centers.
MATERIALS AND METHODS
The Metrix study was an investigator-initiated pilot study in which consenting rheumatologists in Ontario were randomized to either an institutional review board-approved, accredited educational intervention over 6 months, or no intervention.
Rheumatologists who belonged to the Ontario Rheumatology Association were approached by e-mail to determine whether they were willing to participate in our study. There were 10 rheumatologists per group, with 3 men in the intervention group and 6 in the control group; the mean year of graduation from medical school was 1990 (range 1976–2001) for the intervention and 1989 (1976–2001) in the control group. Twenty percent in the intervention group and 50% in the control group were in academic practice. The program was accredited and a modest sum was paid for the chart audit completion. The rheumatologists received their CME credits and remained revenue-neutral for their time to perform the chart audits.
A case report form was developed to extract data from each chart of patients with RA seen serially in each practice. The 1-page form asked about the following: demographics of each patient (age, sex, disease duration, rheumatoid factor status, radiographic damage); current and previous use of disease-modifying antirheumatic drugs (DMARD), biologics, and steroids; musculoskeletal measurements (and the values) that were done at the current visit including tender and swollen joint counts, MD and patient global assessments, Health Assessment Questionnaire-Disability Index (HAQ-DI)15, inflammatory markers, and other measurements. Additionally, the form described what changes were made, if any, to treatment, including changes in DMARD, biologics, and steroids (including intraarticular injections). The intervention group received aggregate results from the chart audit comparing their practice to the others within that group, and no feedback was given in the control group. The rheumatologists who ran the sessions (JP, CT, and AC) were blinded to which doctor had which practice profile, to make the study nonthreatening to the participants.
Intervention group
A needs-based survey was performed to determine gaps in knowledge and barriers to good care, which assisted in the development of the program.
After completion of an initial chart audit, the rheumatologists were randomized to either no intervention (control group) or the educational intervention (intervention group). The randomization using a random number table was done by the central coordinator (KL), who was unaware of chart audit scores. The educational intervention included feedback from the chart audit, comparing their practice to the aggregate in the intervention group. There was also a series of monthly 1- to 1.5-h Web seminars with topics such as the needs assessment, benefits of measuring what physicians do and what measurements could be performed, success of tight control with current data from articles and abstracts on American College of Rheumatology and European League Against Rheumatism guidelines on treating to a target in RA, barriers to care and reasons for them (including lack of physicians, suggesting modifications to treatment), office efficiencies to lessen barriers to care (lack of time was identified as a barrier to targeted treatment), and practices (including what forms the participants were using in their offices). In addition, a journal club involved reading articles on targeting care in RA. Each session allocated half the time to the topic and the remaining to discussion. Each session was attended by at least 1 moderator (JP, CT, AC). Participants who missed an educational session were required to review the content and respond to an online questionnaire. Periodic evaluations and questionnaires about changing attitudes and behavior were done in the intervention group.
Upon completion, another chart audit of 50 consecutive patients with RA was done in each group. A final accredited investigators’ meeting was conducted for both groups to discuss the results of the practice audits and summarize the sessions with a workshop on the components that should be included in an RA followup form that could be modified for each practice. There was no independent verification in any practice to determine whether the data collection was accurate or to prove the data were collected on patients with RA seen consecutively.
After study completion, the control group received the educational intervention.
Analyses
Baseline characteristics of the patients in each group were compared using t-tests and chi-squared statistics. P values were significant if < 0.05. At completion of the study with the repeat chart audit, between-group differences were measured for both the proportion of patients that had components to calculate scores for the Disease Activity Score (DAS), Simplified Disease Activity Index (SDAI), and the Clinical Disease Activity Index (CDAI), and the proportion that had a change in RA treatment if not in a low disease state16,17.
The primary outcome was to determine whether more systematic assessments were done in at least 80% of each rheumatologist’s practice comparing the 2 groups, and whether at least 20% more often, treatment was intensified if patients were not in a low disease state.
At the end of the study, analyses were done of within-group differences between the chart audits of the number of musculoskeletal measurements taken, and the percentage of patients not in a low disease state who did not have a change in therapy in each group, to determine whether the chart audit had changed intervention behavior in the control group.
The coordinator (KL) formatted slides, arranged the Web seminars, scored the chart audits, and organized the poststudy investigators’ meeting. The entire project was evaluated by both groups.
Global ethics approval was obtained for this study.
RESULTS
Initially, 22 rheumatologists agreed to participate but after the first 20 completed the initial chart audit, they were randomly allocated using numbers generated by Excel to the intervention group (n = 10) and control group (n = 10).
There were 500 patients with RA per group for the baseline practice audit, each participating rheumatologist enrolling 50 consecutive patients (Figure 1). Only the intervention group was provided results of the chart audit, comparing their practice to the aggregate in the intervention group. Baseline characteristics of the 2 groups of patients with RA are shown in Table 1. The repeat practice audit was altered to add a question asking why a medication change was not made (such as patient nonacceptance, etc.). This addition was to gain feedback from the intervention group to adequately reflect not only what was done at the visit, but what may have been offered but not done, with justification. Most of the patients had longstanding RA with surprisingly low HAQ scores.
Patient characteristics from the repeat practice audits are shown, including comparisons between and within groups (Table 1, Figures 2 and 3). The proportion of rheumatologists performing outcomes at least 80% of the time both before and after chart audits was compared. Although the frequency of baseline measurements was fairly good in each group and many patients were in a low disease state, the intervention group changed behavior and the control group did not. They made more systematic measurements and made more changes in patients not in a low disease state. The proportion of calculable scores (any of CDAI, SDAI, or DAS) in the preeducation and posteducation groups that demonstrated significantly more calculable scores increased from 43% to 57% (p < 0.05) where DAS could be calculated 41% of the time preeducation and 45% (p > 0.05) post-education; CDAI increased from 18% to 43% (p < 0.001) and SDAI from 14% to 29% (p < 0.01). Mean scores decreased significantly for the 3 calculable scores. There were no differences in calculable scores for the control group (66% pre- and postintervention for any score), which were numerically higher than those of the intervention group; DAS also decreased. However, DAS was significantly lower postintervention in the education group compared to the control group [preintervention DAS was 3.72 in the education group and 3.68 in the control group (p > 0.05) and postintervention DAS was 3.05 in education vs 3.17 in control (p < 0.05)]. There were no other between-group differences pre- and postintervention for the CDAI and SDAI scores.
DISCUSSION
The simple participation in a structured chart audit likely resulted in a change in subsequent practice behavior, because all participants knew that patient encounters would be scored and shared; many commented that this led them to give more systematic care. Moreover, performing the initial 50 patient audits may have taught them what to do for the final chart audit, but this alone did not change behavior because the control group did not change.
A criticism could be that only keen participants who were already doing a good job took part, but even in this group a treatment effect could be found in both the proportion of measurements performed and acting on the results. Thus, even self-selected rheumatologists could change their behavior.
We are uncertain which component of the research exercise changed behavior: the comparative practice audit results, the small-group learning, or both. However, the control group did not change, indicating that a practice audit without feedback did not alter behavior. This would be consistent with previous critical analyses of CME and behavioral change13,14. Another limitation may be the feasibility of implementing a program like this in other jurisdictions. There was preparatory work for each Webcast, time and money to analyze the chart audits by a third party (KL), and compensation to the participants for the chart audit. Nonetheless, the effect size for improvement in making treatment changes is at least modest and could be compared favorably to treatment effects in traditional RA randomized controlled trials involving patients.
Additionally, not all data could be used for determining a change in treatment, because many of the patients did not have calculable scores. However, there was no other way to analyze the data, and because the patient characteristics were similar between the 2 groups, the change in behavior was noted only in the intervention group. Although we demonstrated that more scores could be calculated in the intervention group, we cannot prove that the scores were actually calculated and used for treatment decisions; we can only state that the components were collected. It may be that collecting measurements more frequently, even if scores are not determined, still results in more treatment changes in the patients who are not in remission. Also, there were baseline differences that were not significant in the proportion in which treatment changes were made for the patients with DAS between 2.4 and 3.6 (31% in intervention and 21% in the control group), but the final between-group comparison was significant. Numerically there is an absolute 26% increase in treatment changes in the intervention group (31% to 57%) and 17% in the control group (21% to 38%), resulting in 1.5 times more changes in the intervention group. If, however, the percentage changes in each group were compared (which was not our primary analysis), one could say that there was an 84% increase in the intervention group and an 81% increase in the control group. Therefore, both groups made more treatment changes, but there were numerically more in the intervention group.
More outcome measures were systematically performed only in the intervention group, with no change seen in controls. More scores could be calculated but only in the intervention group, and more patients’ changes were made (if not in a low disease state) in the intervention group only. The intervention group actually changed behavior.
Footnotes
-
Funded by an education grant from Abbott Canada.
- Accepted for publication August 27, 2012.
REFERENCES
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.