Abstract
Objective. To assess the expanded/refined first-time participant training program.
Methods. We conducted a refined new participant program at OMERACT 2018 on days 1–4, in which first-time participants provided feedback with online surveys and a nominal group on Day 4.
Results. Twenty first-time participants attended the introductory session and 8–12 attended followup sessions. A high proportion valued the newbie session (100%), rating it overall (91%), content-wise (62%), for presentation quality (82%), and value for the money (82%) as outstanding or good. The nominal group technique identified opportunities for further improvement of breakouts/voting.
Conclusion. The expanded new participant training program is valued by attendees.
The Outcome Measures in Rheumatology (OMERACT) develops validated outcome measures for clinical trials in rheumatic diseases. The success of OMERACT is due in large part to its adoption of rigorous methodology in the development of the core set for a disease of interest, and completing this process by achieving consensus1,2. The original OMERACT filter for a core measure consists of truth, discrimination, and feasibility (TDF)3. More recently, the OMERACT filter 2.0, i.e., the process for developing core domains for a condition or disease, was defined and described2, which evolved to filter 2.1 before the OMERACT 2018 meeting1.
Endorsement of a core domain or core measurement set for a condition occurs at OMERACT face-to-face meetings in 3 steps: (1) presentation of data supporting the validity of domain/measure by the working group (30 min); (2) discussion of evidence in detail in small group breakouts to achieve consensus (1.5 h); and (3) a report back from each breakout session, clarification of concerns raised during the breakout sessions by working groups, and an open discussion, followed by voting that results in endorsement or no endorsement of each domain/measure (30 min). Successful participation in the consensus process requires methodological knowledge/training about the actual OMERACT process (briefly described above).
Most first-time OMERACT participants are not primarily study methodologists, and given the new vocabulary, the OMERACT process, and the different types of sessions, the learning curve can be steep. At each face-to-face meeting, about 1 in 5 OMERACT attendees have no prior OMERACT experience. After the 2012 meeting, there were requests for more education on OMERACT for first-time participants prior to the next OMERACT meeting. The OMERACT executive agreed on the importance of such training, because all participants’ votes carry equal weight, so that better education would lead to more informed voting.
In 2014, the first formal OMERACT new participant training was implemented4. The results of this training showed an improvement in participants’ involvement and understanding of the meeting. This was repeated in 2016, with similar results, with the addition of more concrete suggestions for improvement of both the training and the OMERACT meeting process5. Our objective was to describe the process and results from the refined and expanded new participant session at OMERACT 2018, also referred to as the 2018 OMERACT newbie program. This program had a renewed focus on the new participant experience before, during, and after the meeting, in addition to methodology training more by discussion rather than didactic teaching.
MATERIALS AND METHODS
OMERACT first-time participant program
A first-time participant program was conducted at OMERACT 2018, consisting of a 1-h introductory session on the morning of Day 1, followed by a 1.5-hour question and answer session. There were 1-h evening sessions on days 1–4. Before the meeting, first-time participants were provided with premeeting reading materials and invitations to the sessions. The OMERACT premeeting reading materials provided by each working group contain a summary of the work completed in the 2 or more years prior to the meeting, including key evidence supporting the validity of core domains/measures. First-time participants were reminded of these meetings at plenary sessions and a schedule was provided at registration.
Introductory session format
Prior to any meetings, all first-time participants completed an anonymous survey assessing their familiarity with OMERACT, including the concepts of the OMERACT filters and the structure of the meeting. At the first session, there were presentations on OMERACT’s history and process, followed by open discussion and a question and answer session.
Daily evening session format
Discussions and debriefings in the 2018 OMERACT first-time participant program focused on the new participant experience before, during, and after the OMERACT meeting. Prior to each evening session, participants completed an anonymous online survey reassessing their comfort with the OMERACT structure and concepts. Two moderators staffed each 1-hour session (JAS, VSS). This was followed by a debriefing6,7 during each evening session, in which each participant was asked to list the “best” and “worst” experience of the day. The intent of these open-ended questions was to gauge the participants’ understanding of the process and to obtain their perspectives as new participants on what they felt went especially well (or poorly). These responses were recorded and analyzed by the session coordinator (SG). A post-meeting survey regarding the first-time participant experience itself was sent to all new participants. The session coordinator collected the responses.
During the last day of the meeting, we reviewed the most frequently cited concern(s) expressed by first-time participants from each day, which were confirmed by the group. Based on this, we used the nominal group technique (NGT) to ask 1 question to assess this concern: “What improvements do you think can make the breakouts and voting process at OMERACT more productive?” The NGT is a variant of traditional methods aimed at identifying the overall opinion of a group. The NGT is a structured process that facilitates development of an inclusive list of issues related to a specific question followed by feedback on the relative importance of these lists through rank-ordering6,7,8. The NGT approach promotes even more participation rates compared to focus groups, with equal weighting of input from all participants. An experienced NGT facilitator (JAS) first presented the questions, followed by participants noting their responses independently and reading each response to the facilitator, who recorded them verbatim. After all the responses had been presented by the participants in a round-robin fashion and listed by the facilitator (JAS), participants were asked to elaborate and discuss their responses. After discussion, none of the responses could be consolidated. Participants chose their top 5 responses and gave them individual rank scores from 1 to 3 (higher score indicating the top choice). Scores were aggregated for group rank-order of these questions, with higher scores representing the highest ranked questions.
First-time participant post-conference feedback
OMERACT first-time participants were asked about their experience in the OMERACT first-time participant session, including questions about the importance of a dedicated newbie session, the content and quality of the presentations and facilitation, the overall value of the OMERACT meeting, and whether they planned to attend OMERACT 2020 (to be held in the United States). There were free-text boxes for “What is the best thing about the first-time participant program?” and “What could be improved about the program?”
RESULTS
Participant characteristics and familiarity and comfort with the OMERACT process
There were 24 first-time participants. Characteristics of the new participants are shown in Appendix 1. Twenty first-time participants attended the introductory session and 8–12 attended each followup session on days 1–4. Eleven (46%) completed the post-meeting survey.
There was a high level of understanding of OMERACT at baseline. In 2018, by the end of Day 1, 100% said they felt extremely welcome at the OMERACT sessions, 88% felt that they could contribute during the breakout sessions, 86% understood the TDF (truth, discrimination, feasibility) OMERACT filter somewhat or extremely well, and 72% had some or extremely good understanding of the different types of OMERACT sessions. These proportions stayed stable or improved slightly through the entire duration of OMERACT. Compared to OMERACT Day 1 evening meetings in 2014 and 2016, a higher proportion reported understanding of the TDF filter in 2018 (12% vs 22% vs 43%, respectively).
First-time OMERACT participant experience
Free-text responses on the positive aspects of the newbie session included the fact that the session exists at all, that the session was structured as a small group with open discussion, the introduction to OMERACT concepts and processes and the ability to ask questions, and good/great facilitation of the session.
Areas for improvement included requests for video examples of the different types of OMERACT sessions, more premeeting materials on outcome measure development, a premeeting Webinar, more structure in the program, and improved facilitation.
Post-OMERACT 2018 survey
All responders felt that it was extremely important or important to have a dedicated newbie session. Almost all (91%) felt that the newbies sessions were either outstanding or good, and most (62%) felt that the content was either outstanding or good. Most (82%) felt that the quality of the presentations was outstanding or good (Table 1). While 9% felt that “way too much” time was devoted to the sessions, 64% felt that “a little too much” time was devoted, and 27% felt the time devoted was “just right,” suggesting that minor adjustments to timing should be considered. No participants rated any aspect of the sessions as bad or unacceptable. Despite the time concerns, 82% felt that the value for money of the newbie meeting was outstanding or good, and 18% thought it to be just right, and most (82%) plan to attend the OMERACT 2020 meeting.
First-time participants’ post-OMERACT conference feedback survey regarding the 2018 OMERACT newbie session (n = 11).
Nominal group
At the end of Day 4, we conducted an NGT with 8 participants addressing 1 question. New participants identified and ranked their responses to the question as to how to make the OMERACT breakouts and voting process more productive (Table 2). The top 4 ranked themes that garnered 58% of the votes (28 of the 48) were the following:
Improved facilitation during the breakouts (15 votes)
The need for more discussion time in general meeting room after breakout sessions for comments from the team/each breakout to be addressed by the working group (5 votes)
Tough structure for the voting process (4 votes)
Encouraging participation in the breakout sessions (4 votes)
2018 OMERACT First-time participant Nominal Group Technique (n = 8) addressing the question “What improvements do you think can make the breakouts and voting process at OMERACT more productive?”
DISCUSSION
Our report describes results of the third formal OMERACT first-time participant training program, results of the post-meeting survey, and innovative solutions to problematic aspects of the new-attendee experience as seen by those with limited OMERACT experience. At OMERACT 2018, baseline understanding and comfort levels of participants were high. A much higher proportion of first-time participants reported understanding the OMERACT TDF filter extremely well, compared to those in 2014 and 2016 (12% vs 22% vs 43%), which supports the expanded/refined program with a focus on the new participant experience before (pre-meeting readings), during, and after the meeting. The OMERACT 2018 newbie program included an updated shorter presentation of the OMERACT process, a detailed open-forum discussion on the OMERACT TDF filter and the OMERACT filter 2.0, a shift in focus to that of the first-time participants’ own experience, the identification of best and worst experiences of this group during the daily evening sessions, and an elaboration of innovative solutions to the areas for improvement. To continue to improve the experience of OMERACT first-time participants, and thereby encourage their continued involvement in OMERACT, their solutions will be provided to OMERACT’s Executive Committee for possible implementation.
The nominal group findings provide insight and present suggestions for future consideration and improvements. These findings should be interpreted with caution, because these opinions and observations come from a small subgroup of all people attending the OMERACT meeting — the first-time participants. Low response rate for the final survey (46%) limits the generalizability of survey findings.
While OMERACT first-time participants generally had positive opinions of the in-meeting sessions, they continue to feel that more premeeting preparation would have been of benefit. There was a specifically identified need for more preparation regarding the central OMERACT process of outcome measure development, as well as video examples of the different types of OMERACT sessions.
APPENDIX 1
Participant characteristics for the 2018 OMERACT first-time participant program, as compared to similar sessions at OMERACT 2014 and 2016.
Footnotes
VSS is an employee of UCB Biosciences, Inc. JAS has received consultant fees from Crealta/Horizon, Fidia, UBM LLC, Medscape, WebMD, the National Institutes of Health and the American College of Rheumatology. JAS is a member of the Veterans Affairs Rheumatology Field Advisory Committee. JAS is the editor and the director of the UAB Cochrane Musculoskeletal Group Satellite Center on Network Meta-analysis. JAS served as a member of the American College of Rheumatology’s (ACR) Annual Meeting Planning Committee (AMPC) and Quality of Care Committees, the Chair of the ACR Meet-the-Professor, Workshop and Study Group Subcommittee and the co-chair of the ACR Criteria and Response Criteria subcommittee. JAS is a member of the executive of OMERACT, an organization that develops outcome measures in rheumatology and receives arms-length funding from 36 companies.
- Accepted for publication November 14, 2018.
