Abstract
Objective. To describe the experience of the first-time participant (newbie) training program at the Outcome Measures in Rheumatology (OMERACT) 2016 meeting.
Methods. We conducted new participant sessions at OMERACT 2016, including a 2-h introductory session on Day 1 followed by 1-h evening followup sessions on days 1–4. Pre- and post-meeting surveys assessed participants’ levels of comfort with the principles of the OMERACT Filter 2.0 (the essential tools for OMERACT methodology) and the different types of OMERACT sessions, and whether participants felt welcome. In addition, on the final day, a nominal group technique was used to elicit problematic components of the meeting and to develop solutions to those problems.
Results. Of the 43 new attendees, 38 participated in the introductory session and 14–18 attended the followup sessions. Comparing Day 1 (preintroductory session) to days 1–3 (post), a similar proportion understood different types of sessions extremely well [45% (pre) versus 47%, 44%, and 36% (post), respectively], and a higher proportion understood principles of the OMERACT filter extremely well [22% (pre) versus 55%, 44%, and 40% (post), respectively]. Most reported feeling welcome (86.7%) and felt they contributed to breakout sessions (93.3%) on the evening of Day 1; results were sustained on days 2–3. The most commonly reported “best” experience included the OMERACT culture and the most common reported experience needing improvement included facilitation issues during breakouts.
Conclusion. The first-time participants came to OMERACT 2016 with a high baseline level of understanding. They rapidly attained a high comfort level with participation and provided concrete and innovative solutions to the most commonly reported experiences needing improvement.
Outcomes Measures in Rheumatology (OMERACT)1 develops validated outcome measures for clinical trials in rheumatic diseases. Several key measures commonly used in clinical trials were developed by OMERACT, which in turn led to the application of evidence-based standards in rheumatology clinical trials. OMERACT participants include clinicians, specialists in outcomes research, clinical trialists, epidemiologists, patient research partners (PRP), and representatives from regulatory agencies and the pharmaceutical industry. The success of OMERACT is due in large part to its adoption of rigorous methodology in the development of the core set for a disease of interest, and achieving consensus from everyone involved2.
Most first-time OMERACT participants are not primarily study methodologists, and given the new vocabulary, the OMERACT process, and the different types of sessions, the learning curve can be steep. Filter 2.0 is the foundation used by OMERACT to assess candidate measures for use in trials3,4. At each face-to-face meeting, about 1 in 5 attendees have no prior OMERACT experience. After the 2012 meeting, there were requests for more education on OMERACT for new attendees before the next OMERACT. The OMERACT executive agreed on the importance of such training because all participants’ votes carry equal weight, and better education would lead to more informed voting.
In 2014, the first formal OMERACT new participant training was implemented5. The results of this training showed an improvement in participants’ involvement and understanding of the meeting5.
In 2016, we continued the OMERACT new participant training with some changes based on informal feedback from participants from the 2014 program. This report describes the process and results from the refined and expanded new participant session at OMERACT 2016, an overview of the new participants’ OMERACT experience, and concrete solutions for problem areas elaborated by the new attendees.
A novel contribution to the literature is the conduct and reporting of the nominal group technique (NGT) in the OMERACT first-time participant (newbie) program, which also provides the OMERACT executive with an actionable list of problem areas that might lead to improvement in OMERACT meetings. Our study was also innovative in that it evaluated the effect of a training program on new participants’ knowledge and skills as effective OMERACT meeting participants, and the change in the effectiveness of this program initiated at OMERACT 2014 and refined and expanded in 2016.
MATERIALS AND METHODS
OMERACT first-time participant program
A new attendee program was conducted at OMERACT 2016 consisting of a 1-h introductory session on the morning of Day 1, followed by a 1.5-h question-and-answer session. There were 1-h evening sessions on days 1–4. Before the meeting, attendees were provided with materials and invitations to the sessions. First-time attendees were reminded at plenary sessions and a schedule was provided at registration. Because there were PRP orientation and daily sessions running in parallel to the new participant program, PRP likely did not take part in these sessions.
Introductory session format
Before any meetings, all first-time participants completed an anonymous survey assessing their familiarity with OMERACT, including the concepts of the OMERACT filters 1.0 and 2.02,4 and the structure of the meeting. At the first session, there were presentations about OMERACT’s history and process, followed by open discussion and a question-and-answer session.
Daily evening session format
Prior to each evening session, participants completed an anonymous survey reassessing their comfort with the OMERACT structure and concepts. Two or more moderators staffed each 1-h session (JAS, VS, GW, and/or CP). This was followed by an NGT debriefing6,7 during each evening session in which each participant was asked to list their key experiences of the day. These responses were recorded and analyzed by the session coordinator (SG). These experiences are presented in Table 1.
Best experiences according to the participants in the program: NGT session. Eighteen people participated in the NGT session on the last day of the OMERACT and allocated either 3, 2, or 1 dots (equal to score of 3, 2, or 1) from the most pertinent to the third most pertinent best experience; total does not add to the exact multiple because some people did not use all dots.
The NGT is a variant of traditional methods aimed at identifying the overall opinion of a group. NGT is a structured process that facilitates development of an inclusive list of issues related to a specific question followed by feedback on the relative importance of these lists through rank-ordering6,7,8. The NGT approach promotes more even participation rates compared to focus groups with equal weighting of input from all participants.
During the last meeting, we used NGT to ask 3 questions: (1) What was your best experience of the OMERACT meeting? (2) What experience of the OMERACT meeting needs improvement? (3) What solutions do you propose to address the OMERACT experience needing improvement? An experienced NGT facilitator (JAS) first presented the questions, followed by participants noting their responses independently and reading each response to the facilitator, who recorded them verbatim. After all the responses had been nominated by the participants in a round-robin fashion and listed by the facilitator, participants were asked to elaborate and discuss their responses. After discussion, none of the responses could be consolidated. Participants chose their top 5 responses and gave them individual scores from 1 to 3 (higher score indicating the top choice/rank). Scores were aggregated for group rank-order of these questions, with higher scores representing the highest-ranked questions.
RESULTS
Participant characteristics
There were 43 first-time attendees. Characteristics of the new participants are shown in Appendix 1. Of the 43 new participants, 38 (88.4%) attended the introductory session. Between 14 and 18 participants (32.6%–41.9%) attended the daily evening sessions from days 1–4.
Participant familiarity and comfort with the OMERACT process
There was a high level of understanding of OMERACT at baseline. In 2016, the proportion of participants not understanding OMERACT sessions well or the OMERACT filter well was low, i.e., 7% each by Day 1 end (after the new participant introductory training session). Conversely, proportions understanding OMERACT sessions and OMERACT filter extremely well were as follows: 45% (pre) versus 47%, 44%, and 36% (post); and 22% (pre) versus 55%, 44%, and 40% (post), respectively. Even as early as after 1 day, most new participants felt they could contribute to sessions (93.3%). These proportions were higher than those at OMERACT 2014 (Appendix 2) and stayed stable or improved slightly through the entire duration of the 2016 OMERACT meeting.
Nominal group
At the end of Day 4, we conducted an NGT with 20 participants addressing 3 questions. New participants identified and ranked their responses to each question (Table 1, Table 2, and Table 3). The top 3 research questions identified for “best experience” were:
OMERACT culture
Systematic format of workshop session
Observing the process of developing instruments and networking (tie; Table 1).
Experiences needing improvement according to the participants in the program: NGT session. Eighteen people participated in the NGT session on the last day of the OMERACT and allocated either 3, 2, or 1 dots (equal to score of 3, 2, or 1) from the most pertinent to the third most pertinent experience needing improvement; total does not add to the exact multiple because some people did not use all dots.
Recommendations from participants during the Nominal Group Technique session in response to the question: “What solutions do you propose to address the OMERACT experience needing improvement?”
The top 3 research issues identified for “experience needing improvement” were:
Facilitation issue during breakouts (clarify objectives, encourage full participation)
Voting process (present questions in an unbiased fashion)
Burden versus feasibility of core set instruments and context of the workshop not being clear (tie; Table 2).
Solutions to these issues included not instructing participants on desired voting outcomes, use voting before discussion to identify areas of confusion best suited for discussion that helps in consensus-building, a requirement to state the time needed to complete an instrument, and a requirement to state objective(s) of the workshop up front.
Participants specifically developed solutions for the experiences needing improvement. As noted above, by far the most commonly reported experiences needing improvement involved the breakout sessions. Suggestions for improvement included training sessions for facilitators, clearly stating objectives at the beginning of sessions, and defining the scope of sessions up front (Table 3).
DISCUSSION
Our report describes results of the second formal OMERACT first-time participant training program, results of the NGT conducted at the end of the meeting, and practical solutions to problematic aspects of the meeting as seen by those with limited OMERACT experience. Unlike OMERACT 20145, baseline understanding and comfort level of participants was high (Appendix 2). This allowed the institution of an NGT, identification of good experiences and those not needing improvement, and elaboration of innovative solutions for those experiences needing improvement. The experiences needing improvement were obvious to participants who had not attended previous OMERACT meetings, and their solutions will be provided to OMERACT’s Executive Committee for possible implementation.
Participants’ reported levels of knowledge of OMERACT processes and concepts were higher than in 2014 (Appendix 2). The higher baseline knowledge may be related to improved premeeting materials and/or more diligence by new participants in understanding premeeting readings. This should prompt consideration of an assessment of premeeting materials at OMERACT 2018. Regardless of the reason for the higher baseline knowledge, it allowed more in-depth discussion to identify strengths and problem areas, and to develop concrete proposals for improvement.
The most commonly reported best OMERACT experiences included the OMERACT culture, the systematic format of workshops, and being able to observe how new instruments are developed. This is gratifying, because in addition to developing new outcome measures, one of the major goals of OMERACT is to develop new talent, especially study methodologists, and to enhance collaboration among those committed to improving evidence-based rheumatology. Based on these results, we plan to modify the session by providing additional premeeting materials and continuing the NGT session to provide substantive, innovative proposals for improvement, spurred by having a fresh set of eyes to identify problematic areas.
There are several limitations to our report. The high baseline level of knowledge prevented any conclusions about the change in knowledge or comfort level with participation over time. This was balanced in part by the ability to use NGT to identify problem areas and propose solutions. The need to keep the surveys brief and maintain the high level of interaction limited the amount of quantitative data. The number of participants decreased between the 2-h introductory session and the subsequent sessions, which may have affected the representativeness of the group. We think that the attendance dropped because subsequent sessions on days 1–4 were not mandatory for the new participants. The decline was similar to one noted in previous OMERACT new participant sessions5. A nominal group process/focus group was not done with the 4 Working Group leaders of the OMERACT new participant program to identify top areas of improvement, which could have provided additional insights. This will be done in the future.
Nevertheless, important positive areas were identified, and ways to improve were proposed. This program is in its infancy and was implemented for only the second time. OMERACT is unique in expecting all participants to actively contribute and vote, regardless of whether they have attended several meetings or are new participants. The objective of our program to orient and inform new participants at the OMERACT meeting, and to learn from this group, was achieved. The new participant program remains worthwhile and should be continued.
APPENDIX 1.
Characteristics of the new OMERACT participants at OMERACT 2016 compared with participants at a similar session at OMERACT 2014.
APPENDIX 2.
Responses to Day 1 questions before introductory new participant session at OMERACT 2016 compared with responses at OMERACT 2014.
Footnotes
VS is an employee of UCB BioSciences Inc.
- Accepted for publication April 5, 2017.
