Abstract
Objective. To describe the experience of a first-time participant (“newbie”) training program at the Outcome Measures in Rheumatology 12 meeting in 2014.
Methods. We conducted newbie sessions at OMERACT 12, including a 2-hour introductory session on Day 1, followed by 1-h evening followup sessions on days 1–4 of OMERACT 12. Pre- and postmeeting surveys assessed participants’ level of comfort with the principles of the OMERACT Filters 1.0 (truth, discrimination, feasibility), and Filter 2.0 (the essential tools for OMERACT methodology), the different types of OMERACT sessions, and whether participants felt welcome.
Results. In all, 25 new attendees participated in the introductory session and 10–16 attended followup sessions. Fewer participants reported being somewhat or extremely uncomfortable with the meeting, comparing Day 1 (preintroductory session) to days 1–4 (post): (1) with different OMERACT sessions: 56% (pre) versus 6%, 0%, 8%, and 6% (post days 1–4, respectively); and (2) with principles of the OMERACT filter, 64% (pre) versus 7%, 0%, 8%, and 0% (post), respectively. Most reported feeling welcome (100%) and that they were able to contribute substantively to breakout sessions (87%) on Day 1 evening; results were sustained on days 2–4.
Conclusion. First-time participant training sessions increased the comfort level of the participants with the OMERACT meeting structure and filter, and increased the ability of the new attendees to feel they could contribute to the OMERACT process.
Outcome Measures in Rheumatology (OMERACT)1 is an independent, methods-based organization with a variety of participants that aims to develop validated outcome measures for clinical trials in rheumatic conditions. OMERACT began in 19922, and several widely used outcome measures including the rheumatoid arthritis improvement core set3, which in turn formed the basis of American College of Rheumatology 20, 50, and 70 response criteria, were developed under the auspices of OMERACT. This was the foundation of evidence-based rheumatology. OMERACT meets face to face every 2 years, with attendees from various backgrounds including outcomes researchers, clinical trialists, epidemiologists, health services researchers, regulatory agency scientists, industry scientists, patient research partners, clinicians, and allied health professionals. A key aspect of the OMERACT meeting is the use of standard, rigorous methodology to examine the evidence for validity of measures for domains that constitute the core set for a disease of interest, and achieving consensus from various participants. Because the meetings are small (n = ∼200), active and substantive contributions by all participants are critical to the success of the conference.
Challenges for someone attending an OMERACT meeting for the first time include understanding the distinctive OMERACT terminology, the structure and purpose of the various sessions, and the process of this unique, methodological conference. While seasoned OMERACT participants know terms such as OMERACT filter and TDF (truth, discrimination, feasibility), and understand the differences between various meeting types (e.g., special interest group, workshop, or module), these are foreign to first-time participants. The OMERACT filter is the critical assessment tool used to determine the suitability of outcome measures for use in clinical trials. At OMERACT, first-time participants are referred to as “newbies.”
At each meeting, the number of newbies varies; at this past meeting they represented 18% of participants. Feedback from the last few OMERACT meetings, especially from the newbies, identified a need for education of new attendees to allow them to be active participants as quickly as possible. This participation is essential, because the entire OMERACT group votes on whether measures developed meet the filter and can be endorsed for use by the greater rheumatology community. Because all participants’ votes carry equal weight, it is desirable that all attendees, including newbies, have the requisite knowledge and familiarity with the OMERACT process and structure. The OMERACT executive made a decision to introduce a 2-h information session for newbies at OMERACT 11 in 2012. This consisted of a 1-hour didactic session summarizing the history and main aspects of the OMERACT process and principles of validated measures, followed by a 1-h discussion. Owing to its success, a decision was made to expand the newbie session at the next OMERACT meeting. The OMERACT executive aimed to develop a more formal program/training for OMERACT 12 (2014) and to assess the effect of this program on knowledge and confidence among newbies. This report describes the process and results from the newbie session at OMERACT 12, as well as an overview of the new participants’ OMERACT experience.
MATERIALS AND METHODS
OMERACT Newbie Program
A new OMERACT attendee program was created for OMERACT 12. The program consisted of a 2.5-h introductory session on the morning of Day 1. This was followed by 1-h evening followup sessions on days 1–4. New attendees were provided with preconference reading materials and invitations to the sessions prior to the meeting. All newbies were invited to attend each session. The information regarding the newbie session was provided in the printed program book as well as announced daily during regular OMERACT sessions.
Introductory Session Format
Prior to any formal presentations, all participants completed an anonymous survey assessing their familiarity with OMERACT, including the concepts of the OMERACT filters and the structure of the meeting. The introductory session consisted of two 20-min presentations: the first focused on the process and history of OMERACT; the second reviewed how instrument validity is tested using the OMERACT TDF filter (formerly known as OMERACT Filter 1.0) and how disease core set domains and select instruments are developed to meet the specifications of OMERACT Filter 2.04,5. The remaining time was devoted to questions and answers.
Daily Evening Session Format
Prior to each followup session, participants completed an anonymous survey reassessing their comfort with the OMERACT structure and concepts. Two or more moderators staffed each 1-h session (JS, VS, GW, and/or CP). Prior to any discussion, participants completed a brief questionnaire. In a nominal group technique debriefing6,7 each participant was asked to list the “best” and “worst” experience of the day. These responses were recorded by the session coordinator (SG) and analyzed by the authors. Common themes were listed on a flip chart and discussed in detail. Moderators asked clarifying questions when required. This was followed by an open question-and-answer session. These questions are presented in Table 1.
Questions for first-time attendees (newbies) at daily OMERACT sessions.
RESULTS
Participant Characteristics
There were 39 newbies at OMERACT 12. Demographic and other characteristics are shown in Table 2. Of the 39 newbies, 25 (64%) attended the introductory session. Between 10 and 16 (26–41%) attended the daily followup evening sessions from days 1–4.
Characteristics of first-time attendees (newbies) at OMERACT.
Familiarity and Comfort with the OMERACT Process
We analyzed the participant ratings of the comfort with the OMERACT process with a daily survey. Results are shown in Figure 1. Two key aspects of participants’ understanding and comfort with the OMERACT process and principles improved with time during the newbie sessions. Fifty-six percent of participants were somewhat or extremely uncomfortable with different types of sessions prior to the newbie session compared to 6.3% on Day 4 (final session); 64% of participants felt somewhat or extremely uncomfortable with the principles of the OMERACT Filter 1.0 (TDF) prior to the newbie session compared to 0% on Day 4. Participants reported feeling more welcome in the meeting sessions, with 40% reporting feeling extremely welcome on Day 1, compared to 63% on Day 4. Overall, 100% felt welcome days 1–3 (extremely or somewhat welcome) and 94% on Day 4.
Analysis of participant comfort with the OMERACT process over the course of the meeting. Y axis shows the number of participants with each response. The intro session feedback was given prior to conducting the first session. Feedback 1, 2, 3, and 4 correspond to feedback on meeting days 1 through 4, after each daily session, respectively.
Participant Comments
Participants identified several of their “best” and “worst” experiences at the end of each day of the meeting. Moderators grouped these comments into common themes at the end of each session. Several themes emerged as summarized in Table 3.
Themes and quotes for the best and worst daily experience.
Best experience
Several key themes emerged during the newbie sessions that were considered highlights of the OMERACT conference: (1) The participants found Filter 2.0 intellectually challenging but appreciated its usefulness to assess the OMERACT process. The level of understanding and comfort with Filter 2.0 increased over time; (2) networking opportunities and the spirit of interactivity; (3) small group discussions; (4) patient involvement in OMERACT; and (5) opportunity to learn and discuss the science of outcome measurement. Other themes are listed in Table 3.
Worst experience
Several key challenges emerged during days 1–4 of the newbie sessions: (1) Challenges in understanding OMERACT Filter 2.0: participants identified several challenges in understanding the concept as well as the content of the new filter, as well as its relationship to Filter 1.0; (2) the voting process (time shortage, lack of adequate data, and the wording of some voting questions); (3) too much information in some sessions (both premeeting and meeting presentations) making them difficult to understand, with a resulting difficulty in making informed decisions during the voting process; (4) lack of the desired format of “more discussion and less presentation” time in some small group sessions; (5) length of the plenary session and lack of substantive voting; and (6) complexity of the OMERACT process.
OMERACT 2014 Session Leader Feedback
We sent a postmeeting survey to session leaders asking whether they agreed that: (1) the new attendees’ training session increased the comfort level of participants, and (2) the attendees’ participation (or comfort with active participation) in the sessions increased over time. In all, 15 of 19 session leaders responded. Overall, respondents agreed that both the participation of new attendees and the new attendees’ comfort in the sessions increased as a result of the training program. Results are shown in Figure 2.
Session leader feedback. Y axis shows the number of session leaders with each response.
DISCUSSION
This report describes the OMERACT first-time participant training program, its effect on participants’ experience with and participation in the OMERACT sessions, as well as qualitative feedback from newbies who attended the OMERACT 12 meeting (2014). We examined the effect of the program on newbies’ comfort with the meeting in general, as well as with the OMERACT principles, at the daily followup sessions, using a brief, structured, anonymous 5-question survey. There was a progressive increase in participant confidence in understanding the OMERACT filter and process with time. In line with an objective of the program to assess participants’ OMERACT experience, they provided us with feedback about the best and worst experiences of the day. These key findings deserve further evaluation by OMERACT.
We noted positive changes in newbies’ confidence in their knowledge of the principles of the OMERACT filters and their knowledge of the different types of sessions, which likely helped them to have a beneficial experience at OMERACT as well as to make substantive contributions to discussions and to the consensus process. Importantly, session leaders also noted that new attendees’ participation in and comfort with the individual sessions increased over the course of the meeting. We can speculate but not conclude definitively that the newbie program may have contributed to the increase in their knowledge and confidence. The Hawthorne effect (a group of people under observation reporting improvement related to the attention and being observed) as well as other exchanges that newbies had during the OMERACT meeting may also have played a role. The extent of the contribution of the newbie session to the increase in participants’ confidence/knowledge as demonstrated by the survey cannot be isolated without performing a randomized study with a control group that does not attend the newbie session. Such a design was not thought to be feasible or desirable given that the objective of this program was to allow optimal participation in the OMERACT meeting by all new attendees.
Participants provided us with worst and best experiences of the day. Some of the most common best OMERACT experiences included the ability to learn the science and process of outcomes research, to acquire new skills, to network with colleagues, as well as the intellectual stimulation. This is gratifying since in addition to developing new outcome measures for clinical trials, one of the major goals of the biannual OMERACT meeting is not only to advance science but also to build a cadre of methodologists and provide a platform for methodologists to interact, advance science, and develop new collaborations. Participants also appreciated the utility of small-group discussions and the active patient involvement in the process of development of outcome measures, hallmarks of the organization that support the data-driven consensus OMERACT process, which involves a variety of people.
The worst experiences noted by newbies are worth as much, if not more, attention because they provide guidance on areas for improvement at future OMERACT meetings. Key negative experiences included too little time for discussion; concerns around voting, including challenges with clarity on some questions (presented for voting after data presentation) related to whether an instrument met the OMERACT filter; presentation of insufficient data during some workshops; and too few votes during the plenary session.
These potential areas requiring improvement have been shared with the OMERACT executive and will be considered in the planning of the next OMERACT meeting. We are planning to modify the newbie session at OMERACT 13 (2016) based on these comments/concerns: more material will be provided (OMERACT glossary, definitions and overview of filters 1.0 and 2.0) to new attendees in advance of the meeting; additional experienced OMERACTERs will serve as moderators, allowing more small-group interaction and education at the initial training session. These additional steps will maximize opportunities for new attendees to incorporate the key concepts of OMERACT to maximize their participation at the meeting sessions. This in turn will help develop a new group of experienced methodologists who can carry on the work of OMERACT in refining and improving evidence-based rheumatology.
There are several limitations to this report. This was a program focused on assisting newbies in learning the OMERACT process quickly and allowing new attendees to maximize their contributions to the OMERACT meeting. Because the program was focused on education and interaction with the newbies rather than on multiple assessments, we made a decision to keep the surveys very brief. This led to more qualitative than quantitative data on the best and worst aspects of the meeting.
Nevertheless, with attendance of the newbie program, clear trends in favor of better knowledge of and comfort with the OMERACT process, sessions, and filter were noted over the course of the meeting. This in turn may have increased effective participation by new attendees at sessions, as evidenced by the feedback from the session leaders. Because the OMERACT meeting is small, full participation by all attendees is ultimately essential to the future success of OMERACT and its important goals of furthering evidence-based rheumatology.
