Abstract
Objective. To identify the major features of OMERACT meetings as valued by frequent participants and to explore whether there are groups of participants with different opinions.
Methods. Using Q methodology (a qualitative and quantitative approach to grouping people according to subjective opinion), participants (who attended more than 1 OMERACT conference) sorted 66 statements relating to the “spirit of OMERACT” according to level of agreement across a normal distribution grid. Data were examined using Q factor analysis.
Results. Of 226 potential participants, 105 responded (46%). All participants highly ranked the focus on global standardization of methods, outcome measures, data-driven research, methodological discussion, and international collaboration. Four factors describing the “spirit of OMERACT” were identified: “Evidence not eminence” (n = 31) valued the data- and evidence-driven research above personality and status; “Collaboration and collegiality” (n = 19) valued the international and cross-stakeholder collaboration, interaction, and collegiality; “Equal voices, equal votes, common goals” (n = 12) valued equality in discussion and voting, with everyone striving toward the same goal; “principles and product, not process” (n = 8) valued the principles of focusing on outcome measures and the product of guiding clinical trials, but were unsure whether the process is necessary to reach this. The factors did not segregate different stakeholder groups.
Conclusion. Delegates value different elements of OMERACT, and thus the “spirit of OMERACT” encompasses evidence-based research, collaboration, and equality, although a small group are unsure whether the process is necessary to achieve the end result. Q methodology may prove useful for conference organizers to identify their delegates’ different needs to tailor conference content.
Outcome Measures in Rheumatology (OMERACT) is an informal international network initiated in 1992 with the aim of improving outcome measurement in rheumatology1. Through a biannual conference and working groups that carry out research between conferences, OMERACT has built data-driven consensus for many rheumatologic conditions2, including rheumatoid arthritis (RA)3, ankylosing spondylitis4, and osteoarthritis5. It has done so by developing widely endorsed “core outcome measurement sets,” each a minimum set of outcome measures covering key domains6 that must be reported in all randomized controlled trials in a given health condition. Working groups prepare or produce an evidence base to support the identification of domains and instruments to measure those domains in their own areas of interest and expertise (e.g., RA flares7) through literature reviews and validation studies. At the conference, plenary presentations alternate with small group sessions (breakout sessions), where delegates have the opportunity to express their views and preferences. The views expressed in the breakout sessions are then reported at a plenary session, where final consensus is formed with the help of interactive voting. Consensus can be used to drive a research agenda, as well as to provide agreement on measures or domains1.
The design and format of the OMERACT conference, which has a limited number of delegate places and is held over 5 days at an integrated venue, has the intention of allowing respectful, deliberative dialogue and relationship building among those involved including clinicians, methodologists, regulatory agencies, industry, and patients, who participate as equal partners in the process8.
Many delegates repeatedly attend OMERACT conferences, even though this takes a week of their time (including travel) and is an exhausting, intense experience, with working hours lasting from the early morning until late at night. However, it is not known which features or characteristics of the conference particularly appeal to people and encourage them to return. It is important to understand the subjective views of delegates on what contributes to the “spirit of OMERACT” to ensure that future conferences maintain the elements that delegates value the most. This is particularly timely because OMERACT has for the first time set out its underlying philosophy6 and published the OMERACT Handbook, which details its operational strategies9.
Traditional consensus methods such as the nominal group technique10 and the Delphi technique11 rely on interactive discussion between participants or repeated interactions until participants are able to reach an agreement. However, this produces an averaging of opinions. The average scores of value ratings attributed by delegates to different aspects of the conference might provide a broad overview, but these methods will not reveal the existence of groups of delegates with different views, nor explore the interactions between different features of the conference program. Q methodology is specifically designed to derive a number of groups (called “factors” in Q methodology), which each represent a different and independent opinion of the issue12.
Q methodology13 combines strengths of qualitative and quantitative approaches to identify factors made up of those who share similar opinions based on their value judgments in prioritizing a large set of descriptors. An appropriate set of descriptors or statements is assembled, each participant sorts them along a continuum of agreement designed to emulate a normal distribution with relatively few extreme values and many central values (Figure 1). The normal distribution is used because it is believed to be the closest distribution to the way that people form opinions14.
The sorting matrix used by participants (66 statements).
Participants sort each statement in approximate rank order of the degree to which they agree with that statement in relation to all the other statements. By using a normal distribution grid, the weight assigned to each statement in the analysis increases slightly for each place closer to the extreme ends of the grid, and the positions (scores) of all the statements are included in the analysis for every participant. The statements sorted closer to the outer edges of the distribution have more influence on the factor groupings in relation to the statements closer to the middle of the distribution. Factors are calculated and the results interpreted within the context of the enquiry5,6. Exploring the similarities and differences between the opinions of these groups of participants will provide greater insight into the aspects that different delegates find attractive, and will also allow a comparison of the characteristics of the participants who fall into the different factors. Therefore, we invited all OMERACT delegates who had chosen to return to the conference at least once to take part in an online Q sort survey.
MATERIALS AND METHODS
Participants (P-set)
Participants gave informed consent by anonymously deciding to participate, and ethics approval was granted by the University of the West of England Research Ethics Committee (Ref: HAS/13/11/151). All OMERACT delegates who had attended at least 2 OMERACT conferences were invited to participate. In all, 332 eligible participants covering all stakeholder groups (clinicians, researchers, fellows, patients, and industry) were identified from previous conference delegate lists and contacted by e-mail to explain the study and invite them to participate online. Two e-mail reminders were also sent, and encouragement to participate continued until participation was closed on the day before a presentation of results from the first 50 respondents was made at an OMERACT conference. Survey reminders were sent to all delegates, because the survey was anonymous. The first delegates attended in 1992, thus an estimate of the potential responder population suggested that 226 would be available to respond, based on UK life expectancy and expected retirements.
Statements (Q set)
Several sources were used to collate or produce a wide range of statements regarding the features of OMERACT that might be important to participants. First, data were collected during an internal review of activities conducted 2 years earlier by an independent consultancy (iDENK15). This included interviews, focus groups, and a survey with OMERACT delegates; attendance at executive committee meetings and conference calls; and interactions with working groups. Second, data were collected as part of an investigation into the way patients have been incorporated as participants in OMERACT, including interviews, reports of attendance at patient sessions, analysis of OMERACT documents, and analysis of OMERACT conference proceedings16,17. Finally, to ensure a full range of potential opinions about OMERACT, delegates who had attended only 1 OMERACT conference (either 9 or 10) and chosen not to return were asked to provide their opinions of OMERACT. The statements were refined through discussion with the research team, including a patient research partner. After removing repeated or ambiguous items, 66 statements were included, each worded to follow on from the stem: “To me, the ‘spirit of OMERACT’ is…” (Table 1).
By-factor ranking of statements (St) given in the “spirit of OMERACT” Q study.
Obtaining participants’ preferences
The Q methodology study was completed online using FlashQ, a software package designed for collecting Q methodology data18. It was hosted for the present study on a University of the West of England Website and managed by the first author (CF). A prestudy e-mail was sent to potential participants, advising them when the study would be starting and providing a very brief outline of its intent. This was followed a month later by the invitation e-mail message, which contained a participant information sheet explaining the study and a link to the Website. Participants were advised to allow 45–60 min to complete the online study. Because data were submitted anonymously, reminder e-mails were sent to all participants at intervals of 2 weeks after the initial invitation to thank those who had responded and encourage those who had not yet responded. Participants had the option of asking the research team to stop further reminder e-mails.
The link to the online study presented participants with instructions for completing the Q sort (Table 2). As a first step, participants were presented with the statements one at a time and asked to consider each one in relation to the statement stem “To me, the ‘spirit of OMERACT’ is…” and to sort them into 3 broad categories: agree most; agree least or disagree; neutral. The statements were presented to each participant in a different random order.
Instructions given to participants.
When this was complete, the second step asked each participant to review their broad categories and arrange each statement in approximate rank order of the degree to which they agreed with that statement relative to the other statements. Each of the 66 statements was placed in a single box on the Q sort grid (Figure 1) of 66 boxes. The grid pattern allowed for the majority of statements to be agreed or disagreed with mildly or neutrally (for example, there were 10 “0” boxes, and 7 “+1” or “−1” boxes each), but only 1 statement could be placed in the highest agreement box (“+7”) or highest disagreement box (“−7”). Thus each participant’s opinions on the statements were constrained into a quasi-normal distribution of degrees of agreement with the statements (Figure 1)19. The precise shape and limits of agreement/disagreement of this distribution (and the grid) are dependent on the number of statements. A participant could rearrange the position of each statement within the grid as the procedure continued, until satisfied with the distribution.
In the third step, participants were asked to comment on the statements they had placed at the extreme positions at either end of the grid, and were also given the opportunity to provide an open comment about their reasons for how they had sorted the statements overall, or any comments on any particular statements. Finally, additional data were also collected on age, sex, number of OMERACT attendances, and delegate category (e.g., industry, patient).
Analysis
The statements are assigned a score for each participant based on which column on the grid they have been placed in (−7 to +7). To provide a broad overview, the mean of the scores given by all the participants to each statement was calculated. For detailed analysis, Q methodology combines qualitative and quantitative methods to produce a rounded interpretation of a single dataset (in contrast to a mixed-methods approach)20. Q methodology analysis involves factor extraction, rotation, and interpretation. Factor extraction and rotation used the PCQ software package21. In Q methodology, participants are treated as variables and are intercorrelated and subjected to by-person factor analysis. The software searches for shared patterns (or sorting configurations) in the data and extracts portions of common variance (factors). For each Q factor to be interpretable, an eigenvalue > 1.0 (indicating factors are unlikely to have grouped participant views by chance), and ≥ 1 Q sort loading significantly upon each factor alone is required22. Following extraction, the factors were rotated using orthogonal varimax rotation to ensure each Q sort defined (has a high factor loading in relation to) only 1 of the study factors, so the overall solution maximizes the amount of study variance explained23. For ease of interpretation it is standard Q-methodological practice to generate a single exemplary Q sort for each factor by merging (according to a procedure of weighted averaging) the Q sorts of all significantly loading participants on the given factor (termed the factor array)23. A decision on the final selection of the optimum factor solution was undertaken collectively by the authors, examining the outputs from the different factor analysis solutions (e.g., number of factors, weightings, explained variance, number of participants excluded from factors). Factor interpretation was based on the factor arrays and the open-ended comments from the factor exemplars (significantly loading participants), which were combined to provide a single gestalt explanation of each factor23,24. Illustrative examples of these open-ended comments are included in the results.
RESULTS
Study population
Of the estimated 226 participants available to respond, 105 participated (46%). The participants were 59 researchers (56%), 25 clinicians (24%), 7 patients (7%), 9 from industry (8%), and 5 fellows (5%). A preliminary analysis of the first 50 participants to respond was presented at the OMERACT 12 meeting in 2014. Recruitment ended the day before the presentation, by which time a further 55 respondents had taken part. For these 2 sets of respondents (effectively, early respondents and later respondents who had received a larger degree of encouragement to take part), separate and combined factor analyses provided very similar results, testing multiple factor solutions. Further, the demographic data were very similar in the 2 groups. Therefore we present a combined analysis of all 105 participants. Table 3 shows the study population and demographic data for each factor. Of the 105 people who participated in this study, 60% were male and 56.2% were researchers (mean OMERACT attendance: 4.36 times, SD 2.54). Median time to complete the online study was 25 min (interquartile range 19–35 min).
Summary characteristics of participants, and factor.
Overview of consensus statements
The mean score for each statement is shown in Table 1. When ordered from highest to lowest, obvious inflexions suggest that statements scoring ≥ +2.43 or ≤ −3.23 have a very high degree of consensus (see bold type, Table 1). These statements indicate the aspects of OMERACT most valued by participants as a whole: First, the specific focus of OMERACT on striving for global standardization and/or validation of methods [Statement (St) 41: overall mean statement score +3.88]; outcome measures (St23: +3.82); and the research being driven by data and/or evidence (St18: +3.02). As participant (P) 38 commented: “It is and should always be the first letter: ‘OUTCOME’ in OMERACT” (P38, clinician, male). Second, participants value the community aspects of lively methodological and interactive discussion [St42: +2.48; St19: +2.46; “Coming away from the meeting exhausted but stimulated” (P31, clinician, female)]; and international collaboration (St24: +2.43).
The great majority of participants disagreed with the statement “OMERACT is just an elitist clique” (St64: −4.28): “Derogatory words that misrepresent what OMERACT is about: Anyone can join who is willing to shoulder the work.” (P50, researcher, male). There was also consensus disagreement with the ideas that you need to be thick-skinned or have sleepless nights to attend (St16: −3.28; St 17: −3.35). Further, while the final evening entertainment was often discussed fondly, it was not considered important in contributing to the “spirit of OMERACT” (St47: −3.23): “Of course the exotic places and the final night, but you cannot put that [as the item you] like the best!” (P22, clinician, male).
Q methodology overall factor solution
A 4-factor solution was used based on (1) the solutions having a maximal explained variance; (2) the solutions having a maximum number of Q sorts loading significantly on 1 factor; (3) all factors with eigenvalue > 1.00; (4) all factors containing statements distinguishing them from other factors; (5) a minimum number being confounded (i.e., significantly loading on > 1 factor); and (6) the researchers’ judgment. Table 3 provides a summary of characteristics of all factors presented. Table 1 presents the average rating (provided by the factor array) given by each of the 4 factors to each statement. The 4 factors explain 44% of the variance and account for 70 of the 105 participants (67%). Participant loading of ≥ ± 0.40 reached significance at p < 0.01, indicating that each loading participant closely exemplifies the factor they load onto23. The factors will be presented in order of the highest number of loading participants.
Factor One: “Evidence not eminence.”
This factor consisted of 31 participants who were predominantly researchers, but included representation from all stakeholder groups; they were predominantly male; and mean OMERACT attendances were representative of the study population (Table 3).
Factor One exemplars highly valued the OMERACT process of data and evidence being used to drive the research agenda [Table 1: St18: score +6]:
“A strong prerequisite to make sound judgments on interventions in clinical trials”
(P34, researcher, male);
“It is based on rigorous data assessment and use of the Filter. I think that this places the OMERACT conference as the ‘crucible’ where ideas for outcome measures are tested”
(P58, clinician, male);
“The goal is to improve what we do for the benefit of the patients. Evidence is the only thing that has consistently improved how we help patients”
(P66, industry, male).
These participants agreed that the OMERACT process is transparent (St35: +5). They valued the exchange of ideas to address shared goals and reported that this sharing of ideas sets OMERACT apart from other conferences:
“This level of commitment and interest in the details can best be summarized by the “geekiness” variable, and is what separates OMERACT attendees from other meetings”
(P33, industry, male).
These participants felt strongly that OMERACT is not “just an elitist clique” (St64: −7). Although this statement score was in agreement with Factors Two and Three, in contrast to those factors, the discourse in this factor suggests that this was due to the belief that the data should always win out over personalities or eminence:
“To win over ‘non-specialists’ on the voting process, one has to be clear and concise. OMERACT members are good at letting the presenters know an argument has not been made on a voting point. I don’t vote for something because I trust the presenter. I vote for something if the presenter has educated me enough to believe him/her”
(P51, clinician, male);
“I believe all have their ideas put to the public blow torch at OMERACT, and famous investigators have failed to convince, hopefully underpinning that evidence not personalities rule OMERACT”
(P90, clinician, male);
“Evidence, not eminence. Eminence can focus efforts towards laudable goals, but it is never the goal itself”
(P66, industry, male).
The discourse from this factor indicates that these participants go to OMERACT to further research objectives and not to socialize. They definitely do not use OMERACT as a chance to escape their working life (St60: −6) and they see the exotic and remote locations as an unnecessary distraction to getting the work done (St45: −5; St46: −5):
“I have other (intellectual and urging) engagements to attend to in daily life, so absolutely no reason to escape from these time-consuming obligations”
(P25, patient, female);
“We rarely go outside”
(P33, industry, male);
“More distraction or deterrents of getting outcomes completed, adds unnecessary complexity”
(P68, industry, male).
Factor Two. “Collaboration and collegiality.”
This factor consisted of 19 participants who were predominantly researchers, and more female participants were included in this factor than the other 3 factors. They had a slightly higher mean number of OMERACT attendances (5.11) than the study population (4.36; Table 3).
Participants loading onto Factor Two particularly valued the collaboration and collegiality that OMERACT provides. The elements of OMERACT most important to these participants were those that relate to the interaction with other delegates. They valued the intimate aspect of the conference and the limited number of delegates; the discourse suggested that they saw these elements as allowing increased opportunities for collaboration and interaction (St20: +5):
“Having our own area for dining means that we continue debates over meals. All being in the same location (rather than a range of conference hotels spread across a city) lengthens the day at both ends, and also allows for socializing, which further bonds the group”
(P03, researcher, female).
The collaborative character of OMERACT valued by this factor contributed to their decision to reject the idea that OMERACT is “just an elitist clique” (St64: −7), in contrast to the reason of evidence over eminence given in Factor One:
“I don’t think of us as elitist … I find it quite collaborative and hard-working”
(P52, researcher, female).
These participants were less concerned than those in other factors with helping to get therapies approved (St7: −5). However, the focus on small group discussions was particularly important to them (St04: +7) because these were seen to be more effective for making progress than big plenary sessions alone; and participants indicated that this is what makes OMERACT innovative and distinct from other conferences:
“In small groups, communication and reaching consensus and how we get there is easier and more interesting and useful”
(P22, clinician, male);
“This is the most importance difference between OMERACT and other conferences, in the dialogue most progress is found”
(P55, researcher, male).
A further element valued by these participants was the opportunity to work together with like-minded people on the same intellectual level. They valued the intellectual stimulation and the opportunity to discuss their work with knowledgeable others (St51: +6; St65: +3):
“There is diverse intellectual discussion within individuals from a variety of backgrounds, which is often productive”
(P39, researcher, male).
The value these participants placed on the collaborative character of OMERACT may explain why it is more important to them than any other factor that patients are invited to the conference (St05: +5). They also scored higher than the other factors on the statement “the freedom to be a geek with others like myself” (St63: 0). The comments indicated that while not all participants in this factor saw themselves as a “geek,” they did appreciate the collegiate character of OMERACT:
“I do not feel like a geek. It is just passionate people, and even though most of them are famous researchers, most of them act so humbly it is really stimulating”
(P22, clinician, male);
“I am considered a geek by colleagues … yet at OMERACT the place is filled with people who want the job of measurement done right”
(P52, researcher, female).
The discourse in this factor suggests that OMERACT is inclusive and welcoming, thus these participants strongly believe that one does not need to be “thick-skinned” to attend (St16: −6); nor do they feel OMERACT is gladiatorial in character, with new delegates having to prove themselves (St34: −5):
“This is unnecessary ego-tripping and can be very destructive. We were all ‘newbies’ once”
(P03, researcher, female);
“OMERACT is a very level playing field where everyone involved in groups contributes. The leaders have worked hard to create an environment and ethos that everyone has an important voice and is encouraged to contribute”
(P89, clinician, female).
Factor Three: “Equal voices, equal votes, common goals”:
This factor consisted of 12 participants who were predominantly researchers, with a slightly lower mean OMERACT meeting attendance number (4.17) than the study population (4.36; Table 3).
Factor Three exemplars valued the elements of the OMERACT process that give all delegates an equal voice in the discussions and an equal vote in the final consensus, while striving toward a common goal. Thus they positively rated the voting being equal and being at the conference (St38: +3; St37: +5), and the use of the Delphi procedure (St55: +3). The discourse in this factor suggested that it is important to these participants that everyone has enough information to contribute equally:
“Ensuring the data is there, that everyone understands it, discussions about strengths and weaknesses, voting, back for the next meeting with more data”
(P54, clinician, female).
Perhaps owing to the value placed on equality, these participants do not agree that OMERACT is an opportunity to meet “famous” researchers and/or rheumatologists (St26: −5). They do not agree that new delegates have to prove their robustness and worth (St34: −6) nor that OMERACT is “just an elitist clique” (St34: −7). The discourse suggests that these participants feel everyone is encouraged to contribute and that their opinions are valued:
“The leaders are enthusiastic and motivating and even though they may be leaders in their field they are approachable and friendly”
(P18, researcher, female);
“Not being a rheumatologist, I learned a great deal and was very much encouraged to contribute and explain my own discipline and its value to rheumatology”
(P07, researcher, male).
These participants value the research being evidence- and data-driven (St18: +7), the focus on guiding the conduct of clinical trials (St48: +5), and striving for consensus (St39: +6). These values and goals are more important to this group than the opportunity to convince their peers that their work is satisfactory (St15: −5), or the chance to get international recognition for their work (St22: −5). This attitude may indicate that the participants’ motivation to attend is something other than the desire for individual recognition.
“Everyone is there to learn with common goals of helping patients”
(P78, industry, male).
Factor 4: ”Principles and product, not process”:
Although this factor has the lowest eigenvalue (6.56), the number is substantially higher than that due to chance alone (eigenvalue < 1.00). In addition, this factor was included because the 4-factor solution strengthened the other 3 factors compared to the 3-factor solution, and it seemed realistic that a small number of delegates might hold the views expressed within this factor.
Factor Four consisted of 8 participants who were predominantly researchers, but the sex and mean number of times attending OMERACT meetings were representative of the study population (Table 3). Factor Four exemplars valued the overall principles and products of OMERACT, but were unsure as to whether the OMERACT process is necessary or appropriate to achieve this:
“OMERACT is a great idea that needs updating”
(P05, researcher, male).
Similarly to the participants in other factors and the overall consensus, these participants value the overall principle of OMERACT, that is, the focus on outcome measures (St23: +7): “OMERACT should focus on this topic” (P16, clinician, male). They also value the products of OMERACT, including guiding the conduct and reinforcing the rules for clinical trials (St48: +5; St50: +5):
“This is a very important feature of OMERACT; we’ll encourage trialists to use the same standardized outcome measures. OMERACT is therefore protecting us against selective outcome reporting from industry”
(P42, researcher, male).
However, the participants loading onto this factor were more inclined than all other factors to consider OMERACT as “an elitist clique” (St64: +3):
“Too expensive and too exotic”
(P16, clinician, male).
These participants do not agree that the OMERACT process is transparent (St35: −5); has fewer visible egos than at other conferences (St44: −7); nor that everyone’s opinions are treated as equal (St03: −3). However, they do value the involvement of a core committed group of people more than the other factors (St40: +4). They did not agree that the voting process is equal or that it should be held at the conference (St37: −3; St38: −6;), which the discourse suggests is because they are unsure whether this process is necessary or appropriate to reach the end product:
“This [voting process] is a bit silly since most things are approved by the time the large group votes. Not all attendees know or care about some topics and really are not qualified to vote (not due to position or training, but due to indifference)”
(P23, researcher, male);
“I think that much of the work is done prior to the meeting and that those driving the agenda will paraphrase the questions until they get the response they want, or will continue anyway without consensus”
(P56, researcher, female).
Despite holding the opinion that the voting process may be unnecessary, these participants still value the collaborative design of the conference, and the involvement of a wide variety of people (St24: +6; St13: +5):
“International ‘buy-in’ is crucial. [It] forces us to seek consensus and inclusion”
(P23, researcher, male).
Despite the seemingly more negative views held within this factor, these participants have a mean number of 4.36 times attending OMERACT meetings and therefore it is likely that the perceived positive aspects of OMERACT (e.g., focus on outcome measures and collaboration) outweigh the perceived negative aspects in this factor.
Confounding sorts (sorts loading onto > 1 factor):
A participant’s individual sort is considered to be confounded if it significantly loads (≥ ± 0.40, p < 0.01) onto > 1 factor23. This study contained 22 confounded sorts: 9 researchers, 5 clinicians, 5 patients, 2 fellows, and 1 person from industry. Half of confounders (5 researchers, 4 patients, 1 fellow and 1 clinician) loaded onto Factor One: “Evidence not eminence” and Factor Two: “Collaboration and collegiality,” and therefore held views that were equally balanced between the 2 factors. Five of the confounders (1 researcher, 2 clinicians, 1 patient, and 1 person from industry) loaded onto Factor One: “Evidence not eminence” and Factor Three: “Equal voices, equal votes, common goals.” The remaining confounders were distributed across different groups. The majority of the patient participants (6/7) loaded either solely onto Factor One, or were confounded across Factor One “Evidence not eminence” and Factor Two “Collaboration and collegiality,” and therefore held views that were equally balanced between these factors. The remaining participant groups were distributed across the different options.
DISCUSSION
Almost half of OMERACT delegates who had chosen to attend more than 1 meeting and were likely to be available to participate in this study provided material to review those aspects of OMERACT that embody the “spirit of OMERACT.” Collectively there is a broad consensus that the focus on outcome measures, global standardization of methods, data-driven research, international collaboration, and methodological discussion contribute to creating the “spirit of OMERACT.” Within this, differences of preference emerge that distinguish 4 relatively distinct factors, for which we have coined the following descriptive labels: Evidence not eminence; Collaboration and collegiality; Equal voices, equal votes, common goals; and Principles and product, not process.
While some delegates believe the evidence- and data-driven elements of the conference should be valued above personality (Evidence not eminence), others value the collaborative and interactive elements of the conference and the feeling of collegiality (Collaboration and collegiality). Some valued the focus on equality, with everyone striving toward the same goal (Equal voices, equal votes, common goals) and others value the principles of focusing on outcome measures and the product of guiding clinical trials, but are unsure whether the process is necessary to reach this (Principles and product, not process).
The different participant groups were distributed among the 4 distinct factors, or were confounded between more than 1 factor. This indicates that participant experiences and opinions of OMERACT cannot be categorized according to participant group. If one group had a particularly distinct opinion or experience, this would have been recorded by the method as a separate factor. It is important to note that although patient participants were not greatly represented in the factor solutions, they were present as confounding sorts. Thus, their opinions were not sufficiently distinct to create a separate factor, but in agreement across different areas of opinion (Factor One or confounded across Factor One and Factor Two).
The finding that OMERACT is valued for its focus on outcome measures, data-driven research, collaboration, and equality through consensus agreement supports the description that OMERACT provides for itself on its Website2 and in the OMERACT Handbook9. The finding in the current study that data-driven research is highly valued by some delegates supports previous research17 that found one obstacle to accepting patient participation at OMERACT was concern that knowledge brought by patients is experiential. Further, the finding from the current study that collaboration and every delegate being given an equal voice in the discussion are important contributors to the “spirit of OMERACT” supports the finding that many professionals had their perceptions changed by patients:
“Patients were a kind of sparring partner when I entered a relatively new area”17.
These findings also support an evaluation of a conference focused on the inclusion of minorities into clinical research trials25, which found that inclusion and collaboration is necessary to make progress.
These are the first data using Q methodology to evaluate a research conference, and it has proved to be a valuable way of measuring delegates’ motivations for attendance. Conference organizers may find this method useful to understand the differences among the delegates, enabling organizers to tailor the conference to delegates’ different needs.
A limitation of Q methodology is that participants are given predetermined statements to sort, and thus novel ideas may be overlooked. However, the statements that participants sorted came from a range of sources, including delegates who had attended OMERACT once and chosen not to return and therefore included a wide range of relevant opinions. In addition, any delegate who had ever attended an OMERACT meeting since it began was contacted to take part, ensuring the widest range of opinions possible.
A further limitation is that the opinions of participants who had never returned to OMERACT were not gathered. Although there were 555 individuals who attended only 1 OMERACT meeting, these represented only one-third (32.9%) of the total meetings attended, and thus two-thirds (67.1%) of the participants overall were people who attended more than once. The decision to include delegates who had attended at least 2 OMERACT meetings ensured they had a more rounded experience of the conference and also accounted for the great majority of attendances over the 11 meetings. It should be noted that patients attend the meetings by invitation and thus could only return if they were given the option. However, the majority of patients who were invited to return accepted the invitation; only 1 patient declined to attend a meeting, because of location. The reasons why some delegates did not return cannot be determined in our study, but because the study includes the bulk of attendees, it reflects the features that they value.
It is possible that sorting 66 statements into categories and then across the fixed distribution grid could be tedious for participants and potentially affect their responses. However, the median time to complete the study indicates that this study did not take as long to complete as estimated (24.87 min). Further, participants’ comments indicated that although the task could be challenging (“Sometimes difficult to rank,” P50, researcher, male), it also prompted them to think about their own priorities (“Makes you realise what you most think is really important when limited,” P77, fellow, female). Overall participants seemed to enjoy taking part in the study, saying it was a “fascinating challenge” (P3, researcher, female), an “interesting exercise” (P20, industry, male), and “a few [statements] even made me laugh” (P35, patient, male).
These data were collected online and thus the qualitative methodology may be slightly weaker owing to the lack of face-to-face interaction between researcher and participant. However, participants were asked to provide open-ended comments about their most strongly sorted statements. These comments cohered with the factor groupings and enabled researchers to explain why participants sorted the statements in certain ways, thus enriching the data.
OMERACT delegates value the focus on outcome measures and global standardization of methods that OMERACT provides. There are 4 ways in which delegates’ views of the “spirit of OMERACT” differ, with some placing more importance on evidence-based research, others on collaboration, some on equality, and a small group being satisfied with the principles of OMERACT but unconvinced by the process. These data provide important information about the aspects of OMERACT that are most valued by frequent delegates and support OMERACT’s descriptions of itself. This information may be useful for potential delegates who are considering attending OMERACT for the first time. Thus, it may be beneficial for the OMERACT committee to explicitly recognize these distinct values.
Acknowledgment
The authors thank the OMERACT delegates who participated in this study, the OMERACT Executive for supporting this study, iDENK, and Dr. Maarten de Wit, whose work informed the statements for this study; Tracey Penberthy at the University of the West of England for setting up and hosting the domain for the online study; and FlashQ (www.hackert.biz/flashq/home) for complimentary use of its software.