Abstract
Objective. To describe the Outcome Measures in Rheumatology (OMERACT) Filter 2.1 methodology for core domain set selection.
Methods. The “OMERACT Way for Core Domain Set selection” framework consists of 3 stages: first, generating candidate domains through literature reviews and qualitative work, then a process of consensus to obtain agreement from those involved, and finally formal voting on the OMERACT Onion. The OMERACT Onion describes the placement of domains in layers/circles: mandatory in all trials/mandatory in specific circumstances (inner circle); important but optional (middle circle); or research agenda (outer circle). Five OMERACT working groups presented their core domain sets for endorsement by the OMERACT community. Tools including a workbook and whiteboard video were created to assist the process. The methods workshop at OMERACT 2018 introduced participants to this framework.
Results. The 5 OMERACT working groups achieved consensus on their proposed core domain sets. After the Methodology Workshop training exercise at OMERACT 2018, over 90% of participants voted that they were confident that they understood the process of core domain set selection.
Conclusion. The methods described in this paper were successfully used by the 5 working groups voting on domains at the OMERACT 2018 meeting, demonstrating the feasibility of the process. In addition, participants at OMERACT 2018 expressed increased confidence and understanding of the core domain set selection process after the training exercise. This methodology will continue to evolve, and we will use innovative technology such as whiteboard videos as a key part of our dissemination and implementation strategy for new methods.
Core outcome sets are increasingly recognized as critical to the design of clinical research1. Following the Outcome Measures in Rheumatology (OMERACT) Filter 2.0 framework2, OMERACT core outcome set development consists of 2 important and sequential components: decisions about what to measure (termed core domain set) and then decisions about how to measure each of the chosen domains — selecting outcome measurement instruments (termed core outcome measurement set). A core outcome set is, therefore, made up of both the core domain set and core outcome measurement set. Outside of OMERACT, the term core outcome set may not differentiate between domains and instruments, and many sets to date may not have progressed to the instrument selection stage. The OMERACT framework has been revised based on discussions initiated at the OMERACT 2018 meeting and will be described in detail in a companion publication3. Our paper focuses on the process to create OMERACT core domain sets and will be useful to both OMERACT working groups and others interested in the methodology of core domain set development. It is worth noting here that there is variability in the terminology used by core outcome set developers and other organizations that use the term outcome domain; an effort is under way to standardize nomenclature across international organizations involved in core outcome set development.
MATERIALS AND METHODS
At OMERACT 2018, 5 OMERACT working groups presented their core domain sets for endorsement by the OMERACT community. The Hip and Knee Osteoarthritis and Juvenile Idiopathic Arthritis Working Groups updated existing core domain sets, with a focus on including the patient perspective. Behcet’s Syndrome, Myositis, and Shoulder Working Groups voted on newly created core domain sets. All 5 working groups followed the “OMERACT Way” methodology for core domain set selection, described in their respective publications, and further detailed below. The Methodology Workshop at OMERACT 2018 focused on outlining the core domain set selection process and featured a breakout group training exercise designed to help workshop participants understand the domain selection process and gain confidence for voting on proposed core domain sets (Figure 1).
The OMERACT Way flowchart to select a core domain set. OMERACT: Outcome Measures in Rheumatology.
Specify the Need for a Core Domain Set
The first step is to formulate a detailed description of the setting or scope to which the core outcome set will apply. Central to this activity is to define the “PICOC statement,” that is, the Patients/Population, Intervention, Comparator/Control, Outcome, and Context, with the understanding that the “O” (Outcome) is what will be defined during the project. The working group will need to generate a comprehensive explanation of criteria including health condition(s; usually disease or disease group) or population to which the intervention will be applied; type of interventions being compared (for example, same or different class of treatments, drugs/biologics, nonpharmacologic, surgery, and other interventions); etc. The working group also needs to decide whether the core outcome set will apply only to randomized trials or to longitudinal observational studies as well. Once the details have been decided upon, the working group must ascertain that no other core outcome sets already exist in the literature, for example, from other professional associations such as the American College of Rheumatology, Osteoarthritis Research Society International, European League Against Rheumatism, and with content experts. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative houses a database on its Website (www.comet-initiative.org) of completed or in-progress core outcome sets. If no overlap is identified, or there is strong justification for developing a new core outcome set despite an existing one (for example, an existing core outcome set that lacked patient participation in its development), then the working group should proceed.
Stakeholder Involvement
OMERACT working groups are composed of an international group of stakeholders including patients and their families, practitioners, principal investigators (trialists, researchers, and methodologists), payers/purchasers, policymakers including regulatory authorities, funders, product makers/private sector, the public, and others, including the press. The core working group consists of at least 3 co-chairs from separate continents and includes patient research partners (PRP) actively engaged in the project, a fellow, and at least 5 content experts. Consideration should be given to involving stakeholders early in the process while recognizing that the extent of involvement may vary over the course of the project4. Working groups are expected to adhere to the OMERACT philosophy regarding communication and engagement of members entitled the “Spirit of OMERACT,”5 especially noting that OMERACT provides clear recommendations regarding involvement of PRP in working groups6.
Framework of Generating, Agreeing, and Voting to Establish a Core Domain Set
1. Generating candidate domains. Working groups undertake both literature reviews and qualitative work with key stakeholders to identify an initial list of domains known as candidate domains. A literature review of randomized controlled trials (RCT) and longitudinal observational studies (LOS) identifies existing domains previously used, often by extracting domains from the identified outcome measurement instruments, and provisionally places them into 1 of the 4 framework core areas: manifestations/abnormalities, life impact, death/lifespan, and societal/resource use (see also the section below on agreeing on domains)3. Each domain and its working definition should be extracted from included trials/studies with sufficient detail to communicate clearly what was being measured. The domain of “pain” could, for example, be pain intensity or its effect on daily activities: these are different aspects or concepts of pain. In this example, pain intensity would fit into the core area of “manifestations/abnormalities” while pain impact on daily activities would fit into “life impact.” At this point in the process, the purpose is to endeavor to identify as many different domains as possible. The review is often led by an OMERACT fellow attached to a working group and involves a librarian or information specialist to design the search strategy.
Qualitative work is undertaken when there are insufficient publications analyzing the patient, provider, or other stakeholders perspectives on an international level. The aim is to obtain a broader and deeper understanding of important perspectives on the effect of a health condition. Such work frequently identifies and operationalizes new domains.
Qualitative work includes interviews, focus groups, and/or surveys of important stakeholder groups, including people with a lived experience of the health condition. Formal qualitative research is an excellent way to obtain the experiences of patients, family, and healthcare providers, with the goal to investigate the character and spectrum of the domain (e.g., impact on function/work/family/social/leisure activities, fatigue, or pain) encountered with the disease/condition7. Sound qualitative methods must be used, preferentially with the collaboration of a qualitative methodological expert. This is to ensure rigor in study design, including theoretical underpinning; patient selection; conduct, recording, and transcribing of interviews; data analysis; and interpretation8. A sufficient number of participants must be involved to obtain a full understanding, with no new information arising in subsequent interviews. Focus groups are often used rather than individual interviews to benefit from participants discussing proposed domains and their shared experiences. Concept saturation is sought, but OMERACT recommends that focus groups aim to be as representative as possible of potential clinical trial participants, with a minimum of 30 participants total, and representation from at least 3 continents, following guidance from Francis, et al9. OMERACT plans to analyze innovative techniques such as online discussion boards that can create virtual focus groups, allowing geographically dispersed participants to provide in-depth responses in a moderated setting10.
2. Agreeing on domains to be included in the draft core domain set: methods to prioritize candidate domains. The purpose of this stage is to refine the initial list of proposed domains to those that participants agree are critically important to a core domain set. Following the revised OMERACT Filter 2.1 framework, more fully described in the companion paper3, domain selection begins with an understanding of the key areas of health included within the concepts of “pathophysiology” and “impact.” For example, in “pathophysiology,” the area of “manifestations” refers to signs, symptoms, or biomarkers that characterize the health condition or disease. Included in the concept of “impact” are the areas of “life impact,” “death/lifespan,” and “societal/resource use.” Societal/resource use is highly recommended but considered optional because of the additional expertise and time needed.
Candidate domains are then formulated within this framework, ensuring that each core area of manifestations, life impact, and death/lifespan is represented by at least 1 domain in the candidate domain list. This formulation also implies that the working group needs to carefully review the candidate domains and consider where there may be enough overlap between similar domains to eliminate or combine domains, a process known as binning and winnowing. For example, activities of daily living (e.g., bathing) and recreational activities could be combined into a single domain called “physical functioning,” with the caution to avoid diffusion of key concepts.
The foundation for consensus is the combination of the literature reviews and qualitative studies used to identify candidate domains. Achieving consensus through sound, transparent methodology is a key tenet of OMERACT. Consensus is not a majority vote; the emphasis is coming to a decision that (almost) everyone can agree upon or at least accept, through thoughtful engagement of participants who understand the content of the work and have participated in the decision-making processes. To ensure that PRP have an equal voice, we recommend engaging patient and consumer groups from different countries, representing at least 3 continents. Frequently, working groups use the Delphi process, a formal consensus method usually conducted online that allows for broad, international stakeholder involvement. OMERACT requires stratification of the Delphi results by PRP versus other participants to see if there is a difference.
From a list of candidate domains, participants select those they believe to be of critical importance for inclusion in a core domain set, and this process is conducted iteratively over 2 or more typically 3 rounds until agreement is reached. Aggregate feedback is given between rounds and participants’ previous votes are viewed, often with an option in the first round to suggest additional domains. OMERACT applies a threshold of ≥ 70% participant agreement that a domain is of sufficient importance to be included in a draft core domain set to achieve consensus. It is usually necessary to prioritize those domains selected as critically important using a ranking process to reduce the number of potential domains. OMERACT has developed guidance on the use of the Delphi method, including a checklist with detailed recommendations, provided in a companion paper11.
Working groups can also use a variety of group facilitation techniques to engage people in understanding and prioritizing domain selection. For example, vignettes with small videotaped testimonials discussing individual domains could be placed online and reviewed at participants’ leisure. Card-sorting exercises, as described in The Workshop Book: From Individual Creativity to Group Action12, in face-to-face meetings may be used, in which participants use file cards and sort them on a wall to prioritize choices for important domains. Dot votes have been used at OMERACT meetings, in which a fixed set of colored dots are allocated to each participant to use as “votes” endorsing candidate domains they consider to be mandatory. “Speed dating” circles have participants move to stations across a room where working group members explain and help to champion a specific domain. These and other techniques are only suggestions; some method(s) should be selected to help the working group membership and the OMERACT community as a whole to better understand the importance of each of the domains. OMERACT favors techniques that are participatory and action-oriented to raise key discussions and prepare group members for an informed consensus vote. Engaging people in the material they are making decisions about is critical to a good consensus process.
Regardless of the consensus methods, key elements to consider when conducting a consensus process are:
Dialogue phase. Sharing candidate domains and their definitions in a meaningful, easily understandable way that includes expression of all stakeholders’ opinions, thinking, and discussion of these opinions; listening, equal participation, and voice, and management of barriers. Good facilitation is essential.
Decision-making phase. Exercises to help participants evaluate and formulate their choices, and listening to others about their choices (e.g., why did people think pain was the most important domain?). Informal voting activities may take place, but for decision making, formal voting must be anonymous as described in the companion paper on consensus11.
3. Voting: The OMERACT Filter 2.1 Onion. Working groups organize the selected domains into what has become known as the OMERACT Onion (Figure 2, with definitions). In the inner circle are the “mandatory” or core domains. This layer is divided into 2 parts: (1) mandatory for all trials, and (based on feedback originating from the working groups voting on multiorgan involvement at OMERACT 2018), (2) mandatory in specific circumstances for which the criteria must be clearly defined, e.g., features relevant to a specific juvenile idiopathic arthritis feature such as eye involvement/uveitis10. The next layer out (middle circle) is the “important but optional domains”: inclusion of these domains is strongly recommended but optional. Finally, the outer circle consists of the “research agenda”: domains of interest that need further consideration to determine their importance.
Representation of the placement of domains in the circles/layers in the OMERACT Filter 2.1 Onion. OMERACT: Outcome Measures in Rheumatology.
At the workshop at the OMERACT conference, the working group must make a proposal for the domains that were prioritized as important, that is, those that reached ≥ 70% endorsement as being critical to include in a core domain set during the consensus process. These will be placed in either the “mandatory” or the “important” layers of the Onion based on working group discussions. The “mandatory for all trials” are those that the working group agrees should be measured in every RCT/LOS. Working groups that use the “mandatory in specific circumstances” layer need to ensure that the “circumstances” are indeed specified and describe the domain(s) that are included in the core set under the appropriate circumstances. Definitions and descriptions of these domains must also be provided in the same manner as other core outcome domains. It is understood that various combinations of domains listed within “mandatory in specific circumstances” may be invoked, depending on the type of RCT/LOS and/or a specific patient’s clinical pathway within a study. This layer will likely be particularly useful when considering core sets for multisystem diseases with variable clinical presentations and provides for flexibility to better reflect the complexity of clinical investigation. OMERACT encourages groups to be parsimonious in what is placed in the inner circle, which means that several important domains will not be designated as “mandatory.” OMERACT suggests aiming for no more than 7 domains in the inner “mandatory” domains layer. This will become the “core domain set,” that is, the minimum number of domains necessary to adequately identify what is important. Feasibility of a core domain set strongly depends on parsimony, given the need to minimize participant burden and cost of conducting trials, across a small set of core outcome domains, because these will be required in all relevant studies of the benefits, harms, and cost of the interventions of interest.
The “important but optional” domains may be those in which groups disagreed (e.g., with only 1 stakeholder group voting for the inner circle) or those in which there was ≥ 70% agreement but were not ranked high enough to be in the inner circle, and the working group agrees that they are important. Domains that the group feels still need additional work to clarify their involvement and placement can be placed in the “research agenda” layer, provided that direction for future work is defined.
Those domains placed as “mandatory” or “important but optional” need a clear, detailed definition with an explanation of underlying theories, if applicable, and an explanation of the scope (breadth and depth) of the domain. This detailed definition is important so that others within and outside the working group clearly understand the domain. It is essential in content- and concept-matching when undertaking instrument selection. We recommend that definitions include direct examples from qualitative studies (e.g., patient descriptors of a particular symptom complex or concept such as “participation”).
The working group seeks OMERACT consensus and endorsement of the core domain set through a vote from the OMERACT community. This vote takes place after a session at the biennial OMERACT meeting, when delegates engage in plenary and small-group discussions based on a detailed summary of the results of the working group activities (Figure 3). For several reasons including feasibility, PRP often represent a smaller proportion of the voting participants. A vote of ≥ 70% agreement with the proposed core domain set by both groups — PRP and all other stakeholders — is required for endorsement.
OMERACT summary of core domain set activities. This can be used to summarize the work that was done during the development of the core domain set. OMERACT: Outcome Measures in Rheumatology.
In addition to the guidance in chapter 4 of the OMERACT Handbook13, the OMERACT Master Checklist and Workbook for Core Domain Set Development have been created to help working groups keep track of their progress and ensure full and transparent reporting according to the Core Outcome Set–Standards for Reporting (COS-STAR) statement14. These resources are available on the OMERACT Website, as well as a whiteboard video summary (https://omeract.org/resources). OMERACT has recently established a Technical Advisory Group whose role is to ensure that the methods and processes described above are followed by the working groups; their endorsement prior to a meeting will allow the wider OMERACT community to focus on the results presented by the working groups.
RESULTS
At OMERACT 2018, based on the methodology described above, 5 OMERACT working groups achieved consensus (i.e., ≥ 70%) on their proposed core domain sets10,15,16,17.
After the Methodology Workshop training exercise at OMERACT 2018, 52% of participants voted that they were “very confident” that they understood the process of core domain set selection and 45% “somewhat confident.” In addition, 60% of participants voted that they would be “very confident” and 38% “somewhat confident” if they were asked to vote on a proposed OMERACT Filter 2.1 Onion as prepared for the domain votes held at OMERACT 2018.
DISCUSSION
OMERACT has worked on developing core outcome sets since 1992. Over the years, the methodology has evolved significantly, and our paper describes the latest guidance developed by the OMERACT Handbook Committee with input from other international groups working in the field of core outcome set development such as COMET. The methods described here were successfully used by the 5 working groups voting on domains at the OMERACT 2018 meeting, demonstrating the feasibility of the process. In addition, 90% of participants at OMERACT 2018 expressed increased confidence and understanding of the core domain set selection process after the training exercise. This methodology defining core domain set development is an ongoing process and will continue to evolve. We will use innovative technology such as whiteboard videos as a key part of our dissemination and implementation methodology for new methods.
Acknowledgment
Institute for Work & Health, Toronto, for assistance with graphic design of the OMERACT Way flowchart to select a core domain set.
Footnotes
PGC is funded in part by the NIHR Leeds Biomedical Research Centre. The views expressed in this article are those of the authors and not necessarily those of the UK National Health Service, the NIHR, or the Department of Health. JAS is supported by the resources and the use of facilities at the VA Medical Center at Birmingham, Alabama. LM is a Principal Investigator on the Australian Rheumatology Association Database, which has received arms-length funding from Pfizer Australia, AbbVie Australia, Eli Lilly Australia, and Janssen Australia. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs or the US government.
- Accepted for publication December 19, 2018.
