Abstract
Objective. While there has been substantial progress in the development of core outcomes sets, the degree to which these are used by researchers is variable. We convened a special workshop on knowledge translation at the Outcome Measures in Rheumatology (OMERACT) 2016 with 2 main goals. The first focused on the development of a formal knowledge translation framework and the second on promoting uptake of recommended core outcome domain and instrument sets.
Methods. We invited all 189 OMERACT 2016 attendees to the workshop; 86 attended, representing patient research partners (n = 15), healthcare providers/clinician researchers (n = 52), industry (n = 4), regulatory agencies (n = 4), and OMERACT fellows (n = 11). Participants were given an introduction to knowledge translation and were asked to propose and discuss recommendations for the OMERACT community to (1) strengthen stakeholder involvement in the core outcome instrument set development process, and (2) promote uptake of core outcome sets with a specific focus on the potential role of post-regulatory decision makers.
Results. We developed the novel “OMERACT integrated knowledge translation” framework, which formalizes OMERACT’s knowledge translation strategies. We produced strategies to improve stakeholder engagement throughout the process of core outcome set development and created a list of creative and innovative ways to promote the uptake of OMERACT’s core outcome sets.
Conclusion. The guidance provided in this paper is preliminary and is based on the views of the participants. Future work will engage OMERACT groups, “post-regulatory decision makers,” and a broad range of different stakeholders to identify and evaluate the most useful methods and processes, and to revise guidance accordingly.
To address the critical need for standardized outcome domains and measurement instruments in rheumatology clinical trials, the international organization Outcome Measures in Rheumatology (OMERACT) has developed core outcome sets of domains and instruments using an iterative, data-driven, consensus-based approach1,2. A key aspect of its research methodology is to obtain input from a broad range of people, including patient research partners, the industry, and regulators. OMERACT’s strategies to promote uptake of recommended core outcome sets have predominantly focused on reaching clinical trialists, methodologists, regulators, and industry partners through journal publications.
For OMERACT 2016, a special workshop on knowledge translation was convened with 2 main goals. The first focused on the development of a formal knowledge translation framework to incorporate state-of-the-art innovations in knowledge translation and to strengthen stakeholder engagement and ensure effective input representing a broad range of interests. The second goal focused on promoting uptake of recommended core outcome instrument sets.
Knowledge translation3 was defined to workshop participants as activities that make users aware of knowledge (i.e., core outcome instrument sets) and that facilitate the use of this knowledge to improve health and healthcare systems with an aim to close the gap between what we know and what we do (i.e., using OMERACT-endorsed core outcome sets in clinical trials, systematic reviews, etc.).
Knowledge translation can be split into 2 components: (1) integrated and (2) end-project knowledge translation. Integrated knowledge translation — also known by such terms as collaborative research, participatory research, engaged scholarship, co-production, and co-creation — is a collaborative or participatory approach that engages end users in the research process, starting with their involvement in defining the research question4,5. This engagement occurs with the expectation that it will result in research outputs that are more relevant, useful, and readily useable to the end users and therefore more likely to be implemented4. Effective engagement requires additional considerations both for patients and other stakeholders6,7. Different stakeholder groups may be broadly defined as the following: patients and their families, the public, providers, payers/purchasers, policymakers, principal investigators (researchers and funders), product makers, and others, such as the press8,9.
End-project knowledge translation is about translating research findings into policy and practice and is essential for optimizing the effect of research. In OMERACT’s case, the adoption of core outcome sets by clinical trialists, systematic reviewers, guideline developers, regulators, and others helps ensure comparability across studies and improves the ability to synthesize and interpret the evidence base of interventions for rheumatic conditions10,11,12. This translation of research into policy and practice is best viewed as a process within which there are various considerations. Among these are defining the specific (current and potential) contexts of use for a given outcome or core set, identification of relevant stakeholders, developing an engagement plan, establishing a strategy for promoting uptake, and enabling plans for implementation and measuring uptake and effect. Table 1 lists specific steps to consider across the different stages.
OMERACT knowledge translation steps.
MATERIALS AND METHODS
We held a workshop session during OMERACT 2016 to generate ideas for developing “best practices” in stakeholder engagement for OMERACT working groups and to promote the uptake of core outcome instrument sets. All 189 OMERACT 2016 attendees were invited to the workshop and 86 attended. Workshop participants included patient research partners (n = 15), healthcare providers/clinician researchers (n = 52), industry (n = 4) and regulatory agency (n = 4) representatives, and OMERACT fellows (n = 11). Two presentations at the start of the session provided participants with a broad introduction to knowledge translation. These were followed by presentations from 2 current OMERACT Working Groups on their strategies for stakeholder engagement and promoting uptake of their work. The Rheumatoid Arthritis Flare Working Group has been working for several years to establish a means to identify clinically significant worsening of rheumatoid arthritis disease activity, primarily as an outcome measure for use in clinical trials but also for potential use in other settings including clinical practice13. The Worker Productivity Working Group has sought to identify instruments that could be used to measure at-work productivity loss due to rheumatologic conditions14. Examples from these 2 groups are presented in our paper as case studies.
Workshop participants then moved to 6 breakout groups each led by 2 OMERACT executive members. Participants were asked to develop and discuss recommendations on strategies for the OMERACT community to (1) strengthen stakeholder involvement in core outcome set development, and (2) promote the uptake of core outcome sets. A rapporteur from each breakout group presented key findings back to the entire group.
RESULTS
Goal 1. Strengthening Engagement with Stakeholders during the Development of Core Outcome Sets
Who to involve?
Establishing the context(s) of use serves as an important starting point for how a group would begin to consider who should be engaged in a research project to ensure use. For maximum effectiveness, broad engagement should occur throughout the entire core outcome set development process, from conceptualizing the question to developing the research agenda and protocol, conducting the research itself, seeking interpretation and comments on the results, and creating audience-specific information to promote the uptake and use of the recommendations (Figure 1). However, the simple formulation that every stakeholder should be equally involved from beginning to end is likely not the most effective or efficient approach.
OMERACT framework of stages of stakeholder engagement during integrated knowledge translation process. COS: core outcome domain or instrument set; OMERACT: Outcome Measures in Rheumatology. Figure 1 was adapted from the following: Canadian Institutes of Health Research, Figure 7: Integrated Knowledge Translation Research Cycle, CIHR Citizen’s Engagement Handbook, page 69, http://www.cihr-irsc.gc.ca/e/42211.html, 2010. All rights reserved. Reproduced with the permission of the Canadian Institutes of Health Research, 2017.
Optimal engagement requires identification of the right people, their involvement at the appropriate phases of the core outcome set research process, and the integration of their perspectives in the best possible way to maximize the effect of their input. The overall research program to develop a Core Outcome Instrument Set represents an ongoing and iterative process, with different types of input required along the way (e.g., qualitative expertise in domain identification and content validation, psychometric expertise in instrument evaluation and development). It is not necessary to involve every conceivable stakeholder at each stage to an equal extent. It is important to create a shared understanding among stakeholders and researchers concerning their expected roles within the overall process, and the commitments that will be asked of them (e.g., time, travel, etc.). It should also be understood that their involvement may differ depending on the stage of the research project. OMERACT has an extensive history of engaging with patient research partners (people living with a disease or condition who actively and equally contribute to research projects) to ensure that the patient perspective is taken into account when identifying important outcome domains15,16,17,18,19,20. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group has a “GRADE Stakeholders Group” project that is currently under way. The group provides guidance on who should be involved in the GRADE guideline development process, and how to identify, engage, and involve them. This GRADE guidance may be relevant for OMERACT to consider when developing integrated knowledge translation strategies.
Participants at the OMERACT 2016 workshop identified those who should be considered (Table 2). To ascertain who best to engage, they recommended identifying and networking with key opinion leaders in relevant clinical areas.
Stakeholders identified by OMERACT workshop participants.
Case studies: Who to involve?
Rheumatoid Arthritis Flare Working Group
Brought together an international, multidisciplinary group consisting of patients, providers (physicians, nurses, psychologists, and other allied health professionals), clinicians who perform clinical trials and those who design studies from academia and industry, funders, research experts (qualitative researchers, biostatisticians, clinical epidemiologists, and psychometricians), and pharma and regulatory representatives.
Engaged with fellows and trainees to ensure a pipeline of individuals committed to ultimate adoption of the outcomes being studied, and for them to observe engagement of partners in research.
Worker Productivity Working Group
Engaged with an international and interdisciplinary group including patient research partners and representatives from epidemiology, health economics, industry partners, rehabilitation, rheumatology, work disability research field, the International Canadian Arthritis Network for Work Outcomes (I CAN Work) and a wide array of arthritis researchers.
How to engage? During the OMERACT workshop, participants suggested the following strategies for better stakeholder engagement:
Involve the “right” stakeholders from the start of the project and throughout the development of the core outcome set while acknowledging that consideration should be given to which should be involved, and to what extent, at each phase of the core outcome set development project.
Provide them in a timely manner with necessary information such as pre-reading materials to ensure informed engagement.
Hold working group meetings at large national and international conferences where those involved often meet to increase opportunities for face-to-face interaction; consider paying expenses for key people to attend.
Expand the stakeholder community using virtual meetings and voting.
Case studies: How to engage?
Rheumatoid Arthritis Flare Working Group
Held ongoing interactions between face-to-face meetings by tele/Web conferences and other means of communication (e.g., e-mail).
Developed pre-briefing/debriefing calls and specific education sessions for patient research partners; tools such as the OMERACT glossary were found to be particularly helpful.
Engaged patient research partners throughout the process in the following ways with major roles in participation/leadership in the Working Group: participating as members of the Steering Committee, analyzing qualitative data, developing the questionnaire, debriefing the questionnaire, interpreting results, facilitating and moderating OMERACT plenaries, presenting research findings at other international meetings, writing publications.
Presented research results at various points in the instrument development process at biennial OMERACT meetings; used marketing materials to raise awareness and obtain endorsement from OMERACT participants. At meetings, working group members wore t-shirts, brooches, and an extra name tag that read “ASK ME ABOUT RA FLARE”. A wide-ranging audience representing multiple constituencies in small breakout groups provided important feedback.
Worker Productivity Working Group
Started knowledge translation engagement efforts early, first to introduce the domain, establish need, and identify people eager to be involved; included a highly active and encouraging patient group from the beginning; continued to hold meetings at biannual OMERACT meetings as well as international workshops in between.
Held an “overt” engagement blitz at OMERACT 2014 using the branding “It Works!” with group members wearing T-shirts to advertise whom to ask questions of throughout the conference, along with sticky notes, pamphlets, and tables summarizing the evidence. A fun, high-energy breakout session was held using the format of speed dating with various working group members providing the OMERACT Filter evidence on different instruments as session participants moved around the room.
Patient research partners contributed equally to design and proof of questionnaires and surveys, interpretation of findings, and assistance with study recruitment through their networks. However, their biggest influence was identifying contextual factors and expanding understanding of the concept of interest so that measuring productivity meant understanding the job situation, e.g., “If they could give me flexible hours I would be at 100%, but now I am at 75%.”
Goal 2. Promote the Uptake of Core Outcome Sets
Participants suggested creative and innovative ways of transferring information beyond traditional peer-reviewed publications and presentations at professional meetings. Table 3 describes specific considerations for promoting uptake of core outcome sets. The potential factors that could influence the implementation of core outcome sets that were described to the workshop participants during the first part of the session are outlined in Table 421,22,23,24,25.
Considerations in promoting the uptake of research results.
Factors that could influence uptake of core outcome instrument sets.
Workshop participants suggested the following approaches to promote the uptake of published core outcome sets and other key OMERACT findings by the broader community:
Revise the OMERACT Website to make it easier for people to find key information, provide an RSS feed to deliver updated Website content, and actively use social media (e.g., Twitter, Facebook).
Highlight OMERACT achievements through an OMERACT newsletter to disseminate highlights from finished work; more lay publications (not just academic journals), e.g., American College of Rheumatology (ACR)/European League Against Rheumatism (EULAR) highlights.
Hold an “OMERACT Findings Symposium” attached to ACR/EULAR or other locations where payers, regulators, outcomes methodologists, and health technology assessment agencies hold their major conferences; send an OMERACT representative to major meetings.
Develop a toolkit using plain language to describe both the methods behind the development of core outcome sets and the resulting set of recommended outcomes; use short messages targeted to different stakeholders; develop an app for the toolkit.
Use standard-practice marketing strategies, e.g., consider the presence, profile, and penetration of OMERACT in the different groups; use story-telling; evaluate marketing plans after implementation.
Conduct strength, weaknesses, opportunities, challenges/constraints (SWOC) analysis for plans to promote the uptake of recommendations.
Early in their research program, OMERACT working groups should develop a promotion/marketing strategy to implement the core outcome set; consider the relevance of OMERACT to each stakeholder group and prioritize groups.
Deposit OMERACT core outcome sets in outcome measurement repositories, e.g., EULAR Outcomes Measures Library (oml.eular.org), US Food and Drug Administration (FDA)’s compendium of clinical outcome assessments, Mapi Research Trust, and Patient-Reported Outcome and Quality of Life Instruments Database (eprovide.mapi-trust.org), and other relevant databases.
Engage patient research partners to work with the committees, associations, and arthritis communities with which they are involved to increase OMERACT’s profile at the grassroots level; link to patient organization Websites (e.g., www.creakyjoints.org).
Continue the concept of “generosity of ideas and collaboration” to help spread information from OMERACT’s work.
Moving from dissemination to facilitating implementation: Engaging payers and other “post-regulatory decision makers”. A specific focus of the workshop was on the potential influence of “post-regulatory decision makers” in increasing the uptake of core outcome instrument sets. The importance of regulators such as the FDA and the European Medicines Agency in promoting implementation of core outcome instrument sets has been well recognized by OMERACT and other developers of core outcome instrument sets for many years. Regulators have defined mechanisms to review, approve, and communicate preferred outcomes through guidance documents, compendia, etc. Researchers from the life sciences industry and elsewhere are highly motivated to pay close attention to the health outcomes that are recognized by regulators, given the implied significance of those outcomes in regulatory decisions. The potential effect of “post-regulatory decision makers” (Table 5) in creating strong incentives for researchers to use health outcomes recognized by these decision makers has been less appreciated. The premise behind working with these groups is their explicit recognition of core outcome sets as influential in their decision making. This would create strong incentives for researchers to use those core outcome sets, much as FDA recognition of core outcome sets is a strong motivator for their use26.
Examples of influential post-regulatory evidence evaluators and decision makers.
Because each of these organizations directly or indirectly influences the speed and extent of market uptake of new drugs, devices, diagnostics, and procedures — and the prices paid for these products — the role of health outcomes in their decision making has important practical consequences for product developers and other researchers. For this reason, when and how to effectively and efficiently engage these stakeholders in core outcome instrument set development and strategies for promoting uptake are important areas for further investigation. It is unlikely that representatives of these groups will be available to actively participate in all phases of core outcome set development. Potential conflicts of interest should always be considered and made explicit.
Case studies: Strategies for promoting uptake
Rheumatoid Arthritis Flare Working Group
Developed text and video stories about the importance of patient-reported outcomes as outcomes from multiple perspectives (patients, clinicians, researchers), results from a research study, and the effect patient-reported outcomes had on making health decisions (www.hopkins-arthritis.org/PCOR).
Made plans to further disseminate this information through social media to a larger community. Such efforts require additional expertise (for example, from medical writers and media professionals) to provide appropriate context and to make information accessible for stakeholders.
Worker Productivity Working Group
Published and presented findings, and involved working group members in promoting results within existing networks.
Considered further dissemination of these findings on the measurement properties of instruments measuring work productivity to potential users. These may include health technology assessment agencies, “owners” of the instruments, work disability researchers, and policy decision makers.
DISCUSSION
Challenges. Workshop participants noted potential challenges in implementing the ideas outlined above. Core outcome instrument sets and the methodology behind developing them is complex, and clear communication of this information can be difficult. Many different stakeholders were identified and methods are needed to prioritize whom to target in which stages of the process. Leveraging networks of patient research partners is one essential strategy to pursue for improved uptake, but may require developing training materials in lay language. Consistent with the recent recommendations for patient research partner involvement in OMERACT research projects27, further work is needed to develop and standardize training. This may involve training of researchers in engagement strategies with patient research partners. The issue was also raised of weighting patient involvement, including weighted patient voting, to ensure that they are not a minority likely to be outvoted by other groups.
OMERACT is an international organization and an ongoing challenge is to ensure geographical representation. Many major national and international conferences do not allow concurrent meetings by other organizations, which may limit the ability to engage with stakeholders at these venues. Recognition of and explicit discussion about real and potential conflicts of interest is important and necessary to ensure the integrity of OMERACT’s program of work. Lastly, the strategies to facilitate implementation of core outcome sets should not be an afterthought; initiating these strategies is resource-intensive and it takes a substantial amount of time and energy, thus requiring planning and budgeting from the beginning of every initiative.
Limitations. This guidance is preliminary and is based on the views of the participants who attended OMERACT 2016. While there was good representation from the different groups at OMERACT, it is necessary to include a larger number of stakeholders when evaluating the strategies discussed in our paper. The focus of discussion was within the field of rheumatology, and further work with core outcome set developers in other fields would be useful. The knowledge translation concepts were adapted from work that was focused on the dissemination and implementation of research findings; core outcome set development and promotion may not be precisely comparable.
Research agenda for promoting uptake. Next steps include prioritizing the approaches suggested above for promoting uptake of OMERACT Core Outcome Instrument Sets. We will also focus on clarifying how best to work with “post-regulatory decision makers” by collaborating with the Center for Medical Technology Policy (CMTP) to define a parallel set of mechanisms through which post-regulatory decision makers could recognize health outcomes that best inform their decision making28. These mechanisms will be identified by conducting a series of interviews with key representatives and a meeting during which we will identify and evaluate potential mechanisms through which these organizations could encourage implementation of core outcome instrument sets. We will engage with OMERACT groups, stakeholders, and other core outcome set developers to evaluate their experiences with implementing these knowledge translation approaches and revise our guidance as necessary.
OMERACT has developed an international reputation for high-quality, leading-edge methodology over the last 25 years, and we now recognize the need to strengthen our engagement with potential users of the products of our work, and to market the evidence-based, consensus-driven core outcome sets that we have established. Further work on promoting uptake of core outcome sets is now under way, through collaboration between the CMTP and OMERACT, with a focus on “post-regulatory decision makers.” The OMERACT executive will engage with OMERACT Working groups to identify the most useful knowledge translation methods and processes. These will be used to inform recommendations in a chapter in the OMERACT Handbook on engaging stakeholders and strategies for promoting uptake of core outcome sets to support individual OMERACT Working Groups. We will undertake evaluation of our knowledge translation strategies on an ongoing basis.
- Accepted for publication May 31, 2017.
